Designing Your Mystery Shopping Questionnaire: Best Practices

Designing Your Mystery Shopping Questionnaire: Best Practices

The questionnaire is the foundation of any mystery shopping programme. All the investment in briefing shoppers, conducting visits, and analysing results only generates value if the questionnaire is designed to capture the right information in a way that produces reliable, actionable data. A poorly designed questionnaire generates noise rather than intelligence, and the actions taken on the basis of that noise may do more harm than good.

Scout Insights designs bespoke mystery shopping questionnaires for each client as part of programme development. This guide shares the principles that produce questionnaires that work. Contact Scout Insights to discuss questionnaire design for your programme.

Start with Business Objectives, Not Questions

The most common mistake in questionnaire design is starting with a list of things to assess rather than with the business question the programme is designed to answer. Before writing a single question, articulate clearly:

• What decision will this data inform?
• What aspects of the customer experience are most critical to our business outcomes?
• What specific behaviours or service standards are we trying to assess?
• What would good look like in each area we are assessing?

Questions that cannot be connected to a specific business objective are candidates for removal. Every question in a mystery shopping questionnaire should be there because the answer will inform a decision or action.

Question Types and When to Use Each

Binary questions (yes or no)

Binary questions are the most reliable assessment tool in a mystery shopping context because they eliminate rater subjectivity. Either the greeter said ‘good morning’ or they did not. Either the promotional item was mentioned or it was not. Either the wait time exceeded the standard or it did not.

Binary questions are best used for observable, specific behaviours where the answer is unambiguous. They cannot capture the quality or manner of a behaviour, only its presence or absence.

Scaled questions (1 to 5 or 1 to 10)

Scaled questions ask the shopper to rate the quality of something on a numeric scale. They are useful when binary capture is insufficient, for example, when you want to assess not only whether a product was recommended but how confidently and knowledgeably it was recommended.

The limitation of scaled questions is inter-rater reliability: different shoppers will interpret the scale differently unless it is carefully anchored with specific descriptions of what each number on the scale means. A scale with defined anchors (1 = employee had no product knowledge, 3 = adequate knowledge, 5 = detailed, confident, and persuasive product knowledge) produces more consistent data than an unanchored scale.

Open-ended observation fields

Open-ended questions ask the shopper to describe what they observed in their own words. These responses are not easily quantified but provide context that binary and scaled scores cannot. They capture the nuance of what happened, the specific language used, and the observations that did not fit neatly into the structured questions.

Open-ended observations are most valuable when read alongside the quantitative scores, providing the ‘why’ behind the numbers. They are also particularly useful for capturing excellence, as a score of 5 and a brief narrative of what made the interaction exceptional is more useful for training purposes than a score of 5 alone.

Designing for Reliability

Limit the total number of questions

Questionnaire fatigue is a real risk in mystery shopping. A shopper who is completing a 60-question questionnaire after a 45-minute visit is more likely to rush, provide less accurate responses, and miss important observations than one completing a focused 20-question instrument. Aim for the minimum number of questions needed to assess the critical areas, not the maximum number that could conceivably be relevant.

Sequence matters

Questions should follow the natural sequence of the customer experience. A questionnaire that asks about the checkout process before the service interaction mirrors the chronological experience of the visit and makes it easier for the shopper to recall specific details accurately.

Test before launching at scale

Every new questionnaire should be tested with a small number of visits before full programme launch. Testing reveals questions that shoppers interpret differently from how they were intended, questions that cannot reliably be answered from the customer experience, and questions that produce so little variation in the responses that they do not generate useful data.

What Not to Ask

• Do not ask shoppers to assess things they cannot observe as a genuine customer (internal processes, staff conversations they are not party to)
• Do not ask questions that are so subjective that different shoppers will reliably give different answers to the same interaction
• Do not include questions that duplicate each other, as this adds length without adding information
• Do not ask for specific employee names unless the programme design has specifically accounted for the privacy and employment law implications of individual attribution

Frequently Asked Questions

How many questions is the right number for a mystery shopping questionnaire?

There is no universal answer, but as a general principle, questionnaires of 15 to 25 structured questions, supplemented by targeted open-ended observation fields, are manageable for shoppers and generate sufficient data for meaningful analysis. Programmes with more than 40 structured questions tend to suffer from shopper fatigue and reduced data quality. Programmes with fewer than ten questions may not capture sufficient complexity to be actionable.

Should the questionnaire be the same across all locations in a multi-site programme?

A consistent core questionnaire across all locations enables benchmarking and comparison, which is one of the primary values of multi-site mystery shopping. Location-specific supplementary questions can be added to capture site-specific priorities without disrupting comparability on core criteria. Scout Insights’ approach to multi-site programmes is covered in our guide to mystery shopping at scale.