How to Analyze Open-Ended Survey Responses
Code open-ended survey answers into themes without losing nuance, overstating frequency, or exposing respondents.
Open-ended answers help explain ratings and reveal issues your answer choices missed. Analysis requires more than pasting comments into a summary. You need a repeatable way to group ideas while preserving unusual but important feedback.
Preserve the original voice while creating structure
Start by exporting the full answer, question wording, and relevant context such as product stage or channel. Remove unnecessary personal information before sharing the data for coding. Read a diverse set of responses without immediately assigning labels. Note recurring problems, unexpected language, and answers that do not actually address the question. A first pass prevents the codebook from merely reflecting the options your team expected to see.
Image source: SurveyWorth original explanatory diagram, 2026. Illustrative, not observed survey data. Based on this article’s “Preserve the original voice while creating structure” section(s); not a product screenshot.
Define the coding unit. One response may contain several distinct ideas, so a single-label approach may erase meaning. “The shipping fee surprised me, and the delivery date changed twice” reasonably receives both cost and delivery-clarity codes. If your research question needs the primary reason, establish a rule for identifying it or ask the respondent directly in the survey.
Create a codebook that another person can use
Each code should have a short name, a definition, inclusion and exclusion rules, and an example. The CDC Field Epidemiology Manual recommends written codebook rules to reduce drift across reviewers and time. “Price” might include total cost and subscription fee, or you may split them if different teams own the fixes. “Other” should not become a large bin of unrelated important issues. If new themes emerge, revise the codebook and recode the earlier answers under the new rules.
For a large set, test the codebook on a subset with another reviewer. Discuss disagreements such as whether “I couldn't find the shipping estimate” is a navigation problem, a missing-information problem, or both. Agreement is useful because it exposes unclear rules; do not treat a single consistency statistic as proof that the interpretation is correct.
A small codebook entry can make the rule concrete:
| Code | Include | Exclude |
|---|---|---|
| Unexpected cost | A fee or total that surprised the respondent | A price they knew but considered too high |
| Delivery uncertainty | An unclear or changing date estimate | A known date that was simply too late |
These distinctions matter because the first and second cases may require different fixes. Save the codebook version with the coded dataset. Voxco’s survey-coding guide distinguishes the act of labeling text from the later interpretation of a theme. Typeform’s open-question guide explains why free text can surface answers that fixed choices miss. Neither source turns a coded theme into a population estimate.
Count with the correct denominator
If 200 people took the survey but only 80 answered the optional open question, a theme mentioned by 20 people is 25% of open-answer respondents, not 10% of all survey respondents unless you explicitly choose that base and explain it. Also decide whether you count people mentioning a theme or total mentions. One long response should not inflate the frequency simply because it repeats a complaint.
A theme count does not reveal its impact. A rare accessibility barrier may deserve urgent attention, while a common cosmetic preference may not. Report frequency alongside severity, affected task, and illustrative de-identified quotes. Avoid quoting a comment that could identify an employee or customer through distinctive details.
Use automated help carefully
Text clustering or AI can suggest candidate codes and speed review. It can miss sarcasm, merge distinct problems, or hallucinate a theme. Check machine-assigned labels against original answers and sample every important category. Protect personal data according to your survey promise and tool agreements. The final report should explain whether coding was manual, assisted, or automated and what human review occurred.
Image source: SurveyWorth original explanatory diagram, 2026. Illustrative, not observed survey data. Based on this article’s “Create a codebook that another person can use; Count with the correct denominator” section(s); not a product screenshot.
An actionable summary links each theme to a proposed follow-up. “Delivery uncertainty” might lead to reviewing estimate copy and testing whether shoppers understand it. “Payment error” might lead to technical logs and a usability test. The purpose of coding is to retain enough meaning to choose the right next investigation.
Keep negative cases visible
A theme is more credible when the report also notes answers that contradict it. If most comments about setup mention confusing instructions but some users say the instructions were clear and the product failed technically, preserve that distinction. Otherwise a single broad “setup problem” label could send the team toward the wrong fix. Coding should simplify the data without erasing the differences that matter for action.
Show the coded result without pretending themes are votes
Consider 40 respondents who answered an optional question about checkout friction. After removing unnecessary identifiers, two reviewers code the text using version 2 of the codebook. Twenty mention missing delivery information, 12 mention payment difficulty, and 8 mention price; some answers receive more than one code. Report “20 of 40 respondents who answered the optional item mentioned delivery information,” not “50% of all customers had a delivery problem.” If 100 people saw the item and only 40 answered, put that response pattern beside the theme table.
Image source: SurveyWorth original explanatory diagram, 2026. Illustrative, not observed survey data. Based on this article’s “Show the coded result without pretending themes are votes” section(s); not a product screenshot.
| Theme | Inclusion rule | Exclusion rule | Count among 40 answers | Next check |
|---|---|---|---|---|
| Delivery information | Timing or cost could not be found before payment. | Dissatisfaction with an actual late delivery. | 20 | Review checkout page and session data. |
| Payment difficulty | Entry, validation, or payment-method failure. | Price objection without a payment error. | 12 | Inspect error logs and test the form. |
| Price | Total cost or fee judged too high. | Shipping date confusion alone. | 8 | Compare price display and alternatives. |
Counts are illustrative and may overlap. Include one carefully anonymized quote per major theme and a negative case where possible. If reviewers disagree, refine a definition and recode earlier answers instead of quietly choosing the preferred interpretation. For example, “the shipping cost was too high” belongs under price, while “I could not find the shipping cost until the last screen” belongs under missing delivery information. A comment can carry both codes only when it makes both points. Record those boundary decisions in a memo and have the second reviewer check the affected earlier responses.
Choose tools for your next survey.
Explore survey creation software and research tools for your project.
Explore survey softwareKeep reading.
How to Test a Marketing Campaign With Surveys
Use surveys before, during, and after a campaign to test message clarity and track audience response without overclaiming causality.
Read the guide
How to Measure Brand Awareness With a Survey
Measure unaided recall and aided recognition with balanced survey questions and careful sampling.
Read the guide
How to Design a Product Testing Survey
Write product concept, prototype, and post-launch survey questions that fit what participants have actually experienced.
Read the guide