Who Reads the Answers?

Three thousand open-ended responses, a button that promises insight, and the difference between analysis you can audit and analysis you can only believe.

The pile

Every researcher who has asked an open-ended question knows the moment. The survey closes, the structured data tallies itself into charts, and then there is the pile: three thousand answers to "What would you change about the program?" Each one written by a person who took the question seriously, or didn't; each one potentially holding the detail that explains the numbers. The traditional options were stark — spend weeks reading and coding, hire someone to do it, or quietly let the text data go unanalyzed. Studies of survey practice suggest the third option has always been more common than anyone admits.

Into that moment, every analysis tool now offers a button. Press it, and the pile becomes themes, sentiment scores, an executive summary. The button is not lying about the labour it saves. The question is what else it does.

What the button does when nobody is watching

Two failure modes are now well documented. The first is fabrication: large language models asked to summarize a body of open-ended responses will, some fraction of the time, produce themes that are not present in the data — fluent, plausible, and traceable to nothing. The second is quieter and, for anyone doing community research, worse: homogenization. Summarization pulls toward the majority narrative. The forty responses that describe a different experience from the other two thousand nine hundred and sixty — the language-access barrier, the rural transportation problem, the reason the program failed for one demographic — are precisely what a compression process compresses away. Community health research, program evaluation, and community-based research exist in large part to hear minority experience. A summarizer optimized for the gist is structurally biased against the reason the question was asked.

There is a respondent-side mirror to this, worth one sentence here: research published in 2025 found that roughly a third of participants on a major research panel admitted using AI tools to help write open-ended answers, and that the resulting text is measurably more homogeneous and more positive than what people write themselves. Machine-flattening can now happen at both ends of the pipe. (Who is actually answering is its own subject, and the previous article in this series.)

The deeper problem is epistemic: an AI summary reads the same whether or not it is faithful. Fluency has never been evidence, but it has never looked so much like evidence as it does now.

The profession has noticed

What distinguishes 2025–26 from the years of general AI anxiety before it is that the professional bodies this field actually answers to have all, within about eighteen months, put guidance on paper.

The American Association for Public Opinion Research released its task-force report on responsible AI integration in survey research in May 2026, and made the subject permanent with a standing AI committee. The ICC/ESOMAR International Code — the ethics baseline for market and social research across ninety-plus countries — was revised in 2025 to name human oversight and disclosure of methods and limitations as explicit obligations. In evaluation, the UK Evaluation Society published good-practice guidelines built on transparency, human control, risk management, and quality assurance; the Canadian Journal of Program Evaluation has published both a pitfalls analysis and a structured ethical decision framework for AI use in evaluation; and the Canadian Evaluation Society now requires entrants to its own awards to declare how AI was used in their work.

Read together, the guidance converges on something specific. None of these bodies prohibits AI in analysis. None of them treats it as just another tool, either. The line they draw, in different vocabularies, is the same line: the machine may do labour; it may not do unexamined judgment. The researcher must be able to say what the machine was instructed to do, verify what it did, and disclose both.

An older answer than it appears

The demand for examinable analysis did not arrive with AI. Qualitative research settled this question decades ago, for humans. When a team codes three thousand open-ended responses, methodology requires a codebook — the explicit list of categories and the written instructions for applying them — and it requires that coding be auditable, typically by having a second coder apply the same codebook and measuring agreement. Nobody accepts "I read them all and here's what I found" from a research assistant without wanting to see the codebook.

That frame sorts the AI question cleanly. A language model applying a researcher-written codebook to responses, one at a time, with its outputs checked against the allowed categories, is a coder — enormously fast, imperfect in measurable ways, working from instructions that can be read, criticized, and revised. A language model asked to "find the insights" is not a coder. It is an uncredentialed analyst whose reasoning cannot be examined, applying a codebook nobody wrote.

The distinction that matters, in other words, is not AI versus no AI. It is confirmatory versus exploratory. Scaling a classification the researcher designed is confirmatory work, and machines are legitimately good at it. Discovering what nobody anticipated is exploratory work — the place where fabricated themes live, where majority-pull does its damage, and where reading the actual words of actual respondents remains the method. The professional guidance above is, at bottom, a demand that tools and researchers stop blurring the two.

What this looks like built into a tool

The approach we take at Septenar follows from that distinction, and it is easiest to describe as a set of structural commitments rather than features.

Classification is rubric-driven, and the rubric belongs to the researcher. When AI scoring is enabled on a text question, the builder chooses the rubric type, defines the categories, and writes — or reviews and edits, if the survey was AI-drafted — the classification instruction itself. Nothing about how a response will be read is hidden: a dedicated review surface shows every scored question's categories and instructions before publishing, and the rubric freezes with the published form, so every scored response remains auditable against the exact instruction that scored it. A rubric can incorporate context from elsewhere in the same submission — a response to "what goes through your mind at the grocery store?" can be classified in light of whether that participant reported food insecurity — which is the kind of conditional reading a human coder does naturally and a context-free sentiment engine cannot.

The outputs are constrained the way a coder's are. A classification that falls outside the rubric's allowed categories is rejected and logged, not invented into the dataset. Scored results land beside the structured data as countable values — traceable to a question, a rubric, and an instruction — not as a paragraph of unattributable narrative. Each scored response appears on the bill as its own line item, because analysis that costs something should be visible as a decision, not ambient.

Two boundaries mark where the machine is not invited. Validated instruments — PHQ-9, GAD-7, and the rest of the Library — are scored deterministically, by arithmetic, never by a model; a screening score that depends on a language model's mood is not a screening score. And exploratory theme discovery, the genuinely hard problem, exists on the platform as an explicitly experimental capability, labelled as such and held apart from the confirmatory path — because the honest state of the art is that machine-discovered themes cannot yet be audited the way rubric-applied ones can, and a tool should not blur that line for marketing reasons.

One more commitment is invisible but load-bearing: the models themselves are interchangeable. Analysis runs in Canada, and the specific model behind the scoring tier can change — will change, as better options become available — without any of the above changing. The guardrails are the architecture: researcher-authored instructions in, category-constrained outputs out, deterministic scoring where determinism is owed. A safety property that depends on a particular model's good behaviour is not a property of the tool.

What it still cannot do

A rubric finds what it was told to look for. If respondents raise language access and the categories are cost, scheduling, and staffing, language access lands in other — or worse, gets misfiled — and no volume of scored responses will surface it. The correction is not a better model; it is the researcher reading fifty or a hundred responses before writing the categories, and revisiting the rubric when other grows suspicious. For response volumes in the low hundreds, the honest advice remains: read them; it is faster than it looks and you will notice what no classifier is looking for. And when the analysis is disclosed — as the AAPOR, ESOMAR, and CES guidance now converges on expecting — the disclosure is only as good as what can be shown. A methods section can state exactly what an auditable rubric instructed; it can state nothing verifiable about a button.

The pile of three thousand answers is real, and the labour it represents was never noble in itself — nobody's research improved because a human spent three weeks doing what a written codebook could direct. What cannot be delegated is the codebook. Who reads the answers matters less than it used to. Who decides what the reading means has not changed at all.

Sources

AAPOR Task Force. Responsible AI Integration in Survey Research. May 2026.
ICC/ESOMAR. International Code on Market, Opinion and Social Research and Data Analytics, 5th edition, 2025 revision.
Beyond the Hype: Uncovering Pitfalls in the Use of AI in Evaluation. Canadian Journal of Program Evaluation, 2025–26; and the AIDEM ethical decision framework, same journal.
UK Evaluation Society. AI in Evaluation: Good Practice Guidelines.
Zhang, S., Xu, R., Alvero, A. on LLM-assisted open-ended responses (homogenization and positivity effects). Sociological Methods & Research, 2025.
Canadian Evaluation Society, SEEK Award AI-use declaration requirement.
CGIAR Independent Advisory and Evaluation Service. Technical note on AI in evaluation, June 2025.

Septenar is a Canadian survey platform. AI-assisted analysis on the platform is rubric-driven and researcher-reviewed; validated instruments are scored deterministically, never by a model; and all analysis runs in Canada. This article is the third in a series: Protecting Participant Privacy in Survey Research covers who your data is protected from, and Who Is Answering Your Survey? covers who it comes from.