A SaaS company adds FAQ schema to forty product pages, waits three weeks, and citations in ChatGPT answers double. The team credits the schema. But nothing else about the site’s structured data, publishing cadence, or PR activity stayed still during those three weeks either. Without a controlled test, that win is a guess wearing a lab coat. Proving that a single AEO tactic raised citation likelihood means isolating that one variable against a stable prompt set, instead of watching a single metric climb and assuming why.
Key Takeaways
- A before-and-after comparison cannot separate your tactic from model updates, competitor activity, or seasonal shifts happening at the same time.
- A real test needs a fixed prompt set, a control group of pages that receive no change, and a clear rule for what counts as a citation versus a mention.
- Citation likelihood compounds, so a test that runs too short will underestimate a tactic that is actually working.
Why Before-and-After Comparisons Fail
The instinct is simple: ship the tactic, check citations a month later, compare. The problem is that AI models change on their own schedule. ChatGPT pulls from across the open web and that web keeps publishing. Gemini cross-references Search, YouTube, and Scholar, any of which can shift independently of anything you did. A citation count going up after you added FAQ schema does not tell you the schema caused it.
The same applies to a named citation from a trade publication. Trust is a graph of agreement across independent sources, not a single score that moves in a straight line when one input changes. A brand can pick up a citation from another article, a competitor can lose one, or a model provider can quietly retrain, all in the same window you are measuring. A before-and-after comparison bundles all of that into one number and calls it a result.
Choosing a Prompt Set That Reflects Real Buyer Questions
The test needs a fixed list of prompts that will not change for the duration of the experiment. Pull them from the questions your sales and support teams actually hear, not from keyword tools built for search. A finance brand testing FAQ schema on its lending pages might use twenty prompts covering eligibility, rates, and process, phrased the way a first-time buyer would type them, not the way an SEO brief would.
Size matters more than people expect. Ten prompts run once will not separate signal from noise; the sample is too small to tell whether one citation shift is a pattern or a coincidence. Twenty to forty prompts, run on a fixed schedule across the same set of models, gives enough data points to see a trend rather than a blip. Keep the wording identical each time you run them. A rephrased prompt is a different test.
Structuring a Proper Control Group
Split the pages or content pieces the tactic could apply to into two groups that are otherwise comparable: similar topic, similar existing authority, similar publish date. Apply the tactic, FAQ schema, a named citation, structured data, to the test group only. Leave the control group untouched. Run the same prompt set against both groups on the same schedule, so any shift in the wider AI layer affects both equally and cancels out in the comparison.
This is the same discipline behind any measurement Answerburst runs for clients: track consistently, across multiple engines, over a long enough window to trust the trend rather than a single week’s reading.
growth in AI brand presence for NewDay USA, tracked weekly across five engines over six months.See the case study
Defining What Counts as a Citation
Before the first prompt runs, agree on what a positive result actually looks like. Share of Citation is how often a model pulls your brand into an answer and names it as the source. Share of Voice is how often you are mentioned anywhere on the web, and it can sit high while your Share of Citation sits near zero. A test that conflates the two will show movement that was never really there.
A citation is not a mention with better luck; it is the model naming your brand as its source.
Log three things for every prompt run: whether your brand appears in the answer at all, whether it is named as a cited source, and what position it holds if the model lists sources inline, the way Perplexity does. Those three numbers, tracked separately, are what let you tell a real lift from background noise.
How Long to Run It, and What Can Quietly Ruin the Result
Citations compound. Once a model cites a brand for a given query, the next citation on a related query becomes more likely, because the model is drawing on a pattern of agreement rather than a single page. That means a test cut short will understate a tactic that works and needs time to build on itself. Four to six weeks is a reasonable floor for a first read; six months is closer to what a genuine trend requires.
Change one variable at a time. If FAQ schema goes live in the same week your PR team lands a named citation in a trade publication, you have run two experiments and can credit neither one cleanly. Keep a simple log of every other change touching the test pages, content edits, new backlinks, site migrations, so that if the result looks odd later, you can check what else moved.
What to Test Next
Pick one tactic and one prompt set, and commit to the full run before touching anything else on those pages. If FAQ schema is the hypothesis, apply it to the test group this week, freeze the control group, and set a recurring weekly prompt run across at least three models, since a tactic that lifts ChatGPT citations does not always move Claude or Perplexity the same way.
Write down your hypothesis and your definition of success before you look at a single result. That single discipline is what separates a test from a story you tell yourself after the fact. Once the run is done, keep the control group in place and test the next tactic against it, rather than starting from a blank baseline each time.
FAQs
What Is a Controlled Test for an AEO Tactic?
It is a method that applies one change, such as FAQ schema or a named citation, to a test group of pages while an otherwise comparable control group receives no change, then measures both against a fixed prompt set on the same schedule so external shifts affect both groups equally.
How Many Prompts Do You Need for a Reliable Result?
Twenty to forty prompts run on a fixed, unchanging schedule across the same models is generally enough to see a pattern rather than a single anomaly. Fewer than ten prompts run once will not separate a real shift from ordinary noise.
Can You Test FAQ Schema and a New Citation at the Same Time?
Not if you want a clean answer. Running two tactics on the same pages in the same window means any citation lift could belong to either one, so the result cannot tell you which tactic to invest in further.
How Is Citation Likelihood Different From Search Ranking?
Search ranking measures position in a list of links. Citation likelihood measures whether an AI model names your brand as the source inside a generated answer, which is a separate outcome and does not always move in the same direction as ranking.
How Often Should the Test Run Again?
Run the prompt set on a fixed cadence, weekly is common, for the full duration of the test, since citations compound over time and a short window will understate a tactic that is genuinely working.
