AI visibility tracking data isn’t entirely reliable. Because generative models often produce different responses, the citation shares and rankings on your dashboard are merely snapshots of a continuously changing target, not fixed facts.
A difference between you and a competitor could be genuine or just fluctuation between measurements. A new IQRush paper due for release next week (we had pre-release access) provides a method to distinguish these, showing that no fixed amount of data can definitively settle the question.
The paper is by Ron Sielinski, who co-founded IQRush, who sell software that measures AI visibility the way the paper argues you should. The reason it’s worth your time is that a separate team published a similar repeated-measurement finding in April, so IQRush is not the only one making this case.
How Much These Numbers Move
Repeatedly querying SearchGPT, Gemini, or Perplexity with the same question can produce different sources each time. They’re built to add some randomness to each response, so each citation is just one of many possible URLs it could have pulled. A prior paper by the same author explored this variability, showing that, for example, when testing SearchGPT on running gear, Tom’s Guide made up about 9.5% of citations, while Runner’s World accounted for roughly 6.0%. On the dashboard, Tom’s Guide appeared more often, but the large margin of error meant the figures overlapped. With only one sample, it wasn’t accurate to say Tom’s Guide outperformed Runner’s World, as the 3.5-point difference was within the margin of error. The new paper aims to prevent this mistake by addressing a simple yet often overlooked question: How much data is needed before rankings are truly meaningful?
When A Ranking Is Worth Trusting
The answer has two parts, and both need to be true for a ranking to be reliable. First, the order must stop changing.
In the beginning, rankings may change frequently as new answers are added because no site has a clear edge yet. It’s only after enough answers are collected that the top sites start to stand out clearly, allowing the order to stabilize. Also, it’s important that the top sites are well apart; if they’re very close, the ranking might not be meaningful, as a tight competition doesn’t really show who’s truly ahead. The paper looks at whether the difference between the top sites is bigger than the margin of error for each. When it is, the ranking reflects a real difference. When it isn’t, it’s probably just statistical noise. Both conditions need to be true at the same time, neither alone is enough. In 30 platform-topic tests, the number of answers needed for both conditions to be met ranged from 33 to 94, counting only answers with citations.
Three out of 30 didn’t reach this point even after 125 questions, all on SearchGPT, where top sites were too similar to tell apart. There is no single cutoff applicable everywhere; what works for one platform and topic may not suit another.
We’ve Been Circling This
In January, I discussed SparkToro’s discovery that AI tools give a different list of recommended brands more than 99% of the time you ask the same question. That article left one question unanswered: how many times do you need to ask before the results stabilize? This paper offers the clearest answer I’ve come across.
Rand Fishkin, who led that study, shares some helpful advice. Before spending any money on tracking AI visibility, he suggests making sure your provider “shows their math.” The IQRush paper is a great way to do this because it provides a simple stopping rule, so you don’t have to rely solely on intuition about how many runs are enough.
It also fits a run of studies SEJ has covered over the past year, each reporting AI citation numbers as if they were fixed. This one turns around, examines the measurement itself, and asks whether those numbers are stable enough to compare in the first place.
What This Changes For Your Reporting
The number on your dashboard is just a single sample. Before trusting it, check whether your tracker performs the same check repeatedly and reports a range, or if it pulls data once and shows a clean figure. The clean figure can actually be a warning sign, not reassurance.
A gain after a content change is easy to misinterpret. For example, a three-point increase in your SearchGPT citation share might seem like proof that your effort paid off, but such a change can fall within the natural variability of successive runs, according to the original paper’s data.
To claim the win, measure before and after more than once each. A single before-and-after reading cannot separate your change from ordinary noise.
The platform you are measuring changes how much data you need, and not in the way you would guess. It comes down to how much independent information each answer carries, not how many citations it hands you. Gemini piles citations onto the same handful of sites within a single answer, so many of those citations tell you the same thing. SearchGPT gives fewer citations per answer but spreads them out, so each answer carries more independent information than the raw count suggests. The same number of answers on two engines does not buy the same confidence, and a budget that settles Gemini can leave you guessing on SearchGPT.
Sometimes the honest answer is that you cannot say yet. Three of the 30 tests never cleanly separated their top sites within the budget. For those, the right call is to hold, not to publish a ranking the data cannot support. A tracker that can tell you “not enough data” is worth more than one that prints a confident order every time you ask.
The top of the ranking is the part you can most defend. With enough answers, the leaders pull away from the middle and tail, though even they are not exact. The margins of error widen fast below the front, until neighboring positions are a coin flip, and even the top 10 were not spotless, with the typical margin of error on a top-10 site running about five positions and one in five wider than 10. Trust the leaders, treat the middle and bottom as rough, and do not report exact positions past the front of the list.
What The Paper Doesn’t Prove
None of this comes from a finished, peer-reviewed study. It is a preprint built on 30 platform-topic tests across three engines, using questions generated by ChatGPT rather than real user searches, over a single stretch of collection. The exact numbers will not transfer cleanly to your topics, so treat them as the shape of the problem, not a lookup table.
Those counts include only answers that carried citations, which matters most on SearchGPT, because a share of its questions return no citations at all. In one topic, 125 questions produced 104 usable answers, a 17% miss, so you would need to submit more questions than those totals suggest.
The check on the method is internal, too. The paper compares a ranking it calls early against that same collection’s final ranking, not against any outside ground truth. That tests whether the stopping rule is consistent with itself, which is why the matching result from the unaffiliated team does real work here. The authors of that April paper, Julius Schulte, Malte Bleeker, and Philipp Kaufmann, are researchers at the University of St. Gallen. They ran a separate dataset and reached the same verdict, that a single reading is unreliable and you have to sample an engine repeatedly to trust what it tells you.
Where This Goes
The paper stops short of the thing most people will want, which is a way to know your run budget before you start collecting. Sielinski leaves that for later work and notes that the number depends on the shape of each platform’s citation pattern, so a single universal budget probably is not coming.
The bigger change is that AI visibility reporting is headed the way ad and analytics reporting already went, toward numbers that carry a margin of error instead of a false decimal point. That is happening while the basic plumbing is still missing, since Search Console still won’t tell you which clicks came from AI. Until it does, the job falls on you to run the check more than once and report the range, not the single number your dashboard hands you.
More Resources
Featured Image: Stokkete/Shuttertstock