LLM SEO tools, and how to tell if their numbers are real
LLM SEO tools check whether ChatGPT, Gemini, Perplexity and Google's AI answers name your business. Their numbers are only as good as the way they collect the answers.
Short answer
- The API and the real ChatGPT app give different answers to the same question. One study found only 6 to 12% overlap.
- Logged in or out, the model, and the run all change the result. One reading means little.
- Most trackers say they use the real app. None of the pages we checked say how many runs they do.
- Before you pay, ask the 8 questions below.
Updated 14 October 2026 · By Michael Nefedov, noetio
The API and the real app disagree
A tool can ask an engine in 2 ways: through the developer API, or through the app people use, like chatgpt.com. They give different answers.
6-12%
overlap between the brands the API and the app named, across ChatGPT, Gemini and Perplexity
Brandlight, June 2026, 2,565 answer pairs. Source
0
of 3,856 searches were shared between chatgpt.com and the GPT-5.2 API, over 27 days
Genezio, February 2026, UK banking questions. Source
Both studies come from companies that sell AI visibility tools.
4 things that change the answer
From our own measurements, September and October 2026. Each square is one run. Blue: the business was named.
API or app
Same Ukrainian question, 18 September 2026.
Logged in or out
Same question, same day, in the ChatGPT app.
The model
Same question through the OpenAI API, 17 September 2026. Our own site cited.
23
different domains cited in 8 runs of one ChatGPT question. The most common one: in 4 of 8.
Our test, September 2026.
The engine
One local clinic, 5 questions × 3 runs, through each API, 8 October 2026.
The samples are small, so read these as warnings, not laws.
How LLM SEO tools collect answers
In each vendor's own words, checked 14 October 2026:
| Tool | Method, in their words |
|---|---|
| Peec AI | The web app, because “API responses often differ from what users see” (docs) |
| Otterly.AI | Each platform's website; “only use API for Claude tracking” (help) |
| Profound | “From the consumer browsing experience” (site) |
| Scrunch | “A combination of browser automation and official platform APIs” (FAQ) |
What none of them say: how many runs per question, and whether the app is logged in or out. We also tried to automate the ChatGPT app ourselves and got bot checks and dropped sessions. A blocked run looks like a real “not named”.
8 questions to ask an LLM SEO tool before you pay
- 1Real app or API, and for which engine?
- 2Logged in or logged out?
- 3Which model, and do you tell me when it changes?
- 4How many runs per question?
- 5Which country and language per question?
- 6Can I see the raw answers and cited pages?
- 7Is a blocked run counted as not named?
- 8Can I add a control question?
LLM SEO: what gets you named
- Be indexed. Google says a page must be “indexed and eligible to be shown in Google Search with a snippet” to be a link in AI Overviews (Google).
- Let the bots in. Sites that block OAI-SearchBot don't show up in ChatGPT's search answers (OpenAI).
- Answer one question per page, clearly, with sources. More in SEO vs GEO.
Compass, our AI SEO agent, measures AI answers this way: API runs for the trend, and checks in the real app for the number that counts.
Questions
Yes. For AI answers that search the web, the basics are the same as SEO: the page must be indexed, crawlable by the engine's search bot, and answer the question clearly. Google says there are no extra requirements to appear in AI Overviews or AI Mode. What changes is how you measure it, because the answers move between runs.
We found no independent test of how accurate any tracker is. Every study we found comparing the API with the real app was published by a vendor. So pick by method: real app or API, logged in or out, how many runs, which country, and whether you can see the raw answers.
Yes, by hand. Open ChatGPT logged out in a temporary chat, start a new chat for every run, ask the same unbranded question 3 times and write down how many times you were named. Do the same for a control question. It is slow, but it measures what a buyer sees.
Compass does this work for you
Compass is an AI SEO agent. It does your SEO and AI search work every day, on your site and beyond it, and nothing goes live without your OK. Get early access below.