Methodology & sources for AI search visibility

Evaluates source attribution in LLM deep research agents (such as ChatGPT, Perplexity, and Gemini deep research modes). Finds that cited URLs are frequently invalid, broken, or unrelated to the claim being attributed. The paper introduces a parser and evaluation framework to measure attribution validity at scale, exposing systematic citation-quality gaps across vendors.

Google AI Overviews appeared on 13.7% of queries overall and 64.7% of question-form queries. Politically sensitive topics saw lower rates. AI Overviews cite domains more credible on average than co-displayed first-page results, but nearly 30% of cited domains do not appear in those results at all. 11% of 98,020 atomic claims were unsupported by the cited pages, with omission the dominant failure mode. Half of cited pages carry display advertising.

As of April 2026, ChatGPT holds 76.85% of worldwide AI chatbot market share, followed by Google Gemini at 9%, Perplexity at 7.73%, Microsoft Copilot at 3.76%, Claude at 2.66%, and DeepSeek at 0.01%.

ChatGPT cites around 7 sources per answer; Perplexity and Google AI Overviews cite more. But pages cited by ChatGPT have a much higher average influence on the answer's wording and evidence. Influence rises with page length, structure, and the density of definitions, statistics, comparisons, and step-by-step procedures.

McKinsey estimates up to 1 trillion dollars in US B2C retail revenue from agentic commerce by 2030. 37% of consumers cite in-stock reliability, speed, and intuitive navigation as a top driver. More than 40% of Gen Z and millennials say experiential retail makes them more likely to shop a retailer. The top decile of retailers is expected to capture more than 85% of sector economic profit.

Organisational AI adoption reached 88% and four in five university students now use generative AI. Generative AI reached 53% population adoption within three years, faster than the PC or the internet. The estimated value of generative AI tools to US consumers reached 172 billion dollars annually by early 2026. Documented AI incidents rose to 362 in 2025, up from 233 in 2024.

Argues that single-snapshot AI visibility measurement understates true brand presence in generative search. Proposes a longitudinal measurement framework that captures variation across runs, prompts, and platforms, demonstrating that any one-time snapshot of citation rate or mention rate can swing materially across repeated queries. Stochasticity itself is a measurement parameter, not noise to discard.

Empirical study of reference hallucinations in commercial LLMs and deep research agents finds that fabricated citations and incorrect attributions occur frequently across vendors. Proposes detection and correction methods that can be applied at inference time to reduce hallucinated references without retraining the underlying model. Open-source benchmark released for reproducibility.

Ofcom's strategic approach sets out how the UK communications regulator will assess AI risks across online safety, broadcasting, telecoms, and post in 2025 to 2026. AI is already shaping how UK adults find information online, with generative AI tools used by a significant minority of adults each week, rising fastest among younger age groups. (agent inferred)

Empirical comparison of Google Search, Gemini, and Google AI Overviews finds that AI Overviews and Gemini converge on a small set of authoritative sources, while traditional Google Search returns more diverse results. AI surfaces also rewrite or paraphrase source content rather than reproducing it verbatim, making click-through behaviour and citation attribution measurably different from classical SERPs.

Generative AI search systems systematically narrow the range of sources users encounter compared with traditional search. Across controlled experiments, participants exposed to AI-generated answers were shown fewer distinct domains and fewer perspectives on the same query than participants using ranked-link search results. The effect compounds with repeated use, reducing source diversity over a session.

Content Credentials, the open provenance standard for verifying how a piece of media was made, moved from specification to consumer reality in 2025. The Content Authenticity Initiative passed 6,000 members. The Google Pixel 10 and Sony PXW-Z300 video camera ship with Content Credentials. A C2PA conformance program, the CAWG 1.2 specification, and developer education at learn.contentauthenticity.org now back the ecosystem.

6.12 billion people use the internet in April 2026, nearly three-quarters of the world's population. 81.2% of online adults used at least one form of AI in the past month, an estimated 4.02 billion people. Roughly 60% of those, about 2.42 billion, use standalone generative AI platforms such as ChatGPT, Gemini, and Doubao. ChatGPT alone has around 1.15 billion monthly active users.

A structural-engineering framework called GEO-SFE separates content structure into three layers: document architecture, information chunking and visual emphasis. Applied to the same underlying text, the framework lifts citation rates in generative engines by 17.3% on average and subjective answer quality by 18.5% across six mainstream AI search engines. The semantic content itself is preserved; only structure changes.

AgenticGEO proposes a self-evolving agentic system for generative engine optimization where multiple agents iteratively rewrite source content based on observed AI citation outcomes, then update their rewriting strategy across rounds. The system outperforms static GEO heuristics on benchmark queries, showing that adaptive, multi-round optimization produces higher citation lift than fixed transformations.

AgentGEO diagnoses citation failures in generative engine optimization by simulating the multi-step retrieval-and-generation process, identifying which step caused a candidate source to be missed, then proposing a targeted repair. Empirical tests across GEO benchmarks show that step-targeted repairs outperform end-to-end rewriting strategies for boosting citation rate.

Adding Schema.org JSON-LD to plain HTML produced only modest retrieval gains. An enhanced entity-page format combining structured data with rich internal linking and navigational affordances delivered a 29.6% accuracy improvement for standard retrieval-augmented generation and 29.8% for the full agentic pipeline with multi-hop link traversal.

Akamai's SOTI report (April 2026) finds AI bot activity surged 300% in 2025, with the media industry ranking second globally at 13% of AI bot traffic. Publishing organisations accounted for 40% of media-targeted AI bot activity. AI training crawlers made up 63% of AI bots targeting media; AI fetchers were 24%. OpenAI generated the highest volume of AI bot traffic against media, with publishing taking 40% of OpenAI's media requests.

Use of Claude.ai diversified between November 2025 and February 2026: the top 10 tasks fell from 24% to 19% of traffic. 49% of US jobs have seen at least a quarter of their tasks performed using Claude. The average estimated hourly wage of tasks on Claude.ai fell from 49.30 dollars to 47.90 dollars. Users with at least 6 months of experience have a 10% higher success rate in conversations.

After Google rolled out AI Overviews, pageviews to Wikipedia articles whose topics frequently trigger AI summaries fell measurably more than pageviews to comparable control articles. The estimated traffic loss attributable to AI summaries is in the range of single-digit to low double-digit percentages on affected article sets. (agent inferred)

67% of senior marketing leaders expect a high level of AI-driven disruption to their vertical's consumer journey, and nearly all expect some disruption. Travel, retail, and news are most exposed (high disruption risk plus weak customer relationships), while financial services, fintech, and media or streaming are most protected.

AI assistants recommend an average of 6 to 11 brands per prompt depending on the category. Established market leaders dominate AI answers in some sectors but are absent in others. Sectors where AI search is shifting brand consideration the fastest include cosmetics, consumer electronics, and financial services. Reddit and Wikipedia are the most-cited third-party sources.

Anthropic operates three distinct web crawlers with separate robots.txt user agents: ClaudeBot collects content for training foundation models, Claude-User fetches pages on demand when users ask Claude a question, and Claude-SearchBot indexes content for Claude's search features. Each can be allowed or blocked independently, letting site owners opt out of training while still appearing in Claude's search answers. All three respect robots.txt and support the non-standard Crawl-delay directive.

Across controlled experiments comparing AI search engines (ChatGPT, Perplexity, Google AI Overviews) with traditional search, AI search significantly reduces clicks to source publishers and concentrates attention on a smaller set of authoritative domains. Users exposed to AI summaries form more confident but less accurate beliefs on contested topics. (agent inferred)

SAGEO Arena introduces a realistic environment for evaluating search-augmented generative engine optimization, simulating the full pipeline from query through retrieval to answer generation. Empirical tests across published GEO methods show that arena-based evaluation reveals failures that simpler benchmarks miss, particularly under realistic source-distribution drift and adversarial competition.

Large-scale analysis of citation validity across LLM outputs finds that a meaningful share of citations are either fabricated (no real source exists at the cited URL), misattributed (real source but unrelated to the claim), or hallucinated (made-up author/title combinations). Citation validity varies substantially across vendors and is worst for niche or recent topics.

CORE introduces a method for controlling which sources appear in generative engine answers by intervening on the retrieval step, allowing search providers to enforce ranking constraints (such as freshness or authority) in LLM-based search. Empirical tests show CORE meaningfully shifts cited-source distributions without degrading answer quality on benchmark queries.

Individual images lack the words and authority signals that generative search rewards, so visual platforms risk being skipped over while users get their answer in the chat. Pinterest's response is to predict what users would search for from each image, group images into theme pages, and link them with authority signals. The live system added 20% organic traffic growth.

OpenScholar is an open retrieval-augmented language model designed to synthesise scientific literature. On a benchmark of expert-annotated questions, OpenScholar matches or exceeds the citation accuracy of much larger commercial systems while being fully open-source. The paper releases both the model and the benchmark for reproducibility.

Bing Webmaster Tools added an AI Performance dashboard in public preview on February 10, 2026. It shows total citations of a publisher's site across Microsoft Copilot, AI-generated summaries in Bing, and partner integrations, plus average cited pages per day, page-level citation counts, and grounding query phrases. Publishers can use the data to see which pages are referenced in AI answers and how that activity changes over time.

Mobile visits to leading AI assistants reached 54.3 million unique visitors in December 2025, up 107% year over year. Desktop visits hit 83.0 million, up 18%. ChatGPT led mobile at 34.5 million (up 84%) and desktop at 56.4 million (up 83%). Gemini grew 137% on mobile and 648% on desktop. Microsoft Copilot more than tripled on mobile (up 246%). Perplexity rose 265% on mobile.

Google was responsible for 73.7% of all US desktop searches across the 41 domains analysed in Q4 2025. Traditional search engines accounted for about 80% of search activity, commerce sites about 10%, social networks 5.5%, and AI tools 3.2%. Amazon, Bing, and YouTube each saw more desktop search activity than ChatGPT. In 2025, Google lost 3.5 points of share.

IETF draft 'draft-ietf-aipref-attach' defines how AI usage preferences expressed by content publishers can be attached to HTTP responses, complementing the AIPREF vocabulary draft (draft-ietf-aipref-vocab). The document specifies HTTP header syntax, machine-readable attachment formats, and conflict-resolution rules when preferences are signalled at multiple levels (server, file, response).

The IETF AIPREF working group is developing a standard vocabulary for websites to express how their content can be used by AI systems. The current draft defines two usage categories, train-ai and search, each of which can be marked allow, disallow, or unspecified. A site might publish train-ai=y, search=n to permit AI training while disallowing search indexing. The format is designed to plug into robots.txt and other carriers.

When users named a product, ChatGPT recognised it 99.4% of the time and Perplexity 94.3%. When they asked discovery questions like best AI tools launched this year, success collapsed to 3.32% and 8.29%. Generative-engine-optimisation scores did not predict discovery. Referring domains, Product Hunt ranking, and Reddit presence did, suggesting traditional SEO foundations carry over to AI visibility.

Fastly's Q3 2025 Threat Insights Report covers AI bot traffic patterns observed on Fastly's network of 130,000+ apps and APIs. The report distinguishes AI crawlers from AI fetchers, examines bot verification challenges, and provides regional breakdowns of AI bot composition. Detailed quarterly metrics on volume, vendor share, and industry vertical impact are reported in the PDF.

AI bots account for around 20% of all verified bot traffic on the Cloudflare network, with crawling for training purposes the largest single use. AI crawling activity for real-time user actions grew roughly 15 times year on year. The most active AI crawlers in 2025 were Meta-ExternalAgent, GPTBot, and ClaudeBot.

Empirically compares source coverage and citation bias between LLM-based search engines and traditional search. Finds that LLM-based search systematically over-represents large, English-language, US-based sources and under-represents smaller and non-English content compared with what traditional search returns for the same queries. Bias is consistent across the major LLM search providers tested.

In the December 9, 2025 update, OpenAI's bot documentation removed the previous claim that OAI-SearchBot feeds navigational links into ChatGPT answers and dropped any reference to OAI-SearchBot supplying training data. ChatGPT-User was expanded to explicitly cover Custom GPT requests and GPT Actions, and robots.txt is no longer applied to user-initiated ChatGPT-User actions. OpenAI also confirmed OAI-SearchBot and GPTBot share crawl results to avoid duplicate fetching.

Quantifies stochasticity in agentic LLM evaluations using intraclass correlation coefficients (ICC). Shows that single-run evaluations of agentic systems are unreliable because run-to-run variance is large relative to the gap between system variants. Recommends a minimum of 5 to 10 repeated runs per evaluation and reports the ICCs for several common agentic benchmarks.

30% to 45% of US consumers say they already use generative AI for product research and comparison. AI now accounts for up to a quarter of referral traffic at some retailers, though still less than 1% of their total traffic. Consumers say they trust retailer-owned agents three times more than third-party agents, but about half are uncomfortable letting AI run an end-to-end transaction.

McKinsey estimates that AI agents could unlock up to 1 trillion dollars in US B2C retail revenue by 2030 as consumers delegate routine shopping to agents and merchants adapt to agent-mediated discovery, comparison, and checkout. (agent inferred)

Akamai's 2025 AI Botnet Report finds AI bot activity surged 300% year-over-year and AI bots now compose nearly 1% of total bot traffic on Akamai's platform. The commerce industry saw more than 25 billion bot requests in July-August 2025. In healthcare, over 90% of AI bot triggers were associated with scraping. North America accounted for 54.9% of AI bot activity in the period.

Across 15 common product-page rewriting tactics tested on a shopping benchmark, no single hand-crafted heuristic reliably wins. A simple iterative prompt-optimisation routine outperforms all of them. The optimised prompts converge on the same pattern across categories, pointing to a stable, domain-agnostic recipe for making product listings more visible to conversational shopping agents.

Click-through rates fell sharply in the year after Google introduced AI-generated summaries, with declines reaching 30% in some B2B categories including B2B software. 85% of B2B buyers purchase from their day-one list, the vendors they had in mind before searching, leaving brands less able to influence shortlists through smart search strategies.

AI search visits in Europe grew from 4% of organic visits in early 2024 to 8% in early 2025, and are projected to reach 25% by the end of 2026 and overtake organic in 2028. LLM referral traffic to leading European retailers is up more than 2,000% in fashion, nearly 1,200% in luxury, and almost 7,500% in specialty retail.

Deloitte forecasts that in 2026, about 29% of adults in developed markets will run at least one search per day returning a generative AI summary, versus 10% using a standalone generative AI app daily. Daily passive AI use is projected to stay about three times standalone use through 2027. By mid-2026, 72% of adults will have generated a search overview versus 61% who used a standalone tool.

Only about one in three consumers say generative AI rivals search engines for finding information, with most still preferring traditional search for general queries. GenAI tools see higher use for creative, brainstorming, and writing tasks than for product research or factual lookup. (agent inferred)

Google is testing Web Bot Auth, a cryptographic protocol that lets bots sign HTTP requests so sites can verify their identity beyond user-agent strings or IP address ranges. During the experimental phase, only some Google AI agents sign requests, and signatures use HTTP Message Signatures (RFC 9421) keyed to https://agent.bot.goog. Google recommends continued reliance on reverse DNS and published IP ranges as a fallback.

IETF Internet-Draft 'draft-meunier-web-bot-auth-architecture' defines an architecture for HTTP message signatures applied to automated bot traffic. The architecture supports cryptographic verification of bot identity, allowing sites to confirm whether a self-identified Googlebot or GPTBot is genuinely from the claimed vendor. Aimed at replacing reverse-DNS verification as the standard mechanism for bot authentication.

OG-RAG grounds retrieval-augmented generation in ontologies rather than free-text documents, retrieving structured concepts and their relationships to provide more precise context to the LLM. On benchmark QA tasks, OG-RAG outperforms standard text-RAG by reducing irrelevant retrieval and improving answer specificity, with the largest gains on multi-hop questions requiring structured reasoning.

Argues that traditional retrieval evaluation metrics (recall, MRR) underestimate the value of retrieval in LLM-based pipelines because LLMs can compensate for partial retrieval through their pre-existing knowledge. Proposes new metrics that measure retrieval value conditional on the LLM's downstream behaviour, finding that some 'high-recall' retrievers are actually worse for LLM-based search.

Defines citation failure as a measurable phenomenon in RAG systems where retrieved documents are not cited even when they support the answer. Introduces CITECONTROL, a method to detect and mitigate citation failure that improves citation recall without degrading answer quality. The method is lightweight and integrates with standard RAG pipelines.

OpenAI launched ChatGPT Atlas, an AI-native web browser that embeds ChatGPT directly into browsing, summarises pages, answers questions in the sidebar, and can carry out multi-step tasks on the user's behalf such as filling forms, comparing products, and completing purchases. (agent inferred)

McKinsey projects that AI-powered search will mediate roughly $750 billion in US consumer revenue by 2028, representing a meaningful share of category-level discovery. Brands that win in AI answers tend to combine strong third-party coverage, structured product information, and active management of their entity presence across the open web.

Evaluates how reliably chat assistants ground answers in the web search results they cite. Tests show that even when models retrieve from credible sources, their summaries frequently include claims not supported by the retrieved passages. Citation alone does not imply groundedness, and the gap is largest for nuanced or contested topics.

AutoGEO is a framework that extracts the preferences generative search engines apply when picking and rewriting content for AI answers. The researchers turn those preferences into rewriting rules, then test them on the GEO-Bench benchmark plus two new benchmarks built from real user queries. Both the prompt-based AutoGEO API and the trained AutoGEO Mini model raise content traction in AI answers while preserving search utility.

Generative search and traditional web search return different things even for the same query. Generative engines pull from a broader pool of sources than Google web search, mix in varying amounts of internal model knowledge versus retrieved pages, and surface different concept sets. That widens the set of pages that can earn visibility, but also breaks assumptions baked into classical ranked-list evaluation.

Examines whether misinformation websites are more permissive to AI crawlers than mainstream sites by analysing robots.txt directives across thousands of domains. Finds that low-credibility and misinformation domains block AI crawlers significantly less often than high-credibility news sources, meaning AI training data is systematically biased toward less reliable material at the source-access stage.

Treating retrieved passages as isolated chunks throws away signal that the original document layout carries. A router that navigates a document's structure tree, scoring both passage relevance and its position in the hierarchy, sets a new state of the art on multi-document question answering. Headings, section order, and parent-child relationships are themselves a ranking signal.

Across six countries, only about 7% of adults say they use ChatGPT or another generative AI tool to access news in a typical week. Trust in AI for news is low: most respondents say AI-generated news content makes them feel uncomfortable, and only a minority think AI will improve journalism. (agent inferred)

The peer-reviewed SOCC 2025 paper 'Rethinking Web Cache Design for the AI Era' by Yazhuo Zhang and colleagues (ETH Zurich) shows that traditional web caches are not designed to absorb high-diversity, low-reuse AI scraper traffic. Read the Docs reported 73TB of HTML scraped in one month; Wikimedia reported a 50% backend bandwidth increase from AI scrapers. Proposes filter-and-tier cache architectures.

DataDome's 2025 Global Bot Security Report finds that LLM crawler traffic quadrupled across DataDome's customer base in 2025, rising from 2.6% of verified bot traffic in January to over 10.1% by August. DataDome detected nearly 1.7 billion OpenAI crawler requests in a single month. AI bot traffic targeted high-value endpoints: 64% reached forms, 23% login pages, 5% checkout flows. Only 2.8% of sites were fully protected.

W3C TDM Reservation Protocol Community Group develops a standardised mechanism for content owners in the EU to reserve their rights against text-and-data-mining uses under Article 4 of the EU Copyright Directive. The protocol specifies machine-readable opt-out signals (TDMRep) that AI training crawlers should honour, complementing robots.txt with a copyright-specific reservation mechanism.

Proposes sub-sentence-level citations in RAG outputs, where each cited passage is matched to a specific sub-sentence in the generated answer rather than to the answer as a whole. The approach improves attribution precision and reduces over-citation, where models cite a source for an entire sentence even when only part of it is supported.

Cloudflare introduced a Content Signals Policy that extends robots.txt with three new signals: search (indexing for traditional search), ai-input (use in retrieval augmented generation or AI answers), and ai-train (use for training or fine-tuning AI models). Each can be set to yes or no, or left blank. For 3.8 million domains already using Cloudflare's managed robots.txt, the company will publish search=yes, ai-train=no by default.

Most retrieval benchmarks cannot tell a good chunking strategy from a bad one because the answers can be found in any reasonable split of the text. A new benchmark built on evidence-dense questions shows that chunking choices visibly change end-to-end answer quality, and that a hierarchical, multi-level chunker improves performance without paying a heavy time cost.

ChatGPT reached around 700 million weekly active users by mid-2025, with roughly 18 billion messages sent per week. About 30% of conversations are work-related while 70% are personal, covering writing assistance, information seeking, and tutoring. Adoption is rising fastest in lower-income countries. (agent inferred)

Three on-page properties showed the strongest association with whether a page got cited by AI answer engines: metadata and freshness, semantic HTML markup, and structured data. Pages that scored at least 0.70 on the GEO-16 quality score and met at least 12 of 16 quality pillars were cited at substantially higher rates than pages that did not.

Generative search engines weaken the link between ranking and visibility, so source articles need new ways to prove they shape AI answers. The benchmark scores creator influence across five dimensions: exposure (does the article surface), faithful credit (is it cited), causal impact (does it move the wording), readability and structure, and trustworthiness and safety.

The US District Court for DC barred Google from exclusive distribution contracts for Google Search, Chrome, Google Assistant, and the Gemini app. Google must share certain search index and user-interaction data with rivals and offer search and ad syndication services. The court extended remedies to generative AI products to prevent the same tactics being used to monopolise GenAI. Google holds roughly 90% of US search queries.

Amazon documents Amazonbot as the web crawler used to power its services. Amazonbot identifies itself with the user-agent string 'Amazonbot' and respects robots.txt directives. The documentation page lists IP ranges, robots.txt examples, and Amazon's contact information for site owners reporting issues. Amazonbot does not currently train Alexa or general-purpose AI models per the published documentation.

More than half of consumers expect to use AI assistants for shopping by the end of 2025. US retail traffic from generative AI browsers and chat services grew 4,700% year over year in July 2025. These visitors are more engaged, spending 32% more time on site, browsing 10% more pages, and bouncing 27% less. By 2029, US AI search ad spend is projected to reach 26 billion dollars.

Brave documents that its search crawler does not advertise a differentiated user agent to avoid being discriminated against by websites that allow only Googlebot. However, if a domain or page is not crawlable by Googlebot, Brave's bot will not crawl it either. The documentation notes that robots.txt is not used to prevent Brave-specific access but applies through the Googlebot directive.

DuckDuckGo documents DuckAssistBot as the crawler used to power DuckDuckGo's AI-assisted answer features. DuckAssistBot is related to DuckDuckGo Search but operates as a separate bot with its own user-agent string and IP ranges. The help page lists how site owners can identify, allow, or block DuckAssistBot via robots.txt directives and clarifies that DuckAssistBot fetches pages on demand rather than for AI training.

Google relies on around 16,000 external Search Quality Raters across 80-plus languages to evaluate search results against published guidelines. Raters never decide rankings directly; they assess Page Quality (using the E-E-A-T framework of Experience, Expertise, Authoritativeness, Trust) and Needs Met. Standards are highest for Your Money or Your Life topics like health, finance and safety, where low-quality pages can cause real harm.

Kagi documents KagiBot as the web crawler for the Kagi search engine. KagiBot identifies as 'Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot)' and originates from four declared IP addresses with reverse-DNS confirmations at kagibot.org. Standard robots.txt directives targeting Kagibot are respected. Kagi is a paid search engine that does not train AI models on crawled content.

The Data Provenance Initiative at MIT Media Lab audited the provenance, licensing, and attribution of over 1,800 text dataset collections used to train large language models. The audit found that more than 70% of datasets had 'unspecified' licenses; correcting the licensing reduced this to around 30%. The corrected licenses were often more restrictive than those originally assigned by repositories.

Meta documents multiple crawlers and their user agents: meta-externalagent for AI training and product improvement, meta-externalfetcher for user-initiated content fetches by Meta AI features, and the older facebookexternalhit for generating preview cards when links are shared. Publishers can use robots.txt to control meta-externalagent and meta-externalfetcher; facebookexternalhit follows different rules because it acts on a user's direct request to share a link.

Mistral AI documents its crawler identification and robots.txt compliance policy. The documentation lists user-agent strings used by Mistral's crawlers (for web indexing, real-time retrieval, and model training), specifies how site owners can target each via robots.txt, and states Mistral's commitment to honour disallow directives. Standard format matching other major AI vendor crawler documentation.

Publicly accessible summary of OpenAI's ChatGPT usage research. Describes the Asking/Doing/Expressing classification (49%/40%/11%) and the dominant consumer use cases: practical guidance, information seeking, and writing assistance. Useful as a citable first-party summary of the underlying research, but, like the underlying paper, it provides aggregate categories rather than a browsable feed of real user prompts.

OpenAI's research on ChatGPT usage classifies consumer conversations into three categories: roughly 49% 'Asking' (information seeking), 40% 'Doing' (practical task assistance), and 11% 'Expressing' (writing and creative work). This provides a defensible intent taxonomy for classifying prompt hypotheses — but it is an aggregate breakdown, not a browsable dump of user prompts.

AI crawlers read content far more than they send referrals back. Anthropic's ClaudeBot crawled around 70,000 pages for every visitor it referred; OpenAI's GPTBot crawled around 1,700 for every visitor; Perplexity around 5 for every visitor. Mistral was the only major AI engine where referrals outweighed crawl volume.

AI crawler traffic is concentrated in news, technology, finance, and retail. The largest category is training crawling, but real-time user-action crawling (where an AI assistant fetches a page during a user conversation) is the fastest-growing segment. Different AI engines crawl with different mixes of purpose, which has direct implications for which crawlers a brand should allow.

Visitors arriving at US retail sites from generative AI sources show measurably higher engagement than visitors from other channels: 8% higher time on site, 12% more pages per visit, and 23% lower bounce rate. AI-driven retail traffic grew sharply through 2024 and 2025, though it remains a small share of total visits.

Surveys the field of evidence-based text generation with LLMs, organising work into three categories: attribution (linking generated text to evidence), citation (formatting and presenting that evidence to users), and quotation (verbatim grounding). Identifies four open problems including attribution granularity, retrieval-attribution coupling, and evaluation of partial attribution.

Across April-July 2025, Fastly observed that AI crawlers made up almost 80% of AI bot traffic on its network (the other 20% being AI fetcher bots). Meta's AI crawlers alone generated 52% of crawler traffic, more than Google (23%) and OpenAI (20%) combined. Nearly 90% of North American AI bot traffic came from crawlers vs 41% in Europe. Fetcher bot bursts reached 39,000 requests per minute.

Generative search engines reward content that anticipates the different roles a user might be playing when they ask a question. Rewriting a page through several informational personas, then refining it, produced larger gains in both subjective impression and measured presence inside generative answers than approaches that optimise on a single axis.

Users prefer concise, source-attributed answers over verbose explanations from AI search. Longer, more lexically complex responses produced an uncanny-valley effect: systems sounded authoritative but lacked critical thinking, lowering trust and raising cognitive load. The pattern challenges the assumption that more elaborate AI output equals better output. (agent inferred)

Cloudflare observed Perplexity using undeclared crawlers to fetch content from sites that had blocked its known bots in robots.txt and WAF rules. When PerplexityBot was blocked, traffic rotated through a generic Chrome-on-macOS user agent, undisclosed IP ranges, and multiple ASNs, hitting tens of thousands of domains and millions of requests a day. Cloudflare de-listed Perplexity from its verified bots list and added detection rules.

Bing identifies several active crawlers. Bingbot is the main web crawler for Bing search. AdIdxBot crawls pages for Bing Ads. BingPreview generates page snapshots. MicrosoftPreview supports preview cards. Copilot uses Bingbot data for grounding rather than a separate crawler. Publishers can issue user-agent specific rules in robots.txt, and Bing respects standard directives and meta robots tags for indexing control.

Argues that the citation behaviour of LLM-based search constitutes an attribution crisis: cited sources are systematically under-credited (fewer click-throughs than equivalent SERP positions), over-extracted (more content reproduced verbatim or near-verbatim), and concentrated on a small subset of high-authority publishers. Quantifies the ecosystem-level economic impact on publishers.

When a Google search result page includes an AI summary, users click on a traditional link in roughly 8% of visits. On result pages without an AI summary, they click in roughly 15% of visits. Users rarely click on the citations inside the AI summary itself, doing so on about 1% of visits.

Identifies attribution bias in generator-aware RAG: when an LLM is told which documents it can cite, the model preferentially attributes claims to documents that align with its own pre-existing beliefs, ignoring contradicting sources even when those contradict the model's output. The bias is measurable and persists across model families.

Shows that a RAG system's answers can be factually correct while its citations are unfaithful, meaning the cited passages do not actually support the generated claim. Across standard benchmarks, correctness and faithfulness diverge measurably, implying that citation-quality evaluation must be a separate metric from answer accuracy in any AI visibility tracking system.

Generative information-retrieval systems that return a written, cited answer instead of a ranked list of links can reduce a searcher's effort and improve their experience without lowering perceived credibility. The comparison covers conversational answer interfaces against traditional ranked-list search, suggesting brands should expect users to do less link-clicking and rely more on what the answer itself says.

The EU General-Purpose AI Code of Practice provides a voluntary route for AI model providers to demonstrate compliance with the AI Act's obligations on copyright, transparency, and safety. The Copyright Chapter requires signatories to honour machine-readable opt-out signals such as robots.txt and TDM reservations, to publish a summary of training data, and to put a complaint mechanism in place for rightsholders.

Perplexity launched Comet, an AI-native web browser built around Perplexity's answer engine. Comet replaces the search bar with a conversational assistant, summarises pages, answers questions about open tabs, and can execute agentic tasks across the web on behalf of the user. (agent inferred)

SIGIR 2025 LiveRAG Challenge Report summarises a community competition where teams built end-to-end live RAG systems evaluated on real-time queries. Reports best-performing strategies, common failure modes, and lessons learned. Notable findings include the dominance of hybrid sparse-dense retrieval and the difficulty of evaluating live RAG without ground-truth answers.

AI search systems concentrate news citations in a small set of outlets, and the cited mix leans politically liberal. Low-credibility sources are rarely cited. News makes up only 9% of all citations across more than 366,000 citations studied, so brands depending on press coverage for AI visibility face a narrow set of gatekeeper publishers, with limited influence from political leaning or quality on user satisfaction.

Rewriting web copy to add credible citations, statistical evidence, and cleaner phrasing measurably increases how much of that copy gets reproduced in AI answers. Optimised travel pages saw a 15.63% rise in absolute word count surfaced inside generative responses and a 30.96% rise on a position-weighted version of the same metric, with small computational cost.

Cloudflare launched Pay Per Crawl in private beta on July 1, 2025, letting publishers charge AI crawlers per request. The system uses HTTP status code 402 Payment Required: a crawler either includes a crawler-max-price header to pre-agree to pay, or gets a 402 with the price and can retry with crawler-exact-price. Cloudflare acts as the merchant of record, identifies crawlers via Web Bot Auth signed requests, and aggregates payments.

Across Cloudflare's network, search and AI crawler traffic rose 18% from May 2024 to May 2025. Googlebot grew 96% in raw requests and now accounts for 50% of crawler traffic. GPTBot rose 305% in requests, with its share climbing from 2.2% to 7.7%. ChatGPT-User requests jumped 2,825%, and PerplexityBot grew 157,490% off a tiny base. About 14% of top domains now use robots.txt rules targeting AI bots specifically.

LongCite enables LLMs to generate fine-grained citations in long-context QA by training the model to attribute each statement in its answer to a specific span in the retrieved long document. The method substantially improves citation precision over post-hoc citation generation and outperforms baselines on the released LongCite benchmark.

Argues that current AI search engines' promise of factual, verifiable, source-cited responses is partly illusory. Empirical analysis across major systems shows frequent unsupported claims even when citations are present, weak ranking of citations by relevance, and citation patterns that systematically advantage well-resourced incumbents. Calls for new evaluation standards before mass deployment.

How a page is split into chunks matters as much for retrieval as which model embeds it. Simple recursive token chunking around 100 tokens with no overlap (R100-0) consistently beat more elaborate strategies. Retrieval-tuned embedding models such as Nomic and Intfloat E5 outperformed domain-specialised ones like SciBERT, suggesting embedding choice and chunk size are the high-leverage levers.

Apple uses one crawler, Applebot, to gather data that powers Spotlight, Siri, and Safari search. A separate user agent, Applebot-Extended, governs whether content is used to train Apple's foundation models, including Apple Intelligence. Sites can disallow Applebot-Extended in robots.txt to opt out of generative AI training while keeping content discoverable in Apple search. Applebot is identified via reverse DNS at applebot.apple.com or a published IP CIDR list.

Metacognitive prompts that explicitly ask the AI to evaluate its own reasoning before answering reduce errors in generative search responses. Users supplied with such prompts also show improved critical evaluation of AI-generated answers compared with users supplied with standard prompts. The intervention is light-touch and does not require changes to the underlying model.

Chunk size in retrieval-augmented generation has a large effect on retrieval quality. Smaller chunks of 64 to 128 tokens are optimal when answers are short and fact-based. Larger chunks of 512 to 1024 tokens work better when broader context is needed. Embedding models react differently: Stella benefits from larger chunks for long-range retrieval, while Snowflake performs better with smaller chunks for entity-level matching.

Audit of major AI scrapers' compliance with robots.txt directives finds that compliance is selective rather than uniform: scrapers honour blocks on some user agents and not others, even within the same vendor, and compliance changes over time as crawlers are updated. The paper provides specific evidence of non-compliance events with named vendors and dates.

ReliableEval proposes a method-of-moments recipe for stochastic LLM evaluation that explicitly accounts for run-to-run variance in model outputs. Across standard benchmarks, the method produces tighter confidence intervals than naive averaging and reveals that some headline LLM performance comparisons are within noise margins. Released as an open evaluation toolkit.

Microsoft launched NLWeb on May 19, 2025, an open project that lets any website expose its content as a natural-language interface using Schema.org, RSS, and other structured data the site already publishes. Every NLWeb instance is also a Model Context Protocol server, making the site discoverable to AI agents. Initial adopters include Shopify, Tripadvisor, Eventbrite, O'Reilly, Hearst, and Chicago Public Media.

Argues that generative AI search has broken the feedback loop that traditionally improved ranking. Traditional web search collects fine-grained user feedback (clicks, dwell time) at the document level; generative AI search receives only coarse-grained feedback on the final answer, even though the pipeline spans query decomposition, retrieval, and generation. Proposes NExT-Search to reintroduce process-level feedback.

C2PA Technical Specification v2.2 defines a standard for cryptographically signed content credentials. The specification was published in 2024 as ISO/IEC 22144, enabling images, video, and other media to carry tamper-evident metadata about origin, edits, and AI involvement. Adopters include Adobe, Microsoft, the BBC, OpenAI, Sony, and Leica, with rollout in cameras, generative tools, and publisher workflows.

The US Copyright Office concludes that generative AI training raises copyright questions at several points: data collection, model training, retrieval-augmented generation, and outputs. Fair use is fact-specific and depends on transformativeness, commerciality, the amount used, and effects on the market for the original work, including market dilution and lost licensing. The Office recommends voluntary licensing markets rather than compulsory licensing schemes.

Introduces the concept of sufficient context as a new evaluation lens for RAG systems. A retrieval is sufficient if it contains enough evidence to answer the query correctly; insufficient retrievals lead to hallucinated answers even when the model has the right reasoning ability. Provides metrics and empirical analysis across standard RAG benchmarks.

Across ~12,000 queries in seven countries and a preregistered randomised experiment on a US sample, participants trusted GenAI search less than traditional search on average, but adding reference links and citations to GenAI answers significantly increased trust, even when those citations were incorrect or hallucinated. Uncertainty highlighting reduced trust whether confidence was high or low.

Judge Sidney Stein denied OpenAI's and Microsoft's motions to dismiss the core contributory copyright infringement and most DMCA claims brought by The New York Times, the Daily News, and the Center for Investigative Reporting. The court allowed the publishers' direct infringement and trademark dilution claims to proceed. It dismissed common law misappropriation and certain DMCA 1202 sub-claims without prejudice. The case moves to discovery.

Imperva's 2025 Bad Bot Report finds that automated traffic overtook human activity for the first time in a decade, reaching 51% of all internet traffic in 2024. Bad bots specifically account for 37% of internet traffic, up from 32% the year prior. 44% of advanced bot traffic targeted APIs. AI is supercharging bot sophistication, with simple high-volume attacks now 45% of all bot attacks.

Bandwidth used by automated crawlers on Wikimedia Commons grew 50% between January 2024 and early 2025, driven primarily by AI training scrapers fetching multimedia at scale. Wikimedia Foundation engineers found that 65% of the most expensive requests on the projects came from bots, even though bots accounted for only about 35% of pageviews, because crawlers hit pages that miss the site's caching layer.

AI Labyrinth is an opt-in Cloudflare feature that serves AI-generated decoy pages to bots that ignore no-crawl directives. A misbehaving crawler follows hidden links into a maze of plausibly written but irrelevant content, wasting compute and exposing itself as a bot. Pages are pre-generated with Workers AI, sanitised against XSS, and stored in R2. AI crawlers send more than 50 billion requests a day across Cloudflare.

Visits to US retail websites originating from generative AI sources grew by roughly 1,200% between July 2024 and February 2025. AI-sourced visitors browsed 12% more pages per session and bounced 23% less than visitors from other channels. AI referrals still represented a small share of total retail traffic, but the per-visit engagement quality was meaningfully higher.

Across eight AI search engines tested, more than 60% of news-attribution queries received incorrect answers. Perplexity got 37% wrong; Grok 3 got 94% wrong. Premium paid models were no more accurate than free ones, and often produced confidently incorrect answers without flagging uncertainty. Several engines retrieved content from publishers that had explicitly blocked their crawlers.

OpenAI's web-search API documentation states that the web_search_call output item will usually (but not always) include the search queries that were searched, and that the sources field can reveal all URLs consulted during the search run. This is first-party proof that some query rewrites can be observed for requests under the caller's control — but the 'usually but not always' caveat means observed fan-out is partial rather than exhaustive.

Roughly 80% of consumers now rely on zero-click results, AI summaries, or assistant answers for at least 40% of their search needs, and AI search use has reduced average organic click-through rates by 15% to 25%. The shift compresses the funnel: brands need to be present and credible in the answer itself, not on the click destination.

Google states there are no extra technical requirements for appearing in AI Overviews or AI Mode beyond being indexed and eligible for a standard search snippet. SEO fundamentals apply: allow crawling in robots.txt, maintain internal linking, keep pages findable. Google describes the query fan-out technique, where the system issues multiple related searches across subtopics, and reports that clicks from AI Overview pages tend to be higher quality.

OECD report 'Intellectual Property Issues in AI Trained on Scraped Data' (AI Paper No. 33, February 2025) examines copyright, trademark, trade secret, and database protection challenges raised by AI training data scraping. The report recommends voluntary codes of conduct based on transparency in the data chain, requiring AI developers to disclose data sources and preserve metadata enabling rightsholders to track unauthorised use.

The FTC's January 2025 staff report on AI partnerships finds that the three largest US cloud providers (Alphabet, Amazon, Microsoft) have used their investments in Anthropic and OpenAI to lock in cloud spend, gain equity and revenue-sharing rights, and access sensitive technical and business information. The Commission flags risks to switching costs, input access for rivals, and competition for engineering talent and compute.

On Vercel's network in late 2024, GPTBot generated 569 million monthly requests and Anthropic's Claude generated 370 million, together roughly 20% of Googlebot's 4.5 billion. None of the major AI crawlers (OpenAI, Anthropic, Meta, ByteDance, Perplexity) executed JavaScript. ChatGPT spent 34.82% of fetches on 404 pages, Claude 34.16%, versus 8.22% for Googlebot. Server-side rendered content is far more visible to AI crawlers.

Empirical comparison of long-context LLMs (where retrieved content is dropped into the model's input window) against retrieval-augmented generation (where retrieval is iterative) finds that long-context approaches underperform RAG when the relevant evidence is buried in noise. RAG still wins for most production information-retrieval tasks despite advances in long-context models.

Google Trends Explore uses Gemini to take an area of interest and expand it into up to eight related search terms, additional ideas, and top/rising queries. This is direct first-party proof that public search behaviour can be expanded from a seed topic into an adjacent-intent neighbourhood — but Google explicitly frames it as web-search demand, not chatbot-prompt demand.

OpenAI documents that ChatGPT Search rewrites a user query into one or more targeted queries and may send additional, more specific queries after reviewing initial results. This is first-party proof that query fan-out behaviour is real inside a production chatbot search system.

Google publishes a list of common crawlers covering Googlebot, Googlebot-Image, Googlebot-Video, Googlebot-News, AdsBot variants and the Google-Extended user-agent token used for Gemini and Vertex AI training opt-out. Each row in the documentation gives the user-agent string seen in HTTP requests, the matching robots.txt token, the IP ranges in common-crawlers.json and the products affected when a site changes its crawl preferences for that agent.

Adding citations, quotations, and statistics to content can increase its visibility in AI-generated answers by up to 41% on average. Pages ranked outside the top of traditional search saw the largest gains. The effect varies by content domain and by AI engine, but the lift from evidence-style content elements is consistent across the conditions tested.

Sebastian Farquhar and colleagues at the University of Oxford propose using semantic entropy to detect hallucinations in LLMs. The method measures uncertainty in the meaning of model outputs across sampled generations rather than uncertainty in token probabilities. Outperforms prior hallucination-detection baselines on standard QA benchmarks and generalises across model families.

Comprehensive survey of retrieval-augmented generation (RAG) covering its history, core components (retrieval, augmentation, generation), evaluation methods, and open challenges. The survey organises RAG variants into a taxonomy and traces the field's evolution from naive retrieval to modular and agentic RAG architectures. Widely cited as the field's canonical reference.

Perplexity documents two user agents controllable via robots.txt: PerplexityBot indexes content for retrieval in Perplexity answers, while Perplexity-User fetches pages on demand when a user submits a query. Pages disallowed for PerplexityBot are not indexed in full, though Perplexity may still display the domain, headline and a brief factual summary. IP ranges are published at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. (agent inferred)

Google's FAQPage structured data documentation announces that as of May 7, 2026, FAQ rich results no longer appear in Google Search. Support in the rich-result report and Rich Results Test ends in June 2026, and Search Console API support is removed in August 2026. While the feature is being deprecated, FAQ markup itself remains valid Schema.org and is still used by AI engines that read structured data.

Google announced an update to its Search Quality Rater Guidelines, adding a second E to E-A-T to create E-E-A-T: Experience, Expertise, Authoritativeness, and Trustworthiness. Experience asks whether content reflects first-hand or life experience with the subject. Trust is positioned as the most important of the four, and the others support it. The guidelines instruct human raters who evaluate search quality.