Perplexity

What Perplexity is

Perplexity is an AI answer engine that searches its own web index for every query and writes a cited answer on top of the retrieved pages. For site owners that design choice is the whole game: unlike a chatbot answering from memory, Perplexity has to point at sources, and every answer is a set of citation slots your pages can win or lose.

That also means Perplexity visibility is earned at crawl time and retrieval time, not at training time. If its crawler can’t reach you, no amount of brand fame in the model’s weights puts you in the citations.

How Perplexity gets your content

Two user agents matter, and they do different jobs:

• PerplexityBot: the index crawler. It collects public pages into Perplexity’s own search index, ahead of any query. Robots.txt token: PerplexityBot.
• Perplexity-User: the on-demand fetcher. It grabs a specific page while a human is asking about it. Perplexity’s own documentation says these user-driven fetches generally don’t follow robots.txt, because the request comes from a person, not a scheduled crawl.

When a query comes in, Perplexity searches the index, pulls the strongest candidate pages, has the model synthesize an answer from them, and attaches numbered citations. Getting cited therefore has a pipeline, and each stage has a lever you control.

The retrieval pipeline and your levers

Robots.txt for Perplexity

The permissive setup, which is what you want if citations and referral traffic are the goal:

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

And the opt-out, if you’ve decided you don’t want to be in the index at all:

User-agent: PerplexityBot
Disallow: /

Remember the asymmetry: disallowing PerplexityBot pulls you from the crawled index, but Perplexity-User may still fetch a URL a user explicitly brings into a thread.

How to check it

1. Grep your access logs for both agents:

   grep -iE "perplexitybot|perplexity-user" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

   This shows which URLs Perplexity actually reads, often not the ones you’d guess.

2. Audit robots.txt for a PerplexityBot disallow, and check whether a blanket User-agent: * disallow is catching it by inheritance.
3. Check your WAF. Hits with status 403 or a challenge page mean your CDN is blocking it even though robots.txt allows it.
4. Filter analytics for referrer perplexity.ai and watch the trend after fixes.
5. Sample real queries. Ask Perplexity the questions your money pages answer and note whether you appear in the citations, and at which position.

Common mistakes

• The one-click “block AI bots” toggle. Cloudflare and similar tools block PerplexityBot at the edge, then people stare at robots.txt wondering why they get zero citations. Check the WAF before you check anything else.
• Treating Perplexity like a training scraper. It’s an answer engine that links out. Blocking it costs you an acquisition channel and protects nothing that matters.
• Burying the answer. Eight hundred words of wind-up before the point means stage 3 of the pipeline picks someone who got to it faster. It’s a hell of a lot cheaper to move your answer up than to out-authority a competitor.
• No dates on content. Perplexity favors demonstrably fresh sources for time-sensitive queries; undated pages lose ties they should win.
• Expecting instant results after unblocking. The index has to recrawl you first. Give it weeks, and verify with log hits, not vibes.

FAQ

Does Perplexity actually send traffic?

Yes: citations are clickable and users who click arrive deep in research mode. Volumes are smaller than Google but conversion quality is consistently reported as high. Track it via the perplexity.ai referrer.

If I block PerplexityBot, do I disappear from answers?

From index-based retrieval, yes, once the block takes effect. A user can still pull a specific URL of yours into a conversation via Perplexity-User. There’s a dedicated check for this in PerplexityBot blocked.

How is getting cited by Perplexity different from ranking on Google?

Perplexity retrieves for a synthesized answer, so extractability, meaning clear claims, data, and structure, can beat raw domain authority. The mechanics are compared in how Perplexity, ChatGPT and Google AI pick their sources.

What should I optimize first for Perplexity citations?

Crawl access, then answer-shaped structure. The full playbook is in our Perplexity citation strategy guide, and PerplexityBot’s behavior versus other crawlers is covered in the AI crawler comparison.

Does Perplexity respect crawl budget like Googlebot?

It crawls far less politely predictable than Googlebot: bursts are common. If load is a concern, rate-limit it at the server rather than banning it outright.