PerplexityBot

What PerplexityBot is

PerplexityBot is the crawler Perplexity uses to build the search index behind its AI answer engine. If it can't fetch your pages, you are not eligible to be cited in Perplexity's answers, and since Perplexity links out to sources more aggressively than almost any other AI product, that eligibility is real referral traffic, not a vanity metric.

The actual user agent string

This is what shows up in your access logs when the index crawler visits:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Perplexity also operates a second, separate agent that fetches a page live when a user's question requires it:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

These two do different jobs, and robots.txt treats them differently, that split is where most misconfiguration happens. The same crawler-versus-fetcher split exists at Anthropic; see the ClaudeBot entry for how their trio compares.

robots.txt directives vs. what Perplexity actually does

The compliance dispute you should know about

In 2025, Cloudflare published research accusing Perplexity of crawling with undeclared user agents to route around blocks; Perplexity disputed the findings. You don't have to pick a side to take the practical lesson: never trust a user agent string alone. Verify by IP. Perplexity publishes its official IP ranges as JSON, linked from its bot documentation pages, precisely so you can separate the real bot from scrapers wearing its name.

A sane starting policy

Most sites that want AI-answer visibility can start here and adjust from logs:

User-agent: PerplexityBot
Allow: /
Disallow: /search/
Disallow: /cart/

Translation: index the content, stay out of the utility pages that waste crawl and can't be cited anyway. If you run paid content, scope the Disallow to the paywalled paths rather than blocking sitewide, a partial block preserves citation eligibility for the pages you actually want marketed. And whatever policy you pick, write it down somewhere humans will find it, because the next person to edit robots.txt will not read your mind.

How to check PerplexityBot activity on your site

1. Grep your access logs: grep -i "PerplexityBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn, this gives you requesting IPs by volume.
2. Validate those IPs against Perplexity's published IP list. Anything claiming to be PerplexityBot from outside those ranges is an impostor.
3. Check the status codes you're serving it. A wall of 403s usually means your WAF or bot-management layer is blocking it even though your robots.txt allows it.
4. Fetch your own robots.txt and confirm which group actually applies, named group beats wildcard.
5. In analytics, segment referrals from perplexity.ai. That's the payoff side of the ledger: citations turning into visits.

Common mistakes and fixes

• Blocking with the wildcard group and assuming it covers Perplexity-User. It doesn't, user-initiated fetches skip robots.txt. Fix: enforce at the WAF/CDN if blocking is genuinely the goal.
• Allowing the bot in robots.txt while your firewall 403s it. The two layers disagree and the stricter one wins. Fix: audit bot-management rules; several vendors block AI crawlers by default now.
• Relying on Crawl-delay. It's not part of the standard and support is inconsistent everywhere. Fix: server-side rate limiting.
• Blocking PerplexityBot, then asking why competitors get cited and you don't. Index eligibility is the entry ticket. Fix: decide deliberately, per the tradeoffs in the GPTBot vs ClaudeBot vs PerplexityBot comparison.
• Judging compliance from UA strings in logs. Scrapers spoof popular bot names constantly. Fix: IP verification before you accuse anyone of ignoring your rules.

FAQ

Does blocking PerplexityBot remove my existing citations?

Not instantly. Indexed pages age out rather than vanishing the moment you block. And Perplexity-User can still fetch a URL live when a user asks about it specifically, so your content can surface in answers even with the index crawler blocked.

Is PerplexityBot collecting training data for an LLM?

Perplexity positions its crawler as building a retrieval index for its answer engine, not as harvesting foundation-model training data. If that distinction matters to your legal or licensing posture, verify against their current documentation, this area shifts fast.

How often does PerplexityBot crawl?

There's no published schedule. In practice, crawl frequency tracks how often your content changes and how discoverable it is. Fresh, frequently-linked pages get revisited more.

Can PerplexityBot render JavaScript?

Assume no. Like most AI crawlers, it works from server-delivered HTML, so client-side-rendered content is effectively invisible to it, the SSR vs CSR breakdown covers why this matters more for AI bots than for Googlebot.

Should I allow it or block it?

If you want Perplexity citations and the referral clicks that come with them, allow it. If you're pursuing content licensing or have contractual reasons to restrict AI access, block it, but do it at both the robots.txt and firewall layers, and know exactly what you're giving up.