PerplexityBot Blocked

Element Code: IN-058

What this check means

This flag says your robots.txt contains a rule that disallows Perplexity's crawler. Perplexity is an AI answer engine: people ask it a question and it replies with a synthesized answer plus citations to sources. To cite you, it has to be able to fetch and read your pages. A disallow rule slams that door.

Perplexity actually runs two distinct agents, and this matters when you decide what to do:

• PerplexityBot is the indexing crawler. It systematically fetches pages to build the index Perplexity draws from. Its declared user agent contains PerplexityBot and points to +https://perplexity.ai/perplexitybot. This is the one most robots rules target.
• Perplexity-User is the on demand fetcher. When a live user's question requires visiting a specific page in real time, this agent goes and gets it. Per Perplexity's own crawler docs, this user driven fetch is not treated the same as bulk crawling.

According to Perplexity's help documentation, if you disallow PerplexityBot it will not index your page's full text, though it may still show the domain, a headline, and a short factual summary. So blocking does not guarantee total invisibility, but it does cut off the deep access.

Why you might care, and why you might not

This is one of the few SEO checks where "blocked" is not automatically a problem. It is a decision.

Allow it if you want visibility in AI answers. Perplexity has real and growing usage, and being a cited source sends qualified referral traffic and builds authority with an audience that is researching, not just browsing. For most publishers, SaaS sites, and content businesses chasing reach, being citable is the whole point.

Block it if you have a deliberate stance: you do not want your content feeding AI answers, you are protecting proprietary or paywalled material, or you have a licensing position. That is legitimate. The mistake is not blocking. The mistake is blocking by accident because you pasted someone's "block all AI bots" robots.txt from a blog post and never thought about it.

How PerplexityBot fits your crawler stack

The two agents and how to target them

Note: there have been public reports, including a 2025 writeup from Cloudflare, that Perplexity fetched some content despite robots and firewall blocks. If your policy is a hard no, robots.txt alone may not be enough and you may need firewall or WAF rules plus IP verification against the JSON lists above. For most sites that simply want to be found, this is moot.

How to detect and confirm the block

1. Read your live robots.txt. Open https://yourdomain.com/robots.txt and search for PerplexityBot. Look for a block like User-agent: PerplexityBot followed by Disallow: /. Also check any User-agent: * group with a broad disallow, since that catches Perplexity too.
2. Watch for a catch all AI block. Many "block AI" snippets list a dozen agents. Confirm whether PerplexityBot and Perplexity-User are in that list.
3. Check server logs. Grep access logs for PerplexityBot. If you see it requesting robots.txt and then nothing else, your disallow is doing its job.
4. Screaming Frog robots test. Configure a custom user agent of PerplexityBot and run the robots.txt tester against sample URLs to see allowed versus blocked.

How to fix it, if you want Perplexity in

1. Locate the rule. Find the group in robots.txt that disallows PerplexityBot, whether an explicit block or a broad User-agent: * one.
2. Allow the agent explicitly. Add a dedicated group so intent is unmistakable:

   User-agent: PerplexityBot
   Allow: /
   
   User-agent: Perplexity-User
   Allow: /

   Placing an explicit Allow group ahead of restrictive wildcard groups removes ambiguity for well behaved crawlers.

3. Remove leftover disallow lines for these agents from any other group so nothing contradicts.
4. Do not accidentally reopen everything. If you had a broad User-agent: * block for a reason, keep it and only carve out the agents you want.
5. Deploy and fetch the live file to confirm the change is public, not just staged.
6. Give it time and check logs for PerplexityBot returning and crawling real pages, not just robots.txt.

Do this, not that

What good looks like

Your robots.txt reflects an intentional choice. If you want the reach, PerplexityBot and Perplexity-User are explicitly allowed, no contradictory lines linger, and your logs show the crawler fetching real content. If you want out, the block is deliberate and, for a strict policy, backed by more than robots.txt. Either way, nothing is on autopilot.

FAQ