AI Summary
Crawl budget is how many URLs Googlebot crawls in a period, and it matters most for news publishers because fresh stories must be indexed before their value fades. The fix is to point crawl budget at new articles through sitemaps and internal links while starving waste from tag pages, archives, pagination, and faceted URLs.
- Crawl budget combines the crawl rate limit with the crawl demand you earn.
- News feels it more due to freshness curves and huge archive inventories.
- Speed indexing with a current news sitemap and homepage internal links.
- Prune tag pages, archives, pagination, and faceted URLs that leak budget.
The linked piece by Barry Adams frames crawl budget for the one context where it matters most acutely: news publishing. For an ecommerce store a slow crawl is an inconvenience, but for a newsroom it can mean a story is indexed hours after a competitor already ranked, which in news is the difference between the traffic and none of it. This article summarises why crawl budget bites harder for publishers and turns it into the fixes a news SEO can action.
What crawl budget actually is
Crawl budget is the product of two things Google balances. The crawl rate limit is how many simultaneous requests Googlebot will make without straining your server, driven by site speed and error rates. Crawl demand is how much Google wants to crawl you, driven by popularity and how often your content changes. A healthy, fast, frequently updated news site earns high crawl demand, but that budget still has to be spent wisely, because Googlebot will not crawl an unlimited number of URLs per visit.
Why publishers feel it more
News has a brutal freshness curve. A breaking story is most valuable in its first minutes and hours, so the gap between publishing and indexing is everything. Publishers also tend to run enormous URL inventories: years of archives, tag taxonomies, author pages, and section pagination that can balloon into millions of low value URLs. When Googlebot burns its visits on those, it has less capacity left for the fresh articles that actually need fast indexing. The goal is to point crawl budget at new content and starve the waste.
Where crawl budget leaks on news sites
The usual culprits are predictable. Tag pages multiply without limit when editors add loose tags to every article. Date based and author archives generate thin, near duplicate listings with little search demand. Deep pagination on section fronts sends Googlebot crawling page 40 of a feed no reader will ever open. Faceted and filtered URLs, session ids, and tracking parameters spawn endless variants of the same content. Each of these quietly consumes the budget you want spent on today's reporting.
Getting fresh articles crawled fast
Speed to index comes down to discovery and signals. Maintain a Google News sitemap that lists articles published in the last 48 hours, and keep your standard XML sitemaps current with accurate lastmod values. Link new articles prominently from the homepage and section fronts, because internal links from high crawl pages are how Googlebot finds new URLs quickly. Keep the server fast so the crawl rate limit stays high. For confirmation, use log file analysis to see exactly which URLs Googlebot is spending its visits on, then prune or block the waste.
# Count Googlebot hits by URL section from an access log
grep -i googlebot access.log | awk '{print $7}' | cut -d/ -f2 | sort | uniq -c | sort -rn | head
# In robots.txt, stop crawl waste on low value facets
User-agent: *
Disallow: /*?filter=
Disallow: /tag/What has changed since this article
The fundamentals hold, but a few things are worth noting. Google News no longer requires a separate submission or a dedicated News sitemap for eligibility, since articles can surface in the News tab through normal crawling, yet a fresh news sitemap still speeds discovery and remains best practice for high volume publishers. Bing and other engines support IndexNow, which lets you ping engines the moment an article publishes or changes, cutting the delay between publish and crawl. And Google now exposes richer crawl data in the Search Console Crawl Stats report, so you can monitor host status and response codes without always reaching for raw logs.
Crawl waste sources and their fixes
| Crawl waste source | Why it hurts | Practical fix |
|---|---|---|
| Tag pages | Multiply endlessly, thin content | Limit tags, noindex low value tag pages |
| Date and author archives | Near duplicate, little demand | noindex or consolidate the useful ones |
| Deep pagination | Wastes visits on unread pages | Cap depth, strengthen section links |
| Faceted URL parameters | Spawn infinite variants | Disallow in robots.txt, canonical to base |
| Session and tracking ids | Duplicate every URL | Strip parameters, use clean canonicals |
| Slow templates | Lower the crawl rate limit | Speed up server and reduce errors |
Related reading
- Crawl budget explained in depth
- How index bloat wastes crawl budget
- Log file analysis FAQ for tracking Googlebot
- Fixing discovered currently not indexed
Frequently asked questions
What is crawl budget for a news site?
Crawl budget is how many URLs Googlebot will crawl on your site in a given period, set by the crawl rate limit your server can handle and the crawl demand your content earns. For news publishers it matters because fresh stories must be discovered and indexed fast, before the news value fades.
Why do news publishers care more about crawl budget?
Because news has a steep freshness curve and publishers run huge URL inventories. A breaking story is most valuable in its first hours, so any delay in indexing costs traffic, and years of archives, tag pages, and pagination can waste the budget that should go to today's articles.
How do I get new articles indexed faster?
Keep a current news and XML sitemap with accurate timestamps, link new articles from the homepage and section fronts so Googlebot finds them quickly, and keep the server fast to protect the crawl rate. Where supported, use IndexNow to ping engines the moment you publish.
Should I noindex tag and archive pages?
Often yes. Tag pages, date archives, and author archives tend to be thin and near duplicate with little search demand, so they leak crawl budget. Noindex the low value ones, keep any that genuinely attract search traffic, and be deliberate about how many tags editors apply.
How can I see what Googlebot is crawling?
Log file analysis is the most direct method: filter your server access logs for Googlebot and group the hits by URL section to see where visits go. The Search Console Crawl Stats report complements this with host status, response codes, and crawl volume over time.
Does faceted navigation hurt crawl budget?
Yes. Filters and facets can generate a near infinite number of parameterised URLs that all serve similar content, and Googlebot can waste a large share of its visits on them. Disallow the parameter patterns in robots.txt and point canonicals at the clean base URL to contain the sprawl.
Source: https://www.seoforjournalism.com/p/crawl-budget-barry-adams-seo
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
