Bulk URL Testing Scripts

Checking one URL by hand is easy. Checking the fifty thousand URLs in a large site is not, and that gap is where a bulk URL testing script earns its keep. Feed it a list from a sitemap, a crawl, or a spreadsheet, and it returns the HTTP status, the final URL after any redirects, and the response time for every entry. This is the backbone of migration QA, redirect audits, broken link sweeps, and uptime spot checks. This guide covers the two concurrency models, the traps that produce wrong data, and copy ready patterns.

The simplest working version

Start with the standard requests library and a thread pool. Threads suit this job because URL testing is input and output bound: each request spends most of its time waiting on the network, so many can run at once without saturating the CPU.

import csv
import requests
from concurrent.futures import ThreadPoolExecutor

def check(u):
    try:
        r = requests.get(u, timeout=10, allow_redirects=True)
        return (u, r.status_code, r.url, round(r.elapsed.total_seconds()*1000))
    except requests.RequestException as e:
        return (u, "ERROR", str(e), 0)

urls = [line.strip() for line in open("urls.txt") if line.strip()]
with ThreadPoolExecutor(max_workers=20) as pool:
    rows = list(pool.map(check, urls))

with open("results.csv", "w", newline="") as fh:
    w = csv.writer(fh)
    w.writerow(["url", "status", "final_url", "ms"])
    w.writerows(rows)

Two details matter. allow_redirects=True makes r.url the final destination, so you capture where a 301 actually lands. r.elapsed gives you the round trip time for free. Twenty workers is a safe default; raise it for third party sites, keep it low for a server you own so you do not stress it.

The faster version with asyncio

For very large lists, asyncio with httpx scales further than threads because it multiplexes thousands of pending requests on a single event loop. A semaphore caps how many run at once so you stay polite.

import asyncio, httpx

async def check(client, sem, u):
    async with sem:
        try:
            r = await client.get(u, timeout=10, follow_redirects=True)
            return (u, r.status_code, str(r.url))
        except httpx.HTTPError as e:
            return (u, "ERROR", str(e))

async def main(urls):
    sem = asyncio.Semaphore(50)
    async with httpx.AsyncClient() as client:
        tasks = [check(client, sem, u) for u in urls]
        return await asyncio.gather(*tasks)

urls = [l.strip() for l in open("urls.txt") if l.strip()]
rows = asyncio.run(main(urls))

HEAD vs GET: pick the right method

If you only need the status code and final URL, a HEAD request is faster because the server skips sending the response body. Use GET when you also need to inspect content, such as checking for a noindex tag or a canonical. Be aware that some servers handle HEAD poorly and return a different status than GET, so when a HEAD result looks wrong, re-test that URL with GET before trusting it.

How to read the output

Traps that produce wrong data

• Not following redirects. If you disable redirects you see the 301 but never learn where it goes. Follow them, and separately record the hop count so you can flag chains.
• No timeout. A single hung URL without a timeout can stall the whole run. Always set one, ten seconds is a sane default.
• No user agent. Some servers block the default Python user agent. Set a descriptive one so you are not testing a blocked response instead of the real page.
• Too much concurrency on your own server. Five hundred parallel requests can look like an attack and trigger rate limiting, which returns 429 or 503 and poisons your data. Ramp up gradually.
• Ignoring transient 5xx. Re-test error rows once before reporting them; a momentary blip is not a broken page.

Bulk URL testing pairs naturally with other technical work. Feed it the URL list from your server logs to confirm which crawled pages now 404, using Python log file parsers to build that list. It also validates redirect maps during a migration, a core part of technical SEO. For large ecommerce sites, run it against filtered category URLs to see how many resolve to thin or duplicate pages, a symptom the faceted navigation SEO strategy guide addresses.

FAQ