Large URL List Validator

What "validate" means here

For each URL in your list, we do a real HTTP request and capture: the response status code (200/3xx/4xx/5xx), the full redirect chain if the URL redirects (final destination after up to 10 hops), the response time in milliseconds, a content-fingerprint check for soft 404s (200 OK with not-found content), and whether the target is currently blocking automated access (403/429 patterns flagged separately so you know it's a block, not a real failure).

The result is a row per URL with every column you'd need to act on the list — status group filter, exportable CSV, machine-readable JSON via the API.

Why "large" actually matters at the engineering level

A list of 50,000 URLs is qualitatively different from a list of 500. A naive validator that opens 50,000 concurrent connections gets rate-limited, blacklisted, or simply times out. A naive sequential validator takes hours per job.

The architecture under this validator: cloud worker pool with proxy rotation, per-domain throttling so a list concentrated on a single hostile target doesn't poison the whole job, retry-with-different-proxy on transient failures, deduplication so the same URL isn't checked twice. Result: a 50,000-URL job typically completes in 10-30 minutes.

• 75,000 URLs per single job (split larger lists across jobs)
• Domain-aware throttling — concentrated and diverse lists both run optimally
• Proxy rotation — single-target hostile domains don't poison the run
• Retry-with-different-proxy on transient failures
• Soft-404 detection via content fingerprint
• CSV upload (any format with a URL column), or paste, or POST to the REST API

Common large-list workflows

Backlink portfolio audit — a brand with 50,000 inbound backlinks needs the dead set surfaced for reclamation outreach.

Site migration QA — the old sitemap pulled from GSC has 30,000 URLs, all need to be redirected correctly after launch, broken redirects need to be caught before users find them.

Content database audit — a publisher with 100,000 articles tags each with one or more outbound references; the references go stale; periodic validation catches the dead ones.

Affiliate / referral link monitoring — an affiliate site with 20,000 outgoing affiliate URLs; vendors retire programs without telling affiliates; the dead set needs to be replaced or removed.

API endpoint health check — devops team validating that 5,000 internal/external service URLs all respond as expected, with timing data.

Beyond manual — automate it

The REST API lets you POST a URL list and get a job ID back, then poll for completion or webhook on done. Useful for integrating large-list validation into nightly CI, scheduled audits, or your own internal tooling.

There's also an MCP server so AI agents (Claude Desktop, ChatGPT Apps, Cursor) can drive the validator as a tool — useful when you want an agent to "verify these 30,000 URLs and tell me which ones broke."