How detectors spot a browser running in a VM or container: software WebGL renderers like SwiftShader and llvmpipe, default 800x600 screens, quantized device memory, and timing artifacts under virtualization.
![The wordmark navigator.plugins[] in white monospace with the brackets and an underline bar in orange](/navigator-plugins-fingerprint-cover.webp)
Traces navigator.plugins from a 15-bit fingerprinting signal to the five hard-coded PDF entries Chrome and Firefox ship today, the empty array that gave away old headless, and why fabricating a PluginArray still leaks.
Traces how a single browser-automation stealth patch moves through its life: a signal is found, the patch hides it, the patch itself becomes a fingerprint, and a new signal replaces the old one. With real examples and the economics of the treadmill.
Traces the architecture of a web-scale crawler from Mercator and the early Googlebot through IRLbot to today: the URL frontier, duplicate elimination, politeness scheduling, and how servers push back.
How the URL frontier orders a crawl: the Mercator front-queue/back-queue split, per-host politeness, freshness versus coverage, and the disk-backed and gRPC designs that run at web scale today.
Traces how crawl politeness works in practice: RFC 9309 robots.txt parsing, the crawl-delay split between Google, Bing, and Yandex, per-host rate limits, sitemaps, and the cryptographic verification replacing the honor system.
A primary-source walk through the URL-seen problem in large crawlers: why naive dedup fails at scale, how Bloom filters answer it, the false-positive math, and the counting, scalable, blocked, and cuckoo variants that followed.
Traces how a working proxy pool is operated: rotation strategies, the difference between a banned IP and a dead one, health-check state machines, sticky versus rotating sessions, and the per-GB cost model that decides whether a crawl is profitable.
A vendor-neutral comparison of the three proxy types: how each is sourced, how each gets detected at the ASN and reputation layer, what a gigabyte actually costs, and which job each one fits.
Traces where residential and mobile proxy IPs actually come from: bundled SDKs, free-VPN monetization, peer-payout apps, and outright malware, plus the consent gap that runs through all of them.
How identity stays coherent when a crawler rotates IPs: binding cookies and sessions to exit nodes, what breaks when a session leaks across IPs, and the signals anti-bot systems use to catch the mismatch.
The strategic choice between holding one exit IP for a session and rotating per request: where statefulness forces stickiness, where rotation buys throughput, and how session-consistency checks punish the wrong call.
Traces client-side rate control for crawlers: token and leaky buckets applied to your own requests, per-host concurrency, adaptive throttling on 429 and Retry-After, and exponential backoff with jitter.
Traces how proxy dollars per gigabyte, CAPTCHA-solve dollars per thousand, and browser compute combine through a success rate into one number that actually matters: cost per successful record.
How crawlers avoid re-fetching unchanged pages: conditional requests with ETag and Last-Modified, 304 handling, content hashing for change detection, and recrawl scheduling driven by per-page change rate.
A decision framework for choosing between a headless browser and a plain HTTP client at extraction scale: JS-dependence, per-page cost, fingerprint surface, brittleness, and the hybrid path most large crawlers actually take.
Traces the real resource cost of driving headless Chrome at scale: per-instance RAM, the multi-process tax, container failure modes, concurrency math, and the cost gap that pushes teams back to HTTP clients.
Traces how CAPTCHA solving is operationalized: the human-farm relay, the shift to ML and audio-transcription solvers, the per-solve price curve from 2010 to 2026, and the latency-accuracy-binding tradeoffs that decide whether a token is worth anything.
Traces how to instrument a scraping system end to end: the metrics that matter, why HTTP 200 is a lie, how to detect soft blocks and empty-payload garbage, and how to build dashboards and alerts that catch silent failure before the data does.
How to pull JavaScript-rendered data without launching a browser: finding the backend JSON, XHR, and GraphQL endpoints a page calls, replaying them, handling tokens and request signatures, and where the approach stops working.
A primary-source walk through intercepting a mobile app's backend: proxying TLS, why certificate pinning stops you, how runtime unpinning works conceptually, and decoding schema-less protobuf payloads.
A field-by-field dissection of the TLS ClientHello, tracing exactly which bytes JA3 and JA4 read: version, cipher suites, compression, extensions, supported_groups, signature_algorithms, supported_versions, key_share, and ALPN.
A reference walk through the full JA4+ suite: how each of JA4, JA4S, JA4H, JA4L, JA4X, JA4T and JA4SSH is constructed, what it captures, and how the a_b_c format lets the parts compose.
A deep dive into uTLS: how the Go library forges a chosen browser's ClientHello through ClientHelloID parrots and handshake control, why Go's crypto/tls is otherwise easy to fingerprint, and where the mimicry still leaks.