About

I am Daniel Mercer, and I write the field guides on this blog. I have spent the better part of a decade working on web infrastructure and the systems that sit between automated clients and the servers they talk to: TLS and HTTP fingerprinting, bot detection, proxy networks, and the protocols underneath all of it. I have built crawlers, and I have built the things that catch them, which is the perspective these posts are written from.

The Crawlex Blog is published by Crawlex, an operator of crawling infrastructure at production scale. The publication is editorially separate from the commercial side: nothing here promotes Crawlex services, and bypass code is never published. The brief is narrow on purpose. Reference posts traced through primary sources, written to age well rather than chase headlines.

When a post states a specific fact, a cookie field, a date, an RFC number, a percentage, it is because a primary source supports it, and where a system’s internals are not public the post says so rather than guessing. If you find something wrong, I want to know.

Corrections, post pitches, or partnership questions: [email protected].