Web Scraping Meta Tags Without Getting Blocked — Lessons Learned

I've spent the last few months building a system that extracts meta tags from URLs at scale. Along the way I hit every wall you can imagine — rate limits, CAPTCHAs, bot detection, encoding nightmares, and HTML so malformed it would make a parser cry. Here's everything I learned, so you don't have to learn it the hard way. The Simple Version (That Breaks Immediately) Extracting meta tags seems trivial: const res = await fetch(url); const html = await res.text(); const title = html.match(/<title>(.*?)<\/title>/)?.[1]; This works for about 60% of websites. The other 40% will teach you humility. Problem 1: Bot Detection Many sites block requests that don't look like a real browser. What Gets You Blocked Missing or generic User-Agent header No Accept, Accept-Language, or Accept-Encoding headers Requesting from cloud provider IP ranges (AWS, GCP, Azure) Making too many requests too fast Missing TLS fingerprint characteristics What Works Set headers that look like a real browser: