Skip to main content
AmigoFish

Billion Dollar Industry | Proxies

From Chai aur Code · · 34 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Check Out Webshare: https://hitesh.ai/webshare Use Code HITESH20 to get 20% off your first purchase.

Training data is a billion-dollar industry, and almost all of it starts with one boring problem: collecting data at scale without getting blocked.

In this video we take a simple goal, downloading 10,000 YouTube videos for an AI dataset, and watch the naive script hit the wall: 429s, captchas and a blocked home IP. Then we engineer it properly: how platforms actually detect bots (IP reputation, TLS fingerprints, headers, behaviour), which type of proxy to use when, rotation vs cool-down, sticky sessions, and how to grow from a single-machine scraper into a distributed, self-healing pipeline, including the bandwidth and cost math behind it.

This is engineering, not system hacks. A proxy is a network routing layer, not a magic switch.

⏱️ Chapters 0:00 Intro 2:21 The goal: download 10,000 YouTube videos 4:02 Training data is a billion-dollar industry 5:01 The naive yt-dlp script 5:43 Webshare: proxy types and the free plan 7:08 Webshare dashboard and proxy list 8:24 Hitting "the wall" in scraping 9:47 Engineering, not system hacks 11:15 Version zero: the naive scraper 11:55 4 missing pieces in basic scrapers 13:03 Detection layers: IP reputation and TLS fingerprint (JA3/JA4) 15:27 Detection layers: headers and behaviour patterns 16:27 The proxy fallacy 17:52 Types of proxies: datacenter, static ISP, rotating residential 19:49 Proxy selection framework 20:36 Rotation vs cool-down 21:33 The sticky session rule 22:25 Architecture v1: single machine 23:49 Smart error classification 24:30 Architecture v2: distributed fleet 25:22 Idempotency and global rate limits 25:55 Building a proxy manager 26:28 Self-healing proxy pool 27:14 Bandwidth math: format selection 28:20 Budgeting data pipelines 29:06 AI dataset processing, filtering and provenance 29:42 Webshare API docs, dashboard and pricing 32:32 Outro

📌 What you will learn • Why scrapers that work for 50 requests fail at 800 • The 4 detection layers every large platform uses • Datacenter vs static ISP vs rotating residential proxies, and how to choose • Backoff, jitter, error taxonomy and IP pools • Queues, stateless workers, idempotency and a shared rate limiter (Redis + BullMQ) • Circuit breakers and health checks for a proxy pool • How format selection cuts bandwidth bills by over 95% • Turning raw downloads into a clean, legally traceable AI dataset

Always respect a platform's terms of service and the law when collecting data.

#proxies #webscraping #aidatasets #systemdesign #chaiaurcode

All source code is available at my Github account: https://github.com/hiteshchoudhary

Our Open-Source Project is here: https://freeapi.app

Join me at whatsapp: https://hitesh.ai/whatsapp

for community discord: https://hitesh.ai/discord

Instagram pe yaha paaye jaate h: https://www.instagram.com/hiteshchoudharyofficial/

HTML video series: https://www.youtube.com/watch?v=XmLOwJHFHf0&list=PLu71SKxNbfoDBNF5s-WH6aLbthSEIMhMI

Complete javascript series: https://www.youtube.com/watch?v=Hr5iLG7sUa0&list=PLu71SKxNbfoBuX3f4EOACle2y-tRC5Q37

Complete Reactjs series: https://www.youtube.com/watch?v=vz1RlUyrc3w&list=PLu71SKxNbfoDqgPchmvIsL4hTnJIrtige

Javascript and react interview series: https://www.youtube.com/watch?v=1wqCyz7XrV4&list=PLu71SKxNbfoCy_MsA98SBfvUKF5eQit6L

Backend development with Javascript: https://www.youtube.com/watch?v=EH3vGeqeIAo&list=PLu71SKxNbfoBGh_8p_NS-ZAh6v7HhYqHW

Python Series: https://www.youtube.com/watch?v=Ca5DLSDfPec&list=PLu71SKxNbfoBsMugTFALhdLlZ5VOqCg2s

Back to Chai aur Code · Report a problem