2d • Wins 🌟
Scraping 30,000 Images and Videos For a Client
Here's how I scraped 30,000+ encrypted videos from a protected CDN for my client.
When you're planning to download photos or videos, you may not immediately see them in the Network tab.
To make it easier, change the filter from "All" to "Media". This will show you the requests you're looking for.
Once you see the requests being made, you will most likely encounter one of the following:
1. Video stream chunks (.ts segments), not single .mp4 files
In this case, you have to bypass the CDN, get each chunk, and stitch the video together to make the .mp4 file.
Many CDNs use AES-128 encryption to protect content.
You need to:
- Extract the encryption key URI from the .m3u8 playlist
- Download the decryption key
- Grab the initialization vector (IV)
Without all three, you can't decrypt anything.
The last segment has PKCS7 padding that needs to be stripped. Skip this step and your final video will be corrupted.
Concatenate all decrypted segments into a single .ts file, then use ffmpeg to convert to .mp4:
ffmpeg -i combined.ts -c copy -bsf:a aac_adtstoasc output.mp4
The -bsf:a aac_adtstoasc flag is critical for proper audio encoding.
2. Full MP4/image files are available (the easier case)
Sometimes you'll find a direct link to the full .mp4, .jpg, or .png file.
In this case: - Right-click the request in the Media tab → Copy as cURL - Paste the contents into a cURL to Python converter or have AI implement it for you - Even when videos aren't encrypted, CDNs still protect them.
You need to mimic a real browser request to succeed. Replace requests with curl_cffi and ensure you can make a request without getting blocked.
Watch out for:
- Expiring URLs with time-based tokens (you'll need to regenerate them)
- Rate limiting (add delays between requests or rotate IPs)
- Dynamic URLs that change per session
7
2 comments
Scraping 30,000 Images and Videos For a Client
skool.com/ai-automation-society
A community built to master no-code AI automations. Join to learn, discuss, and build the systems that will shape the future of work.
Leaderboard (30-day)
1
🔥
+11300
2
🔥
+7846
3
+3370
4
🔥
+3020
5
+2334
Powered by