meta_pixel
Tapesearch Logo
Log in
Syntax - Tasty Web Development Treats

763: Web Scraping + Reverse Engineering APIs

Syntax - Tasty Web Development Treats

Wes Bos

Tech News, Technology, News

4.91.2K Ratings

🗓️ 1 May 2024

⏱️ 53 minutes

🧾️ Download transcript

Summary

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease. Show Notes 00:00 Welcome to Syntax! 03:13 Brought to you by Sentry.io. 05:00 What is scraping? Socialblade. 08:01 Examples of past scrapers. Canadian Tire. 10:06 Cloud app downloader. 16:13 Other use cases. 16:58 Scraping 101. 17:28 Client Side. 19:08 Private API. Proxyman. 22:40 Server rendered. 23:27 Initial state. 24:57 What format is the data in? Google Puppeteer Extension. 27:08 Working with the DOM. 27:12 Linkedom npm package. 29:02 querySelector everything. 31:28 How to find the elements without classes. 34:08 Use XPath selectors for select by word. 34:53 Make them as flexible as you can. Classes change! 35:10 AI is good at this! 36:26 File downloading. 38:20 Working with protected routes. Proxyman. 40:41 Programatically retrieve authentication keys because they are short-lived. Fetch Cookie. 43:20 Deal-breakers. Mechanical Turk. 44:58 What happened with Amazon? Uniqlo Self-Checkout 46:42 Wes’ portable refrigerator utopia. 47:25 Sick Picks & Shameless Plugs. Sick Picks Scott: KeyboardCleanTool. Wes: Yabai. Shameless Plugs Scott: Syntax on YouTube Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott:X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Transcript

Click on a timestamp to play from that location

0:00.0

Welcome to Syntax. Today we've got a show for you on web scraping and reverse engineering

0:06.5

APIs. So it's been a while since we've done a show like this and it's a personal hobby of mine.

0:14.0

I've written so many scrapers in my day

0:17.6

and I've learned quite a few things about

0:20.0

how do you scrape data off of web pages how do you interact with web pages in a way where

0:28.0

maybe they didn't necessarily mean for you to use the website in that way. So today we're going to go into all the

0:34.6

tips and tricks and tools on writing a web scraper for your own benefit. My name is

0:41.3

Wes, I'm a developer from Canada with me as always

0:43.5

to Scott. How you doing today Scott? Oh I'm doing okay man. Hey we had a

0:48.1

wild weekend we had our like school auction because our kids go to public schools here so they you know have

0:54.2

fundraisers and stuff and my wife Courtney decided to be the co-chair of the

0:58.5

auction which we've heard is kind of like a hell job. Everybody is often talked about how that's not something that you want to do, but she decided to step up and try to co-chair it up and my gosh, dude, they crushed it. She, uh, just she raised the, they raised the most money they've ever raised here that everything was like

1:17.4

went off without a hitch the event was great all the parents were happy it was

1:21.8

just like a wild weekend, but man we've been we've been a little

1:26.1

stressed over here at the household kind of leading up to this event so it's it was

1:30.8

really a cathartic to get that out and you know it was such a release that like on

1:35.2

Sunday she was just totally wiped and so it was just like what a what an eventful weekend

1:41.0

but man it was it was really cool I got to do some

1:42.8

dancing at the the event too did you auction yourself off is it was it like one of those

1:48.6

like dinner with mr. Tolinsky I wish no it was uh no it was, no, it was, you know, just, there's like a, all the parents

1:56.6

donated a bottle of wine so then you could, you know, try to win 50 bottles of wine or

2:01.8

whatever or different events people were putting

...

Please login to see the full transcript.

Disclaimer: The podcast and artwork embedded on this page are from Wes Bos, and are the property of its owner and not affiliated with or endorsed by Tapesearch.

Generated transcripts are the property of Wes Bos and are distributed freely under the Fair Use doctrine. Transcripts generated by Tapesearch are not guaranteed to be accurate.

Copyright © Tapesearch 2026.