meta_pixel
Tapesearch Logo
Log in
Syntax - Tasty Web Development Treats

The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

Syntax - Tasty Web Development Treats

Wes Bos

Js, Technology, Css, Html5, Tech News, Webdevelopment, News, Javascript, Html

4.9 • 1.1K Ratings

🗓️ 25 July 2018

⏱️ 65 minutes

🔗️ Recording | iTunes | RSS

🧾️ Download transcript

Summary

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.

Kyle Prinsloo’s Freelancing & Beyond — Sponsor

Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.

Show Notes

3:05 - What Is the undocumented web?

  • When traditional methods don’t give you what you want
  • Wild West

6:10 - Undocumented APIS

17:34 - Faking

  • Faking User Agents
    • Sending headers from requests
    • Free Blackberry wifi on plane

23:52 - Scraping

  • What is craping?
  • Cheerio - jQuery like node package

27:30 - Automation

33:30 - Proxies

  • Need a fresh IP address
  • Need a different country
  • Side note: Canadians buying stuff in USA M4S1P1 === 41100
  • API throttling

39:00 - Tips and Tricks

  • Multiple API keys
  • diskDB - Text files as databases
  • Search Github for API keys
  • Inspect Element and Delete Overlays
  • Delete Cookiesokay

44:45 - Stories

××× SIIIIICK ××× PIIIICKS ×××

Shameless Plugs

Tweet us your tasty treats!

Transcript

Click on a timestamp to play from that location

0:00.0

You're listening to Syntax, the podcast with the tastiest web development treats out there.

0:06.1

Strap yourself in and get ready.

0:08.0

Here is Scott Tolinsky and West Boss.

0:10.5

Hello everybody and welcome to Syntax, my is West Boss I'm a full stack web

0:15.8

developer from Canada with me as always is Scott Tolinsky how you doing today

0:20.3

Scott hey I'm doing okay just hanging out how you doing I'm doing pretty good

0:25.9

excited to it's Friday we usually record on Monday but because of vacation schedules

0:30.4

and whatnot it's a little bit jumbled up but I'm getting prepping for a

0:34.3

talk on CSS grid which I'm excited to give on going to a laryicon so I'm going to

0:38.9

the JavaScript guy going to a PhD conference giving a CSS talk.

0:44.0

I've been seeing your slides though, they look nice.

0:47.0

Yeah, yeah, I'm really happy with it.

0:49.0

That's always a fun part of giving a conference talk

0:51.0

is once you're done done you can actually design the

0:53.2

design the thing and I did this new thing where I did the the examples in the slide

0:59.2

deck and I just wrote them in a style tag and then I wrote some code that would display the style tag and then syntax highlight it.

1:07.4

It was really cool.

1:08.4

Oh, cool.

1:09.4

Yeah, a little bit of automation there, nice.

1:11.4

Big, big fan of that. Any that anyways today we're talking about the

1:14.9

undocumented web which is kind of like everything that is scraping private

1:20.8

apies proxies alternative solutions we say in quotes kind of necessarily

...

Please login to see the full transcript.

Previous episode | Next episode

Disclaimer: The podcast and artwork embedded on this page are from Wes Bos, and are the property of its owner and not affiliated with or endorsed by Tapesearch.

Generated transcripts are the property of Wes Bos and are distributed freely under the Fair Use doctrine. Transcripts generated by Tapesearch are not guaranteed to be accurate.

Copyright © Tapesearch 2025.