The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

Syntax - Tasty Web Development Treats

Wes Bos

Tech News, News, Technology

4.9 • 1.2K Ratings

🗓️ 25 July 2018

⏱️ 65 minutes

🔗️ Recording | iTunes | RSS

🧾️ Download transcript

Summary

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices. Kyle Prinsloo’s Freelancing & Beyond — Sponsor Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off. Freshbooks - Sponsor Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section. Show Notes 3:05 - What Is the undocumented web? When traditional methods don’t give you what you want Wild West 6:10 - Undocumented APIS Instagram’s private API Use dev tools’ XHR Request React Dev Tools will also have the data Ep 008 - Wes Bos Origin Story Realtor.ca - MLS Mapping Simon Willison’s Zeit Day Talk about XML files Charles Proxy App - also good for debugging requests 17:34 - Faking Faking User Agents Sending headers from requests Free Blackberry wifi on plane 23:52 - Scraping What is craping? Cheerio - jQuery like node package 27:30 - Automation Cron Jobs Filling out forms instantly Headless browsers Puppeteer PhantomJS 33:30 - Proxies Need a fresh IP address Need a different country Side note: Canadians buying stuff in USA M4S1P1 === 41100 API throttling 39:00 - Tips and Tricks Multiple API keys diskDB - Text files as databases Search Github for API keys Inspect Element and Delete Overlays Delete Cookiesokay 44:45 - Stories Photo Voting MLS Listing Top Entrepreneur voting cheating Vanity Plate Checker My Hockey Team Sucks Vehicle auction bidding app youtube-dl ××× SIIIIICK ××× PIIIICKS ××× Scott: Octopath Traveler & OST Wes 1: AvE Youtube Channel Wes 2: Mustie1 Youtube Channel Shameless Plugs Scott’s Level Up Tuts Pro Wes’ Courses Tweet us your tasty treats! Scott’s Instagram LevelUpTutorials Instagram Wes’ Instagram Wes’ Twitter Wes’ Facebook Scott’s Twitter Make sure to include @SyntaxFM in your tweets

Transcript

Click on a timestamp to play from that location

0:00.0	You're listening to Syntax, the podcast with the tastiest web development treats out there.
0:06.1	Strap yourself in and get ready.
0:08.0	Here is Scott Tolinsky and West Boss.
0:10.5	Hello everybody and welcome to Syntax, my is West Boss I'm a full stack web
0:15.8	developer from Canada with me as always is Scott Tolinsky how you doing today
0:20.3	Scott hey I'm doing okay just hanging out how you doing I'm doing pretty good
0:25.9	excited to it's Friday we usually record on Monday but because of vacation schedules
0:30.4	and whatnot it's a little bit jumbled up but I'm getting prepping for a
0:34.3	talk on CSS grid which I'm excited to give on going to a laryicon so I'm going to
0:38.9	the JavaScript guy going to a PhD conference giving a CSS talk.
0:44.0	I've been seeing your slides though, they look nice.
0:47.0	Yeah, yeah, I'm really happy with it.
0:49.0	That's always a fun part of giving a conference talk
0:51.0	is once you're done done you can actually design the
0:53.2	design the thing and I did this new thing where I did the the examples in the slide
0:59.2	deck and I just wrote them in a style tag and then I wrote some code that would display the style tag and then syntax highlight it.
1:07.4	It was really cool.
1:08.4	Oh, cool.
1:09.4	Yeah, a little bit of automation there, nice.
1:11.4	Big, big fan of that. Any that anyways today we're talking about the
1:14.9	undocumented web which is kind of like everything that is scraping private
1:20.8	apies proxies alternative solutions we say in quotes kind of necessarily
	...

Please login to see the full transcript.

Previous episode | Next episode

Disclaimer: The podcast and artwork embedded on this page are from Wes Bos, and are the property of its owner and not affiliated with or endorsed by Tapesearch.

Generated transcripts are the property of Wes Bos and are distributed freely under the Fair Use doctrine. Transcripts generated by Tapesearch are not guaranteed to be accurate.