172: Transformers and Large Language Models

Programming Throwdown

Patrick Wheeler and Jason Gauci

Objective C, Java, Programming Throwdown, Education, News, Programming Languages, How To, Tech News, C, Python

4.6 • 604 Ratings

🗓️ 11 March 2024

⏱️ 86 minutes

🔗️ Recording | iTunes | RSS

🧾️ Download transcript

Summary

172: Transformers and Large Language Models

Intro topic: Is WFH actually WFC?

News/Links:

Falsehoods Junior Developers Believe about Becoming Senior
- https://vadimkravcenko.com/shorts/falsehoods-junior-developers-believe-about-becoming-senior/
Pure Pursuit
- Tutorial with python code: https://wiki.purduesigbots.com/software/control-algorithms/basic-pure-pursuit
- Video example: https://www.youtube.com/watch?v=qYR7mmcwT2w
PID without a PHD
- https://www.wescottdesign.com/articles/pid/pidWithoutAPhd.pdf
Google releases Gemma
- https://blog.google/technology/developers/gemma-open-models/

Book of the Show

Patrick: The Eye of the World by Robert Jordan (Wheel of Time)
- https://amzn.to/3uEhg6v
Jason: How to Make a Video Game All By Yourself
- https://amzn.to/3UZtP7b

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Patrick: Stadia Controller Wifi to Bluetooth Unlock
- https://stadia.google.com/controller/index_en_US.html
Jason: FUSE and SSHFS
- https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh

Topic: Transformers and Large Language Models

How neural networks store information
- Latent variables
Transformers
- Encoders & Decoders
Attention Layers
- History
  - RNN
    - Vanishing Gradient Problem
  - LSTM
    - Short term (gradient explodes), Long term (gradient vanishes)
- Differentiable algebra
- Key-Query-Value
- Self Attention
Self-Supervised Learning & Forward Models
Human Feedback
- Reinforcement Learning from Human Feedback
- Direct Policy Optimization (Pairwise Ranking)

★ Support this podcast on Patreon ★

Transcript

Click on a timestamp to play from that location

0:00.0	Programming Throwdown Episode 172 Transformers and Large Language Models.
0:21.8	Take it away, Jason.
0:23.6	Hey, everybody.
0:24.5	I had a really interesting discussion on LinkedIn.
0:29.0	This is like a meta-meta thing, but I post everything on LinkedIn and Twitter.
0:33.9	And my Twitter, nobody follows my Twitter.
0:37.0	And the weird thing is, my ex, yeah,
0:39.9	maybe it's going to be my ex social network. I just can't get anybody to follow me on there.
0:45.5	Everyone follows me on LinkedIn, which is fine. But I even tried, you know,
0:51.8	putting my Twitter link on presentations I give and stuff like that.
0:56.6	And people would rather just find me on LinkedIn.
1:00.4	I find that's amazing that you give presentations where people actually could join and follow you.
1:05.8	I always see people do that.
1:07.0	I've never done it myself, just not on social media.
1:09.4	But I also find it interesting that you give presentations without even be an opportunity.
1:13.2	I gave a amazing presentation to SUNY purchase, which is a university in New York, and it was
1:19.9	on the AI singularity.
1:22.3	And it was a lot of fun.
1:26.6	Had a great time there. I wasn't in person, but had a great time there or i wasn't i wasn't in person but
1:29.9	had a great time speaking oh nice now i kind of want you to give it here i actually would love
1:34.8	i asked if if they would make the video public they said no it was actually technically one of
1:39.9	the lessons like it was part of a course and so that reason, they can't just post it on the
	...

Please login to see the full transcript.

Previous episode | Next episode

Disclaimer: The podcast and artwork embedded on this page are from Patrick Wheeler and Jason Gauci, and are the property of its owner and not affiliated with or endorsed by Tapesearch.

Generated transcripts are the property of Patrick Wheeler and Jason Gauci and are distributed freely under the Fair Use doctrine. Transcripts generated by Tapesearch are not guaranteed to be accurate.