meta_pixel
Tapesearch Logo
Log in
The a16z Show

The Cool Stuff Only Happens at Scale

The a16z Show

a16z

Business, Software Eating The World, Culture, Innovation, Disruption, Entrepreneurship, Science, Technology

4.21.2K Ratings

🗓️ 5 June 2015

⏱️ 22 minutes

🧾️ Download transcript

Summary

Distributed computing frameworks like Hadoop and Spark have enabled processing of "big data" sets -- but that's not enough for modeling surprise/rare "black swan" or complex events. Just think of scenarios in disaster planning (earthquakes, terrorist...

Transcript

Click on a timestamp to play from that location

0:00.0

The content here is for informational purposes only, should not be taken as legal business tax or investment advice or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund.

0:14.3

For more details, please see A16Z.com slash disclosures.

0:18.7

Hi, this is Chris Dixon's the A16Z podcast.

0:21.4

I'm here with Herman Nerula from Improbable and VJ Panday from Stanford, who's also

0:26.5

a professor in residence here at A16Z.

0:28.9

Hey, guys, let's talk about distributed computing.

0:30.8

So my own view, I guess, I'll just start it off, is that over the next few decades,

0:36.7

distributed computing will be a particularly

0:37.9

important topic because we're now with things like AWS awash in computing resources.

0:45.8

You know, compute is becoming approaching zero, storage, networking.

0:50.0

But most of this is, you know, on multiple physical machines. And it's very, very hard for software developers to build software that distributes well.

1:02.0

And so we have things like, I think, of Hadoop and its successor, we think as successor, Spark, as frameworks for doing distributed computing for specific application,

1:11.3

which is data processing.

1:13.0

And I think we'll probably see more of that kind of pattern among other verticals.

1:18.5

And also, at the same time, more infrastructure that helps us, you know, programming languages,

1:24.8

frameworks, et cetera, that help us do this.

1:27.7

So, I don't know, maybe Vij, if we could start off.

1:30.3

How do you view this?

1:31.4

Yeah, I mean, I think this is the key issue, right?

1:33.2

Because everyone can program the standard C paradigm on one processor core.

1:37.7

Now actually, you know, going to multi-core is not so hard, but start thinking about multiple boxes,

1:42.6

thousand boxes, 10,000 boxes.

...

Please login to see the full transcript.

Disclaimer: The podcast and artwork embedded on this page are from a16z, and are the property of its owner and not affiliated with or endorsed by Tapesearch.

Generated transcripts are the property of a16z and are distributed freely under the Fair Use doctrine. Transcripts generated by Tapesearch are not guaranteed to be accurate.

Copyright © Tapesearch 2026.