Hello, everyone. Welcome to the A6 and Z podcast. I'm Sonal and I'm here today with Matei Zaharia,

the CTO and co-founder of Databricks, which is the primary company driving and developing Spark.

And we're actually just coming out of the Spark Summit, which took place this week. And it's one of the

biggest events for developers who are working on Spark, for companies that are interested in Spark,

and pretty much for anyone who cares about trends in the big data space.

Just to start off, Matei, just start by just giving us a description of what Spark is.

So Spark is software for processing large volumes of data on a cluster.

And the things that make it unique are, first of all, it has a very powerful programming model

that lets you do many kinds of advanced analytics and processing, such as machine learning

or graph computation or stream processing. And second, it's designed to be very easy to use,

much easier to use than previous systems for working with large data. So what were some of the

previous systems for working with large data. So what were some of the previous systems for working with large data sets?

Before Spark, the most widely used system was probably MapReduce, which was invented at Google

and popularized to the Open Source Hadoop project.

And MapReduce itself was a major step over just writing distributed programs from scratch,

but it was still very difficult to adopt and use and led to very complicated applications

and also very poor performance in some of them.

So what were some of the reasons for inventing Spark in the first place then?

I mean, besides the problems and the limitations of that,

were you just trying to solve the problems of MapReduce, or were you actually trying to do something

different? Yeah, that's a good question. So we started building Spark after several years of

working on MapReduce and working with companies that were very early on using MapReduce.

...