What is Apache Spark?
Apache Spark is a distributed processing engine that splits work on very large data sets across many machines and runs it in parallel. Programs describe transformations on data, and Spark’s optimizer turns them into a plan of tasks that executors across the cluster carry out, keeping intermediate results in memory where possible. That design made Spark much faster than Hadoop MapReduce for iterative workloads such as machine learning and interactive queries, and it quickly became one of the most widely adopted big data tools.
Spark provides one engine for several kinds of work. Spark SQL and the DataFrame API handle structured data, Structured Streaming processes continuous data streams, MLlib supplies machine learning algorithms, and GraphX supports graph analysis. Developers write Spark applications in Python, Scala, Java, SQL or R, and Spark runs on its own standalone cluster manager, on Hadoop YARN or on Kubernetes, reading data from cloud object storage, HDFS, relational databases, Apache Kafka and table formats such as Delta Lake and Apache Iceberg.
The project is governed by the Apache Software Foundation and developed in the open by more than 2,000 contributors from companies and universities, including Databricks, which was founded by Spark’s original creators. The project says thousands of companies, including 80% of the Fortune 500, use Spark. Recent releases have focused on Spark Connect, a client-server interface for remote clusters; Spark 4.0 in May 2025 made ANSI SQL behavior the default, Spark 4.1 added Spark Declarative Pipelines, and Spark 4.2 in July 2026 introduced geospatial data types and change data capture queries.
Key features of Apache Spark
- 01
Spark SQL and DataFrames
Structured data can be queried with ANSI SQL or manipulated through DataFrame APIs in Python, Scala, Java and R, with the Catalyst optimizer planning efficient execution across the cluster.
- 02
Structured Streaming
Streaming jobs use the same DataFrame operations as batch jobs and process data incrementally from sources such as Apache Kafka, and a real-time mode added in Spark 4.1 targets sub-second latency.
- 03
MLlib machine learning
MLlib provides distributed algorithms for classification, regression, clustering and recommendation, along with pipelines for feature engineering, model tuning and evaluation on large data sets.
- 04
PySpark and pandas API
PySpark exposes Spark to Python developers, including a pandas-compatible API for scaling familiar pandas code, and Spark 4.2 turned on Arrow-optimized Python functions by default.
- 05
Spark Connect
A client-server protocol separates applications from the cluster, so lightweight clients in Python, Scala and other languages can send work to a remote Spark server without a full local installation.
- 06
Spark Declarative Pipelines
Introduced in Spark 4.1, this framework lets engineers declare the data sets and queries a pipeline should produce while Spark works out dependencies, ordering, parallelism, checkpoints and retries.
- 07
Flexible deployment
Spark runs on a laptop, on its standalone cluster manager, on Hadoop YARN or on Kubernetes, and it is offered as a managed service by Databricks, Amazon EMR, Google Cloud Dataproc and Microsoft Fabric.
Who uses Apache Spark?
- Data engineers run large batch ETL jobs that clean, join and aggregate terabytes or petabytes of data for data warehouses and lakehouses.
- Streaming teams process event data from Apache Kafka in near real time for fraud detection, monitoring and personalization.
- Data scientists prepare training data and fit machine learning models on data sets too large for a single machine.
- Analysts run interactive SQL queries over files in cloud object storage stored in formats such as Parquet, Delta Lake and Apache Iceberg.
History of Apache Spark
Matei Zaharia started Spark in 2009 as a research project at UC Berkeley’s AMPLab, looking for a faster and simpler model for cluster computing than Hadoop MapReduce. The code was open-sourced in early 2010, and Berkeley researchers built related projects on top of it, including the Shark SQL engine and the GraphX graph library.
In 2013 the project moved to the Apache Software Foundation, the same year several of its creators founded Databricks, and it became a top-level Apache project in February 2014. Version 1.0 arrived later that year. Spark 2.0 in 2016 introduced Structured Streaming and unified the DataFrame and Dataset APIs, and Spark 3.0 followed in 2020. The 4.x series began in May 2025, and Spark 4.2 was released in July 2026.
- 2009
Matei Zaharia starts Spark as a research project at UC Berkeley’s AMPLab.
- 2010
Spark is released as open-source software.
- 2013
The project enters the Apache Software Foundation’s incubator.
- 2014
Spark becomes a top-level Apache project in February and reaches version 1.0.
- 2016
Spark 2.0 introduces Structured Streaming and a unified DataFrame and Dataset API.
- 2020
Spark 3.0 is released with a new adaptive query execution framework.
- 2025
Spark 4.0 makes ANSI SQL mode the default and adds the VARIANT data type.
- 2026
Spark 4.2 adds geospatial data types and SQL change data capture.
Apache Spark pricing
Pricing modelFree and open source
Free to download and use under the Apache 2.0 license; managed Spark services from Databricks and the major cloud providers charge for the compute they run.
Prices change often. Check the official Apache Spark website for current plans.
Apache Spark alternatives
Well-known alternativesApache Flink, Ray, Dask, Trino and Apache Beam
Apache Spark: frequently asked questions
Who develops Apache Spark?
Apache Spark is developed by an open-source community under the Apache Software Foundation, which has governed the project since 2013. It was created by Matei Zaharia at UC Berkeley in 2009, and Databricks, the company he co-founded with other members of the original Spark team, is among the contributors.
What is Apache Spark used for?
Apache Spark is used to process and analyze data sets too large for a single computer. Common uses include ETL pipelines that prepare data for warehouses and lakehouses, SQL analytics on data lakes, real-time stream processing and training machine learning models on large volumes of data.
Is Apache Spark free?
Yes. Apache Spark is free and open source under the Apache License 2.0, so anyone can download, use and modify it. Many organizations pay for managed Spark services, such as Databricks, Amazon EMR or Google Cloud Dataproc, which charge for the computing resources used.
When was Apache Spark released?
Apache Spark was first released as open-source software in early 2010, after starting as a UC Berkeley research project in 2009. It joined the Apache Software Foundation in 2013, reached version 1.0 in 2014 and released version 4.0 in May 2025.
Is Apache Spark faster than Hadoop MapReduce?
For many workloads, yes. Spark keeps intermediate data in memory and optimizes whole query plans, so iterative algorithms and interactive queries usually run much faster than on Hadoop MapReduce, which writes results to disk between steps. Spark can still run on Hadoop clusters through YARN and read data from HDFS.
Sources
- Apache Spark spark.apache.org
- Apache Spark project history spark.apache.org
- The Apache Software Foundation announces Apache Spark as a top-level project news.apache.org
- Spark release 4.2.0 spark.apache.org
- Apache Spark, Wikipedia en.wikipedia.org
Last reviewed . Spotted an error? Send a correction.