Apache Spark architecture has three parts: a driver that plans the work, a cluster manager that hands out machines, and executors that run tasks on those machines in parallel. This guide explains each part, how a PySpark job becomes a DAG of stages and tasks, and how to see it all with runnable code.
By the end you will be able to answer the classic interview questions: what the driver does, why a groupBy creates a new stage, how many tasks a stage runs, and how client mode differs from cluster mode.
Run the examples: Every code block is self-contained. Press Run on any block; this page runs Spark in local mode, so the driver and the executor share one machine, but the plan, jobs, stages and tasks are exactly the ones a cluster would use.
Spark architecture at a glance
| Component | Where it runs | What it does |
|---|---|---|
| Driver | One process: your laptop (client mode) or a cluster node (cluster mode) | Runs your program, builds the plan, splits it into stages and tasks, schedules them and collects results |
| SparkSession / SparkContext | Inside the driver | Entry point to Spark; holds the connection to the cluster |
| Cluster manager | Standalone master, YARN ResourceManager or Kubernetes | Starts executors and gives them CPU and memory; it never runs your code |
| Executors | Processes on worker nodes | Run tasks, keep cached data in memory or on disk, write shuffle files, report back to the driver |
| Task | One thread in an executor | Processes one partition of data for one stage |
(Diagram: Apache Spark architecture diagram: the driver program with its SparkContext talks to the cluster manager, which starts executors on worker nodes; each executor runs tasks and keeps a cache. See it in the original article.)
Figure 1: Spark application architecture. The driver talks to the cluster manager for resources and to the executors for work.
The driver: the brain of a Spark application
The driver is the process that runs your main program, for example the PySpark script you submit. It:
- creates the
SparkSession(and itsSparkContext), the entry point to Spark; turns your DataFrame code into a logical plan, lets the Catalyst optimizer improve it and chooses a physical plan;
splits each job into stages (the DAG scheduler) and each stage into tasks (the task scheduler);
sends tasks to executors, tracks them, and retries a failed task on another executor;
receives results for actions such as
collect(),count()orshow().
If the driver stops, the application stops: the executors belong to it. That is why collect() on a very large DataFrame is dangerous: every row is copied into the driver's memory.
In PySpark: Your Python code runs in a Python driver process that controls a JVM through Py4J. DataFrame operations are executed by the JVM, so they run at JVM speed. Python UDFs and RDD lambdas run in Python worker processes next to each executor, and rows are serialised between the JVM and Python, which is why built-in functions are faster than UDFs.
The cluster manager: who gets which machines
The cluster manager owns the cluster's CPU and memory. When an application starts, the driver asks it for executors (for example, 10 executors with 4 cores and 8 GB each); the cluster manager starts them on worker nodes. It does not schedule tasks; the driver does that.
- Standalone: the simple cluster manager that ships with Spark.
Hadoop YARN: the resource manager of Hadoop clusters (EMR, on-premises Hadoop).
Kubernetes: runs the driver and executors as pods.
Apache Mesos: deprecated since Spark 3.2; avoid it for new work.
Managed platforms such as Databricks, AWS Glue and EMR run one of these for you, so you only choose the cluster size.
Executors: where the work happens
Executors are processes on the worker nodes. Each executor has a number of cores; one core runs one task at a time. Executors also store cached DataFrames and the shuffle files that one stage writes for the next. Normally an executor lives as long as the application; with dynamic allocation Spark can add and remove executors as the load changes.
A useful rule: the number of tasks that can run at once = executors x cores per executor. With 10 executors of 4 cores, 40 tasks run in parallel; a stage with 200 partitions needs 5 waves of tasks.
Transformations, actions and lazy evaluation
DataFrame operations come in two kinds. Transformations (select, filter, withColumn, join, groupBy) describe a new DataFrame and return immediately. Actions (count, show, collect, write) ask for a result, and only then does Spark run anything. This is lazy evaluation: Spark waits until it sees the whole pipeline, so it can optimise it as one plan.
(Diagram: Spark RDD workflow diagram: create an RDD, apply transformations such as groupBy, join and union to get new RDDs, then an action such as collect, count or write returns the results. See it in the original article.)
Figure 2: transformations build new datasets; an action produces a result.
(Diagram: Lazy evaluation in Spark: a chain of transformations (T) is only recorded, and everything runs when an action (A) is called. See it in the original article.)
Figure 3: lazy evaluation. Transformations are recorded; the action triggers execution.
Run this block: the transformations finish instantly, and the work only happens at count().
from pyspark.sql import SparkSession, functions as F import time spark = SparkSession.builder.appName("LazyEvaluationDemo").getOrCreate() orders = spark.range(0, 2000000).withColumn("amount", (F.col("id") % 500) * 1.5) t0 = time.time() big = orders.filter(F.col("amount") > 600).withColumn("with_tax", F.col("amount") * 1.18) print("Transformations defined in", round(time.time() - t0, 3), "seconds: nothing has run yet") t0 = time.time() print("Rows with amount over 600:", big.count()) print("The action count() ran the job in", round(time.time() - t0, 2), "seconds") For a deeper look, see the tutorial on lazy evaluation in PySpark.
Narrow and wide transformations
A narrow transformation needs only the rows of its own partition: filter, select, withColumn, union. Spark runs a chain of them in one pass over each partition.
A wide transformation needs rows from many partitions: groupBy with an aggregation, join (unless one side is broadcast), distinct, orderBy, repartition. Spark must shuffle: write the data out by key and read it back on other executors. A shuffle is the expensive part of most jobs, and it is where a new stage begins.
(Diagram: RDD partitioning diagram: a dataset is split into partitions that are spread across the nodes of the cluster. See it in the original article.)
Figure 4: a dataset is split into partitions; each task works on one partition.
From code to execution: jobs, stages and tasks
- Each action starts a job.
The DAG scheduler cuts the job into stages at every shuffle (wide dependency). Narrow steps between two shuffles stay in one stage.
Each stage runs one task per partition. Tasks of a stage run in parallel on the executors.
A stage starts only after the stages it depends on have finished writing their shuffle output.
(Diagram: Spark jobs, stages and tasks diagram: the driver starts jobs, each job is split into stages, and each stage into parallel tasks. See it in the original article.)
Figure 5: the driver turns each action into a job, each job into stages, and each stage into tasks.
(Diagram: Spark DAG visualisation: four stages, where parallelize, filter and map run inside a stage and the join needs a new stage. See it in the original article.)
Figure 6: a DAG of stages. Narrow steps (filter, map) stay inside a stage; the join starts a new one.
The DAG (directed acyclic graph) is the graph of these steps: directed because data flows one way, acyclic because no step depends on its own output. Spark uses the DAG to plan the stages and to recompute a lost partition from its parents if an executor fails, which is how Spark is fault tolerant without copying all the data.
You can see the stage boundary in the physical plan. Every Exchange is a shuffle, so it splits stages:
from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.appName("StagesDemo").getOrCreate() sales = spark.createDataFrame( [("Delhi", "Laptop", 900.0), ("Pune", "Phone", 300.0), ("Delhi", "Phone", 250.0), ("Jaipur", "Laptop", 1100.0), ("Pune", "Tablet", 400.0), ("Delhi", "Tablet", 380.0)], ["city", "product", "amount"], ) narrow = sales.filter(F.col("amount") > 280).withColumn("amount_usd", F.round(F.col("amount") / 83, 2)) print("Narrow only (no Exchange, one stage):") narrow.explain() wide = narrow.groupBy("city").agg(F.sum("amount").alias("revenue")) print("With groupBy (Exchange = shuffle = new stage):") wide.explain() wide.orderBy("city").show() In the second plan, look for Exchange hashpartitioning(city, ...) between a partial and a final HashAggregate: Spark aggregates inside each partition first, shuffles the partial sums by city, then finishes the sum. Adaptive Query Execution (on by default since Spark 3.2) may also show AdaptiveSparkPlan and combine small shuffle partitions at run time.
Partitions decide how many tasks run
The number of partitions sets the number of tasks per stage, so it decides how much parallelism you get. After a shuffle, DataFrames use spark.sql.shuffle.partitions (200 by default), which AQE can reduce when the data is small.
from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.appName("PartitionsDemo").getOrCreate() sc = spark.sparkContext print("Spark version:", spark.version) print("Master:", sc.master) print("Default parallelism:", sc.defaultParallelism) print("Shuffle partitions setting:", spark.conf.get("spark.sql.shuffle.partitions")) df = spark.range(0, 100000) print("Partitions of spark.range():", df.rdd.getNumPartitions()) df8 = df.repartition(8) print("After repartition(8):", df8.rdd.getNumPartitions(), "partitions, so 8 tasks in the next stage") per_partition = df8.withColumn("partition", F.spark_partition_id()).groupBy("partition").count() per_partition.orderBy("partition").show() This page's compiler is a small shared sandbox, so it sets the shuffle partitions and the default parallelism to 1; on a cluster you will usually see 200 shuffle partitions and a default parallelism equal to the total number of executor cores. repartition(8) still gives 8 partitions here, because you asked for them explicitly.
repartition(n) shuffles to exactly n partitions; coalesce(n) only merges existing partitions without a full shuffle. See repartition vs coalesce for when to use each.
Deploy modes: local, client and cluster
| Mode | Where the driver runs | Typical use |
|---|---|---|
Local (master = local[*]) | On one machine, together with the executor threads | Learning, tests, small data (this page) |
| Client mode | On the machine that submits the job (laptop, notebook or edge node); executors run in the cluster | Interactive work: notebooks, shells |
| Cluster mode | Inside the cluster, started by the cluster manager | Production jobs: they keep running if your laptop disconnects |
(Diagram: Spark client mode diagram: the driver runs on the client machine and the executors run in the cluster managed by YARN. See it in the original article.)
Figure 7: client mode. If the client machine goes away, the application stops.
(Diagram: Spark cluster mode diagram: the job is submitted to YARN, and both the driver and the executors run inside the cluster. See it in the original article.)
Figure 8: cluster mode. The driver runs inside the cluster, so the job survives the client disconnecting.
Strictly, local is a master setting rather than a deploy mode, but it is the third way people run Spark. Since Spark 3.4, Spark Connect adds a thin-client option: your Python program sends DataFrame plans to a remote Spark server over gRPC instead of running a driver itself.
Spark architecture interview questions
In a Spark application, which component coordinates task execution and schedules tasks across executors?
The driver program. The cluster manager only allocates resources (it starts the executors); worker nodes host the executors; Spark SQL is a module, not a scheduler.
What is the difference between the driver and an executor?
The driver plans and schedules the work and collects results; executors run the tasks and hold cached and shuffle data. There is one driver per application and usually many executors.
What triggers a new stage?
A shuffle, caused by a wide transformation such as groupBy with an aggregation, a non-broadcast join, distinct, orderBy or repartition.
How many tasks does a stage have?
One per partition of the data that stage processes.
How is a job related to an action?
Every action starts at least one job; transformations alone never start a job.
What makes Spark fault tolerant?
Lineage: the DAG records how every partition was built, so a lost partition is recomputed from its parents. Failed tasks are retried, and lost shuffle output is regenerated by re-running the stage that produced it.
Why is a Python UDF slower than a built-in function?
Built-in functions run inside the JVM executors. A Python UDF sends rows to Python worker processes and back, which costs serialisation time; pandas UDFs reduce that cost by moving data in Arrow batches.
Practise next
- Broadcast joins and skew joins in PySpark: how to avoid the shuffle of a join.
Repartition vs coalesce: control the number of partitions and tasks.
PySpark practice challenges: solve real problems in the browser.
Online PySpark compiler: run your own Spark code with no setup.
Edited with AI assistance; all code was run on Spark 3.5.
This article was first published on PySpark.in, where every code example runs in the browser on a real Spark session.
