Fabric Spark Settings EXPLAINED!

In this training session, Zane Goodman from Pragmatic Works walks learners through one of the most important areas of Microsoft Fabric: Spark settings. While Spark is a widely discussed topic, Zane focuses on helping users understand how nodes, executors, pools, environments, and configuration options work together to support efficient data processing in Fabric.

Core Apache Spark Concepts

Before exploring Fabric-specific configurations, Zane first breaks down the fundamental components that define Apache Spark:

• Nodes: Virtual machines similar to a standard computer, each with allocated CPU and memory resources.
• Parallel Processing: Spark uses multiple nodes (e.g., five nodes instead of one) to run tasks simultaneously.
• Executors: Objects running inside nodes that process assigned tasks using the node’s compute power.
• Driver Node: The coordinator that receives the job from the Fabric notebook, partitions the data, and assigns tasks to executors.

Zane explains how Spark divides work into smaller partitions and distributes execution across executors. Once processing finishes, Spark brings all partitions back together to produce a final dataset or display result in the notebook.

Navigating Fabric Workspace Spark Settings

Inside the Fabric workspace, Spark settings can be configured to manage performance, cost, and scalability. Zane demonstrates where these options live and how they impact Spark sessions.

The Starter Pool

Fabric workspaces come with a default Starter Pool, designed for fast session startup—typically around five seconds. This is possible because its nodes are prehydrated and ready for immediate use. Key characteristics include:

• Preallocated node resources
• Memory‑optimized node family
• Preset node size options (small, medium, large, etc.)

While many settings in the Starter Pool cannot be modified, Zane highlights the importance of understanding node size because each size uses a specific number of virtual cores (vCores). One capacity unit (CU) equals two vCores, so an F64 capacity provides 128 vCores. This vCore availability determines how many nodes a workspace can provision and how large those nodes can be.

Customizing Node and Executor Counts

Users can configure:

• Node Count: How many nodes Spark should allocate.
• Executor Count: How many executors will run tasks (always one fewer than total nodes because one node must act as the driver).

Zane demonstrates how increasing or decreasing node count automatically adjusts the maximum executor count. Choosing appropriate values matters because each node consumes workspace capacity.

Environments and Custom Compute Settings

Fabric allows users to create Environments—a powerful feature for managing packages, libraries, and shared settings across notebooks. When enabled, users can:

• Install custom libraries
• Share configurations across multiple notebooks
• Define Spark settings at the environment level

This promotes consistency within teams and avoids repetitive setup for each new notebook.

Creating Custom Spark Pools

Zane walks through the “New Pool” option, covering advanced settings such as:

• Autoscale: Fabric automatically adjusts the number of nodes based on workload activity.
• Dynamically Allocated Executors: Spark determines how many executors are needed based on task volume.

Autoscale helps reduce unnecessary node usage, saving cost. Dynamic executor allocation ensures Spark doesn’t allocate more compute than required, improving performance efficiency.

Job Settings and High Concurrency

The Jobs section includes additional important settings:

• Reserve Maximum Cores: Ensures critical jobs get priority by reserving the full compute power needed.
• Spark Session Timeout: Automatically ends idle sessions to free capacity.
• High Concurrency Mode: Allows multiple notebooks or pipeline notebooks to share the same Spark application, reducing startup delays.

The Bursting Feature (Capacity Level Setting)

Zane highlights a newer and potentially risky setting: Allow Job Bursting. When activated, Fabric can temporarily triple the available vCores for large jobs. While this accelerates processing, it can also exhaust capacity quickly and block other users’ jobs.

This setting is powerful but should be carefully managed—especially in shared enterprise environments.

Final Takeaways

Zane concludes by emphasizing the importance of understanding both Spark fundamentals and Fabric-specific settings. These configurations directly impact performance, cost, and collaboration within a workspace. Whether users are working with starter pools, custom environments, or high-concurrency pipelines, knowing how Spark allocates compute ensures smooth and efficient operations across data engineering workloads.

Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Fabric and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.