Why Sail?
When Spark was invented over 15 years ago, it was revolutionary. It redefined distributed data processing and became the backbone of data infrastructure for companies across every major industry.
For over a decade, it has powered everything from ETL to machine learning pipelines at scale. But as real-time demands increase, cloud costs rise, and AI workloads evolve, Spark’s architecture is showing its age.
Due to its JVM foundation, Spark struggles with latency, scalability, and operational complexity. This results in higher cloud expenses, slower product cycles, and increased operational overhead.
Our open-source framework, Sail, built natively in Rust, eliminates these problems entirely.
- Rust-native engine with memory safety
- Spark Connect compatibility
- Lightning-fast Python UDFs
- Stateless and lightweight workers
- Columnar format and zero-copy data transfer
- 2–8× faster execution
Runtime
Predictable Execution Times
Built in Rust, Sail adopts deterministic memory management. Compute operations are not interleaved with garbage collection pauses, resulting in more consistent task completion times with far fewer tail latency spikes.
Sail ensures low memory management overhead and predictable execution times, which reduces risk, complexity, and costs for teams delivering time-sensitive workloads.
Execution Speed
Native Performance with Columnar Format
Sail leverages the Apache Arrow in-memory format and the Apache DataFusion query engine. The columnar in-memory format allows SIMD instructions to process multiple data records in a single CPU cycle, yielding higher throughput per core. In contrast, JVM-based and row-based solutions add layers between the code and the metal, process data records in loops, and limit the performance that can be extracted from the hardware.
Sail consistently delivers 2× to 8× faster execution times, translating to shorter time-to-insight and lower resource usage.
Data Flow
Zero-Copy Data Transfer & No Serialization
The Sail process embeds a Python interpreter to execute Python UDFs (User-Defined Functions). No data serialization or copying occurs between built-in operations and your custom Python code. Sail workers in a cluster exchange data using the Arrow format with no data serialization between query execution stages.
Python UDFs are highly performant in Sail. Join and aggregation operations in Sail also come with low data shuffling overhead.
Cloud Efficiency
Lightweight Workers that Scale Instantly
The Sail process starts within seconds and consumes only a few dozen megabytes of memory when idle. In cloud environments where elasticity is essential, Sail reduces the need for capacity planning and manual tuning compared to JVM-based solutions with resource-intensive executors.
Sail empowers businesses to achieve dramatically lower cloud infrastructure costs and a smoother experience, especially in containerized environments.
Safety & Reliability
Memory Management & Concurrency You Can Trust
Sail benefits from Rust’s unique approach to memory management. The rules enforced at compile time eliminate whole categories of memory and concurrency bugs. Sail’s internals have unparalleled robustness compared to JVM-based solutions.
Sail reduces production risk, debugging time, and operational costs by offering a solid engine for your data needs.
Compatibility
Migration Made Easy
Your Spark session acts as a gRPC client that communicates with the Sail server via the Spark Connect protocol. With Sail, there’s no need to rewrite your Spark applications. You can immediately deploy Sail in shadow mode for your production pipelines or migrate your workloads incrementally.
Sail removes barriers for teams to modernize their data stacks. Switching to Sail can be a straightforward business decision.