What is SDF?
SDF is a next-generation build system for data infrastructure that compiles all data contracts, queries, and policies at once to build code-level, time-level, and access-level dependencies.
SDF Features:
Scalability: It is the first scalable build system for data infrastructure.
Local execution: It enables local execution of SQL development with intelligent workflows and data governance packed into 60MB of Rust-based magic.
Data governance: It enables precise column-level lineage analysis that generates a full-featured data catalog out of the box.
Easy to use: It is a powerful and easy-to-use data development platform that enables organizations at any scale to fully leverage their data while respecting all relevant policies and protections.
Cloud-based: It runs its soon-to-be open-source engine in the cloud to take your deployments from a single command to an automatically scheduled workflow with policies enforced in real-time.
SDF Benefits:
Data Consistency: It ensures that data is organized in a standardized structure, making it easier to maintain consistency across datasets, which is essential for accurate analysis and reporting.
Interoperability: It formats allow data to be easily shared and accessed across different systems and platforms, promoting smooth data exchange between various applications and technologies.
Ease of Data Processing: Structured data in this format simplifies processing by making it easy to perform automated tasks like parsing, filtering, and analysis, reducing the need for manual intervention.
Use cases:
Build code-level, time-level, and access-level dependencies for data infrastructure.
Compile data contracts, queries, and policies at once to streamline SQL development.
Generate a full-featured data catalog out of the box with precise column-level lineage analysis.
Enable organizations at any scale to fully leverage their data while respecting all relevant policies and protections.
Schedule workflows with policies enforced in real-time.