Streaming Caffeine E10: Ozan from Synnada, about Arrow Data Fusion, Rust, Database, SQL, AI

TL;DR

Learn about Data Fusion, an AI-native data infrastructure, that simplifies the integration of AI into data systems.

Transcript

hi everyone this is Jo welcome to the new episode of streaming Cafe in I even have this special t-shirt I'm happy to share with with you U and today I'm so gladed to invite my friend ozen from uh Sada and uh the co-founder and the CEO so o do you mind sharing little bit about yourself and your comp first yes um well thank you for inviting me first ... Read More

Key Insights

• 🚀 Data Fusion aims to provide an AI-native data infrastructure, abstracting away the complexities of embedding AI into applications and handling tasks like updating and optimizing models. It aims to provide the same comfort as Cloud-native environments for AI applications.
• 💡 Lambda architecture, which separates batch and streaming data, is a burden on data teams and often requires integrating multiple technologies. Data Fusion seeks to solve this problem by offering a unified solution for batch and streaming data processing without the need for multiple systems.
• 🔧 Arrow and Data Fusion are heavily relied upon by Sinada to build a solid and efficient database. Arrow provides a standard for representing data, while Data Fusion offers a query engine that allows for efficient and flexible data processing.
• ⚙️ The deconstructed architecture of Data Fusion allows users to mix and match components to build the data system that suits their needs. It offers extensibility and customization, allowing users to define their own functions and optimize their system accordingly.
• 🌐 The Arrow ecosystem, including Data Fusion, is gaining traction in various areas, not just databases. It is being used in visualization applications and can be a foundation for building different data systems.
• 🌐 By leveraging the modular nature of Data Fusion, developers can build high-performance data systems without sacrificing extensibility. It allows for the construction of complex systems while keeping them relatively simple and easier to operate.
• ⚡ AI-powered applications can benefit from a unified SQL interface for both batch and streaming data processing. This allows for easier interaction with databases and simplifies the development of AI applications that span both data processing modes.
• 📺 There are plans to organize a Data Fusion event in the future, where developers, startup founders, and engineers can come together to discuss Data Fusion's roadmap, use cases, and future applications. The community is also active on Discord, providing a platform for users to connect and learn more about Data Fusion.

Questions & Answers

Q: What is AI-native data infrastructure?

AI-native data infrastructure refers to a system that simplifies the integration of AI into data systems, allowing for seamless model updates, model observation, and model optimization.

Q: How does Data Fusion leverage the Arrow ecosystem?

Data Fusion utilizes Arrow data fusion and Arrow Flight to create a strong and efficient database foundation, enabling efficient data representation, interchange, and processing.

Q: What is the advantage of using Data Fusion in building data systems?

Data Fusion offers modularity and extensibility, allowing users to select and customize the components they need for their data system, resulting in simplified and efficient development.

Q: How does Data Fusion address the batch and streaming dichotomy in data systems?

Data Fusion optimizes query plans to make them streamable, leveraging data ordering and avoiding pipeline breaks, enabling the same query to run efficiently on both batch and streaming data.

Q: Can user-defined functions be incorporated into Data Fusion?

Yes, Data Fusion allows users to define their own aggregations, window functions, and more, providing flexibility and customization options.

Q: Is Data Fusion compatible with web assembly (Wasm)?

Data Fusion can be compiled to Wasm, allowing it to run in a web browser. However, there may be limitations regarding threading support that need to be addressed.

Q: Are there any upcoming conferences or events related to Data Fusion?

The Data Fusion community is actively using Discord as a platform to connect and share information. Additionally, there are plans to organize a Data Fusion event in the future, which will provide an opportunity to learn more about the project and its applications.

Summary & Key Takeaways

• Data Fusion aims to be an AI-native data infrastructure that abstracts away the complexities of embedding AI into applications.

• The system utilizes the Arrow ecosystem, including Arrow data fusion and Arrow Flight, to create a solid and efficient database foundation.

• Data Fusion deconstructs the fundamental components of data systems, allowing users to mix and match functionalities and customize their own data system.