The Future of the Modern Data Stack | PAKCon 2021

TL;DR

Discussing modern data stack trends, including modularity, productivity, data governance, and on-premise vs. cloud deployment.

Transcript

ready to go cool and we're off uh welcome everyone also welcome alastair who's joining us remote to this panel on the future of the modern data stack um so very much looking forward to to discuss this in the round for the next 40 minutes or so i wanted to give us just to get us all on the same page it's sort of a bit of a primer on what the modern ... Read More

Key Insights

• 😤 Data teams face challenges in handling data volume, complexity, data lineage tracking, and data governance in the modern data stack.
• 🥅 Decentralization through approaches like the data mesh concept may not always align with the goal of creating a single source of truth for data.
• 😤 Reusable code, good documentation, and engineering discipline are crucial for effective collaboration and knowledge sharing in data teams.
• 💿 Advanced tools like CI/CD practices, staging environments, and testing mechanisms can enhance reliability and trust in data infrastructure.
• 😶‍🌫️ The debate continues between on-premise and cloud deployment, balancing factors like security, compliance, and cost for data stack implementations in organizations.
• 😤 Collaboration among data science, engineering, and product teams is essential to drive value creation and meaningful insights from data products.

Questions & Answers

Q: What are the key principles of the modern data stack?

The modern data stack emphasizes modularity, a best-of-breed approach, and productivity enhancements borrowed from software engineering practices.

Q: How do organizations tackle the challenges of handling data volume and complexity?

Organizations implement data governance, metadata management, and tools like stream processing platforms to handle data volume and complexity effectively.

Q: What are the benefits of data governance and metadata management?

Data governance and metadata management ensure data security, compliance, lineage tracking, and optimize data usage and decision-making processes within organizations.

Q: What is the debate between on-premise and cloud deployment in the modern data stack?

Organizations face the dilemma of choosing between on-premise and cloud deployment based on factors like security, control, compliance, and cost, with a shift towards cloud solutions for scalability and accessibility.

Summary & Key Takeaways

• Modern data stack principles include modularity, best-of-breed approach, and productivity enhancements from software engineering practices.

• Challenges include handling data volume, complexity, and the need for data governance and metadata management.

• Discussion highlights the debate between batch and streaming processing, the role of analytical databases, and the on-premise vs. cloud debate.