Data Engineering at Project A - Dr. Martin Loetzsch explains what to expect as a job candidate | Summary and Q&A

TL;DR

Project A's data engineering and data science team explains their approach to building data-driven applications, emphasizing the importance of simplicity in technology and the diverse learning opportunities they encounter.

Key Insights

• 😤 The data engineering and data science team at Project A focuses on building data-driven applications for their portfolio companies, including data warehouses and predictive models.
• 👨‍💼 They benefit from working with different companies, as it provides them with exposure to diverse business models, tech stacks, and company cultures.
• 😤 Simplicity in technology choices is prioritized to facilitate easy adoption and maintenance by small teams or individuals within the portfolio companies.
• 🤗 Project A prefers open-source tools and uses Postgres extensively for data integration work.
• 🤙 They have developed their own tool, called "mara," to orchestrate SQL and Python scripts, finding it to be a simpler and more suitable alternative to airflow.
• 🏃 Over 20 of their portfolio companies have successfully implemented mara.

Transcript

Read and summarize the transcript of this video on Glasp Reader (beta).

Questions & Answers

Q: How does Project A support its portfolio companies in building data-driven applications?

Project A's data engineering and data science team assists portfolio companies by developing data warehouses and creating tools such as pricing and marketing automation, and predictive models for user behavior. They provide support and expertise in these areas.

Q: What advantages does Project A have in working with multiple portfolio companies?

Working with different portfolio companies exposes Project A's team to various business models, technology stacks, and company cultures. This diverse experience enhances their learning curve and enables them to transfer solutions and knowledge from one project to another efficiently.

Q: Why does Project A prefer simpler technologies for their portfolio companies?

Project A focuses on simplicity in technology choices because they eventually hand over the software they build to small teams or even a single person within their portfolio companies. By using simpler technologies, such as open-source tools and Postgres, they prioritize ease of learning and maintenance.

Q: What alternatives does Project A use instead of airflow for orchestrating SQL and Python scripts?

Rather than using airflow, which they consider too complex for their portfolio companies' needs, Project A developed their own tool called "mara." Mara aims to strike a balance between simple scripts and airflow, and they have successfully implemented it in over 20 of their portfolio companies.

Summary & Key Takeaways

• The team helps portfolio companies by building data-driven applications, including data warehouses and tools for pricing, marketing automation, and predicting user behavior.

• They gain valuable experience and knowledge by working with different portfolio companies, exposing them to various business models, tech stacks, and company cultures.

• They prioritize simplicity in technology choices to ensure easy adoption and maintenance by small teams or individuals in their portfolio companies.