DataHub 201: Impact Analysis

TL;DR

This video discusses the impact analysis feature in Data Hub, using a fictional company's data stack as an example.

Transcript

so if y'all have been joining town halls for a while every once in a while we'll have data hub 101 where we kind of do a deep dive into a specific feature um this time around we're going to do some advanced functionality and advanced use cases within data hub and today we're going to talk about impact analysis so um for the sake of this we are goin... Read More

Key Insights

• 🔍 The content discusses the complexity of modern data stacks, with various teams utilizing different databases and tools for data storage, analytics, and transformation.
• 😮 Long Tail Companions, a fictional company, is used as an example to demonstrate the challenges of making schema changes in a production application and understanding their downstream impact.
• 🔎 Data Hub offers an impact analysis feature that allows users to proactively understand the potential impact of making changes to a specific data set.
• 📊 Through Data Hub's impact analysis workflow, users can search for a data set and explore its lineage and dependencies across various platforms and tasks.
• 📈 The impact analysis feature provides insights by showing the number of dashboards, charts, data sets, and tasks that may be affected by a change, allowing users to identify potential issues.
• 💼 Users can obtain a summary of entities affected by the change, including their names, owners, tags, terms, and explicit links, enabling proactive communication and collaboration.
• 💪 The impact analysis feature can save significant time and effort in researching and understanding dependencies by providing a centralized and comprehensive view of potential impacts.
• 👍 Access to impact analysis within Data Hub can greatly benefit data professionals, potentially saving years of work and facilitating smoother data management and decision-making processes.

Questions & Answers

Q: How does Data Hub help with impact analysis in a complex data stack?

Data Hub's impact analysis feature allows users to identify and understand the downstream effects of making changes to data sets in a complex data stack. It provides a comprehensive view of the dashboards, charts, data sets, and tasks that may be impacted by a change, along with contact information for the owners of those resources. This enables proactive communication and collaboration to avoid adverse reactions and ensure smooth data management.

Q: What are the key components of Long Tail Companions' data stack?

Long Tail Companions' data stack consists of multiple teams and technologies. The adoptions team uses Postgres and MongoDB for application storage, the e-commerce and data science team leverages Kafka for click stream analytics, and a data platform team syncs data from all source systems to S3 using Airflow jobs. Spark is used for data transformation and processing, and Snowflake is the central warehouse. The analytics engineering team utilizes Snowflake, DBT, and Looker for data transformation, validation, and surfacing.

Q: How does the impact analysis feature in Data Hub help in identifying impacted dashboards?

Data Hub's impact analysis feature allows users to search and select a specific data set or entity for analysis, such as the pet profile data set in MongoDB. By analyzing the lineage and downstream transformations, it identifies 17 Looker dashboards and provides a cohesive list of entities that might be impacted. Users can quickly download a summary with information about the dashboard owners and contact them for further collaboration and communication.

Q: What additional information does Data Hub provide for impacted entities?

In addition to identifying impacted dashboards, Data Hub provides information about the owners of those resources, including their contact details. It also provides information on tags, terms, and explicit links to the entities. This allows users to easily collaborate, conduct further research, or hand off the information to relevant stakeholders, streamlining the impact analysis process.

Q: How can impact analysis in Data Hub save time and effort?

Impact analysis in Data Hub can save significant time and effort by providing a comprehensive view of the downstream effects of making changes to data sets. Instead of spending years researching the impacts manually, users can quickly identify the dashboards, charts, data sets, and tasks that may be affected. The ability to proactively communicate and collaborate with relevant stakeholders further expedites the process, making data management more efficient.

Summary & Key Takeaways

• The video introduces Long Tail Companions, a fictional company with a highly fragmented data stack.

• It highlights the different teams and technologies used by Long Tail Companions for data storage, analytics, and processing.

• The video emphasizes the importance of impact analysis in understanding the downstream effects of making changes to data sets, and demonstrates how Data Hub's impact analysis feature can help identify and communicate these impacts.