Chris Re

TL;DR

Deep Dive is a dark data system that extracts and transforms unstructured data into structured databases with higher quality than human annotators, using scalable inference engines.

Transcript

perfect thanks yeah so I'm excited to tell you about some work that we've been doing over the last couple lab which is on a system called deep dive which is what we call a dark data system now during this talk it won't have time to sort of parcel out credit to all the people who deserve it there are a bunch of people on this team and a bunch of peo... Read More

Key Insights

• 🕶️ Dark data systems, like Deep Dive, can extract structured data from unstructured sources, providing valuable insights and analysis opportunities.
• 🕶️ Recent advancements in dark data systems have improved the quality and efficiency of data extraction, integration, and cleaning processes.
• 😒 The use of scalable inference engines and relaxation of consistency in statistical algorithms enables faster and more efficient data processing.
• ✋ Deep Dive aims to make its system accessible to non-computer scientists through a high-level programming language and abstraction of underlying algorithms.
• ✋ Deep Dive has shown to have higher quality and reliability compared to human annotators in various applications.
• 🚒 The scalability and parallel processing capabilities of modern hardware enhance the performance of dark data systems and inference engines.
• 💁 Dark data systems can be used in various domains, such as climate and biodiversity, healthcare, and law enforcement, for extracting and integrating massive amounts of information.

Questions & Answers

Q: What is Deep Dive and how does it work?

Deep Dive is a dark data system that extracts and transforms unstructured data into structured databases. It uses probabilistic or statistical inference to build pipelines and determine where to spend effort for higher application quality.

Q: How does Deep Dive compare to human annotators?

Deep Dive has been compared to human annotators in various applications and has shown to have higher quality in extracting, integrating, and cleaning data. It can also process data in a shorter amount of time and with more reliability.

Q: Who can use Deep Dive?

Deep Dive aims to make its system accessible to people who are not computer scientists. It has raised the level of abstraction in programming, allowing users to focus on specifying features and random variables without needing to understand the underlying algorithms.

Q: Can Deep Dive handle large-scale data processing?

Yes, Deep Dive has developed scalable inference engines that can handle massive amounts of data and make use of modern hardware's parallel processing capabilities. These engines allow for faster and more efficient data processing.

Summary & Key Takeaways

• Deep Dive is a dark data system that extracts unstructured data, such as emails and web pages, and transforms it into structured databases.

• Recent advancements in dark data systems have improved the quality of extraction, integration, and cleaning processes, surpassing human annotators.

• Deep Dive aims to make its system accessible to non-computer scientists and has developed a high-level programming language to simplify the process.