The human insights missing from big data | Tricia Wang

TL;DR

In this TED Talk, the speaker discusses the limitations of big data and the importance of integrating it with qualitative data to make better decisions.

Transcript

In ancient Greece, when anyone from slaves to soldiers, poets and politicians, needed to make a big decision on life's most important questions, like, "Should I get married?" or "Should we embark on this voyage?" or "Should our army advance into this territory?" they all consulted the oracle. So this is how it worked: you would bring her a question... Read More

Key Insights

• 🔮 The desire for prophecy and the need to make the right decision has led to the rise of big data and its use as an oracle.
• 🐶 The big data industry is worth $122 billion, but the returns on investment are surprisingly low, with over 73% of projects not being profitable.
• 📊 Quantifying specific environments has been successful in areas such as power grids and genetic code, but not all systems that involve human beings can be neatly contained and predicted.
• 📱 Nokia's failure to recognize the rise of smartphones in low-income markets despite qualitative evidence highlights the limitations of relying solely on big data.
• 🔎 Integrating big data with qualitative, thick data can provide a more complete picture and lead to significant insights, as seen in Netflix's success in understanding binge-watching behavior.
• ⚖️ The quantification bias, valuing the measurable over the immeasurable, can lead to blind spots and the misinterpretation of data.
• 💡 Integrating big and thick data can transform businesses and even have life or death implications, but it requires a commitment to make better data, algorithms, outputs, and decisions.
• 🔐 Avoiding the risk of missing something requires acknowledging the limitations of big data and utilizing qualitative insights to enhance predictive models.

Questions & Answers

Q: What is the significance of oracles in ancient Greece?

Oracles played a crucial role in ancient Greece, as they were consulted by people from all walks of life, including slaves, soldiers, poets, and politicians, for guidance on important decisions. They would enter a trance-like state and provide predictions as answers to questions posed to them.

Q: How do people today consult an oracle?

Nowadays, people consult what can be considered as a new oracle called big data, which includes technologies like Watson, deep learning, and neural networks. They rely on these systems to find answers to questions such as efficient shipping methods, predicting sales volumes, or estimating the likelihood of genetic disorders in children.

Q: What are the challenges associated with using big data?

Despite the significant investment in big data, many projects are not profitable. Over 73% of big data projects fail to deliver better decision-making or breakthrough ideas. Using big data is challenging because it requires understanding the complexities of human behavior, which is not always quantifiable. Additionally, relying solely on big data can create an illusion of knowing everything while increasing the chances of missing important factors.

Q: How can organizations improve their use of big data?

Organizations need to integrate big data with what the speaker refers to as "thick data." Thick data refers to qualitative information, such as stories, emotions, and interactions, that cannot be quantified. By combining big data's scale and machine intelligence with thick data's human insights, organizations can gain a more complete understanding of their data and make better decisions. Examples, like Netflix's integration of thick and big data to enhance their recommendation algorithm and transform the viewer experience, highlight the power of this approach.

Q: Why is integrating big data and thick data crucial?

Integrating big data and thick data is crucial because it helps organizations see what is missing in their models and understand why certain patterns or behaviors occur. While big data provides insights on a broad scale, thick data offers depth and context by considering the human narrative. This integration allows for the exploration of why certain phenomena happen and enables organizations to make more informed decisions that consider both quantitative and qualitative aspects.

Summary & Key Takeaways

• People have always sought prophecy and guidance to make decisions, from ancient oracles to modern big data systems.

• Big data is a $122 billion industry, but over 73% of projects are not profitable and do not lead to better decision-making or breakthrough ideas.

• The integration of big data and thick data (qualitative data from human experiences) can lead to more accurate insights and better decision-making, as demonstrated by companies like Netflix.