How to Design Trustworthy AI Systems with UX

TL;DR

Trustworthy AI must augment human judgment while keeping people safe, in control, and responsible for decisions. Teams should address ethics and bias early, align around shared technical principles, examine whether training data reflects discrimination or limited examples, and evaluate the complete human interaction rather than focusing only on algorithms or last-minute ethical dilemmas.

Transcript

Yeah. Great. We're gonna get, get started. Good morning. Happy Friday. I'm glad so many of you came out on the last day of RSA, and the last presentation until the closing. Some quick, quick legal disclaimers, and then I'll move into the topic. So I'm gonna be talking about artificial intelligence and how we can make it more trustworthy. And artifi... Read More

Key Insights

• Artificial intelligence is an augmentation tool, not a replacement for human responsibility. Teams must design systems that support human decisions while keeping people safe and in control, regardless of how intelligent the technology may appear to be.
• Biased historical data produces biased artificial intelligence outcomes. Mathematical processing cannot change the underlying truth of discriminatory records, so an algorithm trained on harmful past behavior can reproduce that behavior instead of reducing it.
• Resume screening can discriminate when employee data is treated as a model of ideal applicants. If existing employees are disproportionately male, white, or Asian, indicators associated with women can be learned as negative attributes during automated applicant evaluation.
• Mortgage algorithms can perpetuate racism when they learn from lending records shaped by prior discrimination. Historical patterns of higher interest rates for Black and Latino borrowers can become inputs that continue unequal treatment in later automated decisions.
• Criminal justice predictions can reflect unequal policing and incarceration rather than individual behavior. Systems trained on those records may assume that Black people will commit more crimes and that white people present less risk, reinforcing existing racial bias.
• Limited training examples can prevent a system from recognizing valid cases outside its learned range. A model taught to identify only red, green, and white cars may fail when it encounters blue cars, demonstrating bias and overfitting.
• Technology ethics is early, purposeful development work. It should shape the system before launch instead of appearing only as a hypothetical decision at the moment of action, and it must address conduct and human interactions as well as code.
• Bias is a manageable mental shortcut shaped by experience, culture, education, resources, race, gender, sexuality, theology, and tradition. Awareness helps people examine whether automatic assumptions match their conscious beliefs and change how they interpret situations.

Questions & Answers

Q: How can teams design trustworthy AI systems?

Teams can design trustworthy AI by treating artificial intelligence as support for human judgment, keeping people safe and in control, and preserving human responsibility for decisions. Ethical work should start early, before the system is functioning or launched. Teams should examine algorithms, data, conduct, interactions, intended benefits, prohibited behavior, and progress toward shared principles throughout development.

Q: Why does biased data create biased AI outcomes?

Biased data creates biased outcomes because an algorithm learns patterns already embedded in its training records. Mathematics does not correct the truth represented by those records. If past decisions reflect racism, sexism, unequal policing, or other discrimination, a system trained on them can reproduce and confirm those patterns when evaluating new people or situations.

Q: How can AI resume screening discriminate against women?

An automated resume system can discriminate when it learns from a workforce that does not represent the full applicant population. In the example discussed, Amazon employees tended to be male and white or Asian. The system consequently treated phrases such as women's chess team or women's college as negative attributes, reproducing the imbalance contained in its employee data.

Q: Why can mortgage approval algorithms reproduce racism?

Mortgage algorithms can reproduce racism when historical lending data contains the effects of racial discrimination. The transcript describes Black and Latino borrowers receiving higher interest rates in the past. When an automated system uses those records to determine new rates or approvals, it can continue the same unequal treatment because the discriminatory pattern remains embedded in the data.

Q: What is wrong with using AI to predict future crime?

Crime prediction systems can learn from records shaped by unequal policing and incarceration. If certain neighborhoods are policed more and Black people are incarcerated more frequently, historical data can make those patterns appear predictive. The resulting system may release white people more quickly and assume Black people will commit more crimes, while also failing to consider individual differences.

Q: How does limited training data affect AI recognition?

Limited training data can make a system overfit to the examples it has seen and fail on valid examples outside that range. The transcript illustrates this with a model trained to recognize only red, green, and white cars. When it later encounters blue cars, it may not identify them as vehicles because that variation was absent from training.

Q: When should technology ethics be applied to AI development?

Technology ethics should be applied early and purposefully, long before a functioning system reaches the moment of decision or is launched. It is not merely a last-minute philosophical question like choosing between outcomes in a trolley problem. Ethical development should guide data selection, algorithms, team conduct, human interactions, system behavior, values, limits, and progress tracking.

Q: What role does bias awareness play in responsible AI?

Bias awareness helps team members recognize how their worldviews shape data, designs, and solutions. Those worldviews can reflect social class, resources, education, race, gender, sexuality, culture, theology, tradition, and personal experience. Because bias is a mental shortcut rather than an unchangeable rule, people can examine it, compare it with conscious beliefs, and manage its influence through collaboration.

Summary & Key Takeaways

• Historical data can reproduce historical discrimination when used to train artificial intelligence. Resume screening, mortgage lending, and criminal justice examples show that algorithms do not correct biased records automatically. Teams must inspect how data was created, whose experiences it represents, and which harmful patterns a system could perpetuate before deployment.

• Technology ethics consists of shared standards for right and wrong that guide practical development work. Ethical analysis should begin long before a functioning system makes decisions. It must cover algorithms, team conduct, human interactions, system behavior, organizational values, prohibited actions, and methods for tracking progress toward responsible outcomes.

• Trustworthy AI development requires awareness of human bias and collaboration across different worldviews. Bias functions as a mental shortcut and is not automatically aligned with conscious beliefs, but people can recognize and manage it. Shared principles help diverse teams question assumptions, balance competitive pressure, and keep humans responsible and in control.