Why Artificial Intelligence Needs Legal Counsel

TL;DR

Artificial intelligence needs legal oversight because learning systems can produce harmful, biased, or noncompliant decisions after deployment. Organizations should involve general counsel and IT management throughout AI projects, explicitly include laws, regulations, contracts, privacy duties, and user expectations in system specifications, and continuously test whether outputs reflect reality and organizational goals.

Transcript

So I think, uh, we'll get started. Okay, great. Thank you guys for joining us, um, on a Friday. Everybody enjoyed the conference so far? Yep. Yes. Good sessions, good networking. Did anybody walk the expo floor four or five times? Got lost at least twice, right? "Wait, where am I?" Well, thank you so much for joining us on this last day early in th... Read More

Key Insights

• Artificial intelligence is used throughout business for cybersecurity threat hunting, security operations, supply-chain activities, fraud detection, and automated analysis. Its value comes from processing large volumes of information and augmenting human judgment, especially when analysts cannot efficiently correlate every incoming signal themselves.
• Traditional programming follows defined rules whose operation can be monitored for legal and regulatory compliance. Machine-learning systems differ because they learn from data after developers establish the algorithm, creating uncertainty about what the system may infer and whether its resulting behavior will remain compliant.
• AI projects should be designed to avoid identifiable and preventable problems. Organizations should not adopt the technology merely because competitors use it or because they want to get ahead of attackers. Each implementation needs a defined purpose and a deliberate assessment of added risk.
• Legal and technical collaboration is limited despite widespread concern. In Kroll and Duff & Phelps' global survey, 93% of general counsels expressed concern about security and privacy surrounding IT, but only about one third reported regular meetings between the general counsel's office and IT management.
• System specifications determine which obligations an AI system can account for. If laws, regulations, contractual duties, privacy requirements, and organizational expectations are absent from the specification, developers cannot safely assume the finished system will recognize or follow those constraints on its own.
• Automated systems can satisfy a narrow technical goal while causing serious unintended harm. A Close-In Weapons System successfully directed fire toward threats, yet shells that missed continued traveling during one exercise, landed on another ship's bridge, and killed multiple people.
• Historical data can cause an AI system to reproduce problematic patterns instead of producing genuinely objective decisions. A financial-services system trained on 250,000 prior loan records learned that postal code was associated with repayment and began weighting approvals or denials according to where borrowers lived.
• Cybersecurity AI must be evaluated for accuracy, bias, privacy, and compliance, not only speed or analytical power. A system that highlights false positives, misrepresents reality, or examines information in an unlawful way can create additional risk even while making security analysis more efficient.

Questions & Answers

Q: Why does artificial intelligence need legal counsel?

Artificial intelligence needs legal counsel because its operation is constrained by laws, regulations, contractual obligations, privacy duties, and user expectations that may not be obvious to technical developers. Since a learning system can develop decision patterns after its initial algorithm is established, counsel should help ensure those constraints become explicit specifications and remain part of ongoing compliance review.

Q: How is AI different from traditional programming?

Traditional programming generally applies a known set of rules, such as defined if-then logic, so organizations can monitor how the program functions and check it against legal or regulatory requirements. Artificial intelligence and machine learning can teach themselves from supplied data. That learning process creates uncertainty about future decisions, hidden bias, and whether outputs will continue following required rules.

Q: Where is artificial intelligence used in business?

Artificial intelligence is used across business, with examples including cybersecurity threat hunting, security tools, security operations centers, supply chains, and fraud detection. In cybersecurity, it helps process and correlate the large volume of attacks and information reaching analysts. It can automate overlooked hygiene functions and strengthen analysis, but its conclusions still require scrutiny for accuracy and compliance.

Q: What should organizations include in an AI system specification?

An AI system specification should explicitly include the laws, regulations, contractual obligations, privacy requirements, organizational goals, and user expectations that govern its operation. The presentation emphasizes that requirements omitted from the specification should not be expected to appear automatically in the finished system. Legal counsel and technical teams therefore need regular communication while requirements are defined and reviewed.

Q: How can training data introduce bias into lending decisions?

Training data can cause a system to discover correlations that influence decisions without evaluating whether those correlations are appropriate. In the financial-services example, an AI system reviewed 250,000 prior loan records and learned that postal code was connected to repayment. It then gave greater weight to borrowers' locations when approving or denying loans, undermining the intended objectivity.

Q: What does the Close-In Weapons System example show about AI risk?

The Close-In Weapons System example shows that successful optimization of one objective does not guarantee overall safety. Its radar and gun were designed to make bullets converge on incoming threats, but the consequences of missed shells were not adequately considered. During an exercise, continuing shells reached another ship's bridge and killed multiple people, demonstrating the danger of incomplete specifications.

Q: How should AI be evaluated in cybersecurity operations?

Cybersecurity AI should be evaluated for the accuracy of its intelligence, the presence of bias, its handling of privacy, its treatment of false positives, and its compliance with applicable rules. Although AI can help a security operations center correlate overwhelming amounts of information, efficiency alone is insufficient if the system misrepresents reality or creates new legal and privacy risks.

Q: Why should general counsel meet regularly with IT management?

Regular meetings allow general counsel and IT management to connect technical design choices with security, privacy, regulatory, and contractual obligations. The cited global survey found that 93% of general counsels were concerned about security and privacy surrounding IT, yet only about one third had regular meetings with IT management. That gap leaves many organizations worried without taking coordinated action.

Summary & Key Takeaways

• Artificial intelligence is increasingly used across business, including cybersecurity threat hunting, security operations, supply chains, and fraud detection. These systems help process large volumes of information and support human analysis, but their ability to learn after initial programming makes their future behavior less predictable than traditional rule-based software.

• The central risk is that an AI system may optimize its assigned objective while ignoring consequences that developers never included in its specification. Dangerous automation, privacy invasion, social manipulation, inaccurate security conclusions, and decisions that conflict with laws, regulations, contracts, or user expectations can result from incomplete requirements.

• Effective AI governance requires sustained cooperation between technical teams and legal counsel. General counsel should help translate legal and contractual obligations into system requirements, while developers should examine training data, outputs, bias, false positives, and unintended effects. Organizations should adopt AI to solve defined problems, not merely follow competitors.