How to Protect Data in AI Systems: Security Basics

TL;DR

Protecting data for AI starts with classification: knowing whether you hold sensitive personal information, personally identifiable information, or confidential data, because you cannot protect what you have not identified. Next comes managing access through roles instead of direct access, keeping data read only where possible, applying least privilege, and enforcing identity management. Privileged users need shared IDs limited, secrets rotated in vaults, and behavior monitored for anomalies.

Transcript

Howdy everyone. If you're like me, everywhere you're turning now, you're hearing about AI this, AI that how do I get AI into automation. How do I leverage and use AI in my products. How do I use it in my business? The thing about AI is AI doesn't exist without data. You have to have data. And the thing that you need to think about is, how am I goin... Read More

Key Insights

• AI does not exist without data, so every AI initiative is also a data protection problem. Before building AI into products, automation, or business processes, organizations must decide how the underlying data will be protected as it feeds models and systems.
• Data classification is the first and most fundamental protection strategy, and it is frequently overlooked. Teams must know whether the data being extracted is sensitive personal information, personally identifiable information, or confidential information, because unidentified data cannot be properly protected.
• The core access rule is no direct access. Users, engineers, AI systems, and business applications should not hit data directly. Instead a role holds the privilege against a set of data, and entities are assigned to roles through a layer of abstraction.
• Read only access should be applied wherever possible for the consumption layer, meaning people who read and use data and AI systems that consume it. Privileged users who must change data are the exception, since read only would block their work.
• Least privilege means a user or an AI system gets access only to what is needed for the specific task being executed. If more access is required, a separate narrow role is added, and a person or AI system can hold multiple roles.
• Identity management underpins access control. Users are authenticated so their identity is known, a separate system determines what they are entitled to do, and that entitlement maps to roles. Business applications entering through certificates or APIs also need managed identities.
• Shared IDs should be limited or eliminated because a generic application ID accessible to many people destroys the ability to know who actually performed an action. Vaults and secret rotation support one to one credentials for applications and engineers.
• Data poisoning is an AI specific threat where attackers corrupt the data used for training, which in turn manipulates how the AI operates. It sits alongside familiar concerns such as database breaches, data theft, and ransomware.

Questions & Answers

Q: What is the first step in protecting data used for AI?

Classification of data comes first. It sounds like the simplest and most fundamental step, but it is often overlooked. Classification means understanding what kind of data you are extracting: whether it is sensitive personal information, personally identifiable information, or confidential information. Without knowing what kinds of sensitive information you hold, you cannot know how or what you should be protecting, so every later control rests on this foundation.

Q: How should access to data be managed in an AI environment?

Access management rests on four practices. First, no direct access: users, engineers, AI systems, and business applications should never hit the data directly. Instead a role holds privilege against a set of data, and entities are assigned to that role through a layer of indirection or abstraction. Second, keep data read only wherever possible. Third, apply least privilege. Fourth, enforce identity management so every accessing entity is known, authenticated, and authorized.

Q: What does least privilege mean for AI systems accessing data?

Least privilege means that when a user or an AI system comes in, it should not get access to everything. It should only get access to what it needs to do the job it is trying to execute. That translates into a very specific role granting access to only a few things. If something else is needed, another role is created, and a person or an AI system can be associated with multiple roles, each granting a narrow piece of information needed to perform a task.

Q: Why should shared IDs be eliminated for business applications?

An application should not use a generic ID that many people on the other end also have access to. When credentials are shared, control is lost over knowing who is actually trying to do something, which breaks identity management. The recommended approach is to limit or eliminate shared credentials on both the application and engineering sides, and to consider mechanisms such as vaults and secret rotation so each business application uses a one to one ID or credential to get in and access data.

Q: What is data poisoning in AI?

Data poisoning is a security concern introduced with AI systems. Attackers poison the data used to train models, and that corruption manipulates how the AI works. It sits alongside the older data security concerns organizations already built systems around, such as someone hacking into a database, breaching it, stealing the data, or launching ransomware against it. Because AI is trained on data, corrupting the training data becomes a way to corrupt the AI's behavior itself.

Q: How did data storage evolve before modern AI systems?

Information has existed since the beginning of human history, moving from hieroglyphs to scrolls to papers in books held in libraries. In the 1960s, as mainframes and computers entered the mainstream of business, storage became formalized through integrated data systems and information management systems, which stored data but were not very good at retrieving it. In 1970, E.F. Codd from IBM wrote the seminal paper on relational database management, giving businesses data that could be retrieved easily.

Q: What is the difference between a data lake and a Lakehouse?

Structured data historically lived in databases running on servers. As servers became overloaded, data was distributed to many servers, which evolved into cloud and then hybrid cloud. Data lakes were built on top of a small amount of data structure, providing looser storage. Organizations then realized they still wanted some of the benefits of using databases and servers, so that model was expanded into the Lakehouse, which combines lake storage with database and server benefits.

Q: Which access rules still apply to privileged users such as engineers and admins?

Privileged users include data engineers, data scientists, and admins who change and manipulate data. Read only obviously cannot apply to them, because their job is to alter data. However, least privilege still applies, and identity management still applies. In addition, shared IDs should be limited or eliminated, vaults and secret rotation should be considered, and their behavior should be monitored for anomalies, since this group holds more access privileges than users on the operational side of the stack.

Summary & Key Takeaways

• Data storage evolved from hieroglyphs, scrolls, and books in libraries to formalized systems in the 1960s such as integrated data systems and information management systems. These stored data but retrieved it poorly. E.F. Codd of IBM wrote the seminal relational database management paper in 1970, creating the foundation for retrieving data easily for business purposes.

• Structured data in databases moved onto servers, then to distributed servers as servers became overloaded, then to cloud and now hybrid cloud. Data lakes added less structured storage, and Lakehouse combined lake benefits with database and server benefits. Users, data engineers, data scientists, admins, and business applications all query, manipulate, and read that stored data.

• AI systems now extract data to train models, populate vector databases, and support RAG and gen AI systems, and AI data can flow back into business processes to manipulate enterprise data. Alongside older threats such as breaches, theft, and ransomware, AI introduces data poisoning, where attackers corrupt training data to manipulate how the AI behaves.