A Crash Course into Attacking AI [1/4]: What is AI security?

August 03, 2024•6 min read

A Crash Course into Attacking AI [1/4]: What is AI security?

What do you think of when I talk about AI security? Perhaps you think of stopping killer robots, using machine learning to detect cyber security attacks, making sure AI isn’t discriminatory, or preventing the development of harmful AI systems. These are all important aspects of ensuring AI systems are being developed and used responsibly.

I use the term AI to describe a system that can generate outputs to achieve human-defined objectives without explicit programming or human direction. This includes chat bots, language translation, document readers, social media recommendation systems, data processing for predictions or recommendations, and more. AI is everywhere, which makes responsible, safe, and secure AI practices so important. If you’re interested in learning about these related, responsible AI fields, I’ve left more details at the end of this blog!

What is AI Security?

AI security prevents AI systems from being hacked. This is different from using AI to help with our cyber security practices, which aims to use AI to protect systems from threats. You might see this field also referred to as AI Cybersecurity, Cybersecurity for AI, AI Security Operations (AISecOps), or Machine Learning Security Operations (MLSecOps).

What do you think of when I talk about AI security? Perhaps you think of stopping killer robots, using machine learning to detect cyber security attacks, making sure AI isn’t discriminatory, or preventing the development of harmful AI systems. These are all important aspects of ensuring AI systems are being developed and used responsibly. I use the term AI to describe a system that can generate outputs to achieve human-defined objectives without explicit programming or human direction. This includes chat bots, language translation, document readers, social media recommendation systems, data processing for predictions or recommendations, and more. AI is everywhere, which makes responsible, safe, and secure AI practices so important. If you’re interested in learning about these related, responsible AI fields, I’ve left more details at the end of this blog! What is AI Security? AI security prevents AI systems from being hacked. This is different from using AI to help with our cyber security practices, which aims to use AI to protect systems from threats. You might see this field also referred to as AI Cybersecurity, Cybersecurity for AI, AI Security Operations (AISecOps), or Machine Learning Security Operations (MLSecOps). Meme format of a soldier protecting a child with an AI security label over the soldier and AI systems over the child. Image: Google Images The incentive for hackers to attack AI systems grows as AI gets adopted in various industries and continues to get integrated into different critical systems. Yet awareness of AI security is lagging. 77% of companies reported that they identified breaches to their AI in the past year AI systems have additional unique weaknesses that need to be considered in addition to existing cybersecurity vulnerabilities. Adversarial Machine Learning is the field that develops attacks that exploit unique AI weaknesses. As Adversarial Machine Learning is a maturing field, there is a lack of appropriate models to concisely communicate the consequences of Adversarial Machine Learning attacks. Here at Mileva, we have developed the 3D models to help AI businesses manage and understand their AI security risk by categorizing attacks by the three main impacts they have. (Notes: The 3D framework is currently under consideration for the ICITA conference). The 3Ds of AI Security Deceive: Techniques that manipulate AI systems to produce incorrect outputs. Disrupt: Techniques that interfere with the normal functioning of AI systems. Disclose: Techniques that extract sensitive information from AI systems. An image of a chainsaw is labeled ‘chainsaw’, and an image of a chainsaw with $$$ pasted over it is labeled ‘piggybank’. Photograph: OpenAI, The Guardian An image of a self-driving car stopped in the middle of the road with a cone on its bonnet. The title reads Protesters Take on Self-Driving Cars. Photograph: ABC7 An image of a ChatGPT message leaking information. Redacted items are personally identifiable information. Image: Google DeepMind I’ve provided (my personal favorite) examples of AI security attacks at work. They can seem a bit silly, but demonstrate how AI systems can fail in ways that we don’t expect! How is AI Security different from cybersecurity? “We’ve been able to Deceive, Disrupt, and Disclose systems with other cybersecurity methods for a while now, what makes AI security different?” How some people think about AI Security… … vs. How AI Security currently fits into Cybersecurity Image of a character being blocked by a locked door despite giant holes in the wall. Image: Google images Image of a door with a cheeto holding the lock together. Image: Google images AI systems have a stochastic element (the statistical word for randomness) that traditional, deterministic software systems don’t have. As a result, current cybersecurity measures have a blind spot to AI weaknesses that haven't necessarily been present in non-AI systems. Key features of Adversarial Machine Learning attacks: A broader attack surface: There are different ways to influence machine learning models to get the outcomes you desire. Some require you to only have access to the AI system’s user interface while others alter public data that you know is being scraped to train AI. Can be more subtle: Some attacks are minimally intrusive, making it difficult to determine whether someone is just using the system or is trying to subvert it without intentionally investigating. Can have widespread impact: Many AI solutions rely on common models, data sources, and/or libraries. If you can attack the state-of-the-art models, you can likely attack many systems. What are the challenges in the AI security field? AI Security is gaining traction and the community is growing. Just in the last year, we have seen: Big tech firms continued to develop AI security functions and tooling, including (but not limited to) Google releasing their first AI red-teaming report and Microsoft announcing PyRit, a GenAI red teaming tool. Hunt’r, the world’s first bug bounty platform for AI/ML, was announced. Various public institutions continue to develop and release advice for the security of our AI systems. A screenshot of the frequency the term ‘AI Security’ gets searched over time as a percent over the maximum search frequency. The graph shows interest in AI Security has grown rapidly over the last 4 years. Image: Google Trends However, AI security has a ways-to-go. The percent of organizations that have practices in place to secure their AI is too low and doesn’t match the rate at which the technology is being adopted. AI Security faces several challenges being a (relatively) emerging field: Awareness of AI security: The general level of AI security literacy in practitioners needs to grow. AI technologies are diverse: Spanning various techniques and application contexts, AI security developers have to balance designing generalizable AI security tools with the specificity to be useful. The field is still in the process of maturing and standardizing security testing and vulnerability disclosure practices. It’s an interdisciplinary field: To develop new tools or techniques requires a good understanding of very broad disciplines: data science, computer science, AI systems, and cybersecurity operations. So what’s next? Our mission in Mileva is to tackle these challenges as we believe in the importance of AI security. This article is the first of three in a series designed to introduce AI security to people who haven’t heard about the field before! We need more cybersecurity practitioners, data scientists, and tech strategy and policy professionals involved in the practice; the more secure our AI systems will be as a whole! By the end of the series, I hope to leave readers with insight on why AI attacks work and how an AI hacker might go about designing an AI attack. The next blog will go through case studies to answer: What can AI attacks do? Stay tuned!

Meme format of a soldier protecting a child with an AI security label over the soldier and AI systems over the child. Image: Google Images

The incentive for hackers to attack AI systems grows as AI gets adopted in various industries and continues to get integrated into different critical systems. Yet awareness of AI security is lagging.

77% of companies reported that they identified breaches to their AI in the past year

AI systems have additional unique weaknesses that need to be considered in addition to existing cybersecurity vulnerabilities. Adversarial Machine Learning is the field that develops attacks that exploit unique AI weaknesses. As Adversarial Machine Learning is a maturing field, there is a lack of appropriate models to concisely communicate the consequences of Adversarial Machine Learning attacks. Here at Mileva, we have developed the 3D models to help AI businesses manage and understand their AI security risk by categorizing attacks by the three main impacts they have. (Notes: The 3D framework is currently under consideration for the ICITA conference).

The 3Ds of AI Security

Deceive: Techniques that manipulate AI systems to produce incorrect outputs.
An image of a chainsaw is labeled ‘chainsaw’, and an image of a chainsaw with $$$ pasted over it is labeled ‘piggybank’. Photograph: OpenAI, The Guardian
Disrupt: Techniques that interfere with the normal functioning of AI systems.
An image of a self-driving car stopped in the middle of the road with a cone on its bonnet. The title reads Protesters Take on Self-Driving Cars. Photograph: ABC7
Disclose: Techniques that extract sensitive information from AI systems.
An image of a ChatGPT message leaking information. Redacted items are personally identifiable information. Image: Google DeepMind

I’ve provided (my personal favorite) examples of AI security attacks at work. They can seem a bit silly, but demonstrate how AI systems can fail in ways that we don’t expect!

How is AI Security different from cybersecurity?

“We’ve been able to Deceive, Disrupt, and Disclose systems with other cybersecurity methods for a while now, what makes AI security different?”

How some people think about AI Security…

A person standing in front of a broken roomDescription automatically generated

Image of a character being blocked by a locked door despite giant holes in the wall. Image: Google images

vs.

…How AI Security currently fits into Cybersecurity

A close up of a doorDescription automatically generated

Image of a door with a cheeto holding the lock together. Image: Google images

AI systems have a stochastic element (the statistical word for randomness) that traditional, deterministic software systems don’t have. As a result, current cybersecurity measures have a blind spot to AI weaknesses that haven't necessarily been present in non-AI systems.

Key features of Adversarial Machine Learning attacks:

A broader attack surface: There are different ways to influence machine learning models to get the outcomes you desire. Some require you to only have access to the AI system’s user interface while others alter public data that you know is being scraped to train AI.
Can be more subtle: Some attacks are minimally intrusive, making it difficult to determine whether someone is just using the system or is trying to subvert it without intentionally investigating.
Can have widespread impact: Many AI solutions rely on common models, data sources, and/or libraries. If you can attack the state-of-the-art models, you can likely attack many systems.

What are the challenges in the AI security field?

AI Security is gaining traction and the community is growing. Just in the last year, we have seen:

Big tech firms continued to develop AI security functions and tooling, including (but not limited to) Google releasing their first AI red-teaming report and Microsoft announcing PyRit, a GenAI red teaming tool.
Hunt’r, the world's first bug bounty platform for AI/ML, was announced.
Various public institutions continue to develop and release advice for the security of our AI systems.

A graph showing a lineDescription automatically generated with medium confidence

A screenshot of the frequency the term ‘AI Security’ gets searched over time as a percent over the maximum search frequency. The graph shows interest in AI Security has grown rapidly over the last 4 years. Image: Google Trends

However, AI security has a ways-to-go. The percent of organizations that have practices in place to secure their AI is too low and doesn’t match the rate at which the technology is being adopted. AI Security faces several challenges being a (relatively) emerging field:

Awareness of AI security: The general level of AI security literacy in practitioners needs to grow.
AI technologies are diverse: Spanning various techniques and application contexts, AI security developers have to balance designing generalizable AI security tools with the specificity to be useful. The field is still in the process of maturing and standardizing security testing and vulnerability disclosure practices.
It’s an interdisciplinary field: To develop new tools or techniques requires a good understanding of very broad disciplines: data science, computer science, AI systems, and cybersecurity operations.

So what’s next?

Our mission in Mileva is to tackle these challenges as we believe in the importance of AI security. This article is the first of three in a series designed to introduce AI security to people who haven’t heard about the field before!

We need more cybersecurity practitioners, data scientists, and tech strategy and policy professionals involved in the practice; the more secure our AI systems will be as a whole!

By the end of the series, I hope to leave readers with insight on why AI attacks work and how an AI hacker might go about designing an AI attack. The next blog will go through case studies to answer: What can AI attacks do?

Stay tuned!

Further Reading

In this blog, I briefly mentioned the other rated AI disciplines. I’ve linked some useful jumping off points for readers who are interested in learning more about them!

Responsible AI
A set of principles to guide the responsible design, development and use of AI systems
https://store.training.tafensw.edu.au/product/responsible-artificial-intelligence/

AI Safety
A field concerned with the impact of and potential harm of AI systems to society
https://www.safe.ai/blog/ai-safety-ethics-and-society

AI Alignment
A subfield of AI Safety concerned with making sure an AI system's objectives match its intended use.
https://brianchristian.org/the-alignment-problem/

AI for Cyber Security
A research and development area working on creating AI powered tools and solutions to help cybersecurity professionals.
https://www.sans.org/cyber-security-courses/applied-data-science-machine-learning/

Back to Blog