iscover
Share this article on:
Powered by
MOMENTUM
MEDIA
The German Federal Office for Information Security (the Bundesamt für Sicherheit in der Informationstechnik, or BSI) released a report this month on the dangers of how AI can be influenced by altering the data it relies upon to build its models.
The report outlines three possible attack methods — evasion attacks, information extraction attacks, and poisoning and backdoor attacks.
An evasion attack is when a threat actor introduces a “malicious input” during the inference phase of an AI’s machine learning model. This involves introducing an amount of perturbation to the input, such that the model can longer accurately predict what it is seeing, for instance. This perturbation could very well be visible to the human eye but cannot be perceived by the AI.
Evasion attacks can also take place when building AI “student” models off pre-existing “teacher’ models. Any attack on the teacher model could be passed on to the student, and in cases where the teacher model is publicly available and widely used, this could be particularly damaging.
Information attacks — also known as reconstruction or privacy attacks — involve reconstructing an AI model from its training data. A threat actor might try to steal a model by reconstructing it based on the answers given by the original, for instance, which are in turn fed back into the adversaries own model.
Other information attacks include membership inference attacks — wherein a threat actor may try to build a new model based on the differences between AI’s training data and newly inputted data — and attribute inference attacks, where a threat actor uses publically available data to get an AI model to infer private data, such as an address.
Poisoning and backdoor attacks work by targeting the data a model is built on itself, by flipping the label of an input to muddy the end result of a query or by directly building a set of triggers within a dataset to produce a specific result.
“An attack is successful when a backdoored model behaves normally when encountering benign images but predicts the adversary’s chosen label when presented with triggered images,” the report stated.
All these attacks do rely on some access to the dataset the model is built on, making them non-trivial to execute, but the possibility for damaging outcomes is enough that the BSI feels that these threats are absolutely essential for developers to understand and know how to defend against.
But outright attacks are not the only thing AI developers need to be aware of, according to the report.
“Apart from malicious attacks on machine learning models, a lack of comprehension of their decision-making process poses a threat,” the report said. “The models could be learning spurious correlations from faulty or insufficient training data.”
“Therefore, it is helpful to understand their decision process before deploying them to real-world use cases.”
David Hollingworth has been writing about technology for over 20 years, and has worked for a range of print and online titles in his career. He is enjoying getting to grips with cyber security, especially when it lets him talk about Lego.
Be the first to hear the latest developments in the cyber industry.
