π AI Red Teaming Case Study: When RAG gets RAGged Up π
Hey cloudy defenders, what happens when your precious RAG architecture gets RAGged up ? π₯
Well, RAG becomes the very sword adversaries wield against you to cause havoc ! This could be stealthy -> completely under the radar, remaining undetected for a long time.
How can this happen ? π€
π Let's quickly review a case study published by MITRE ATLAS, courtesy of the folks at Zenity π
π This case study, aptly titled "Financial Transaction Hijacking with M365 Copilot as an Insider", demonstrates how attackers could carry out multi-step attacks against RAG-powered AI infrastructure. The immediate impact is financial loss: funds get transferred to the attacker's bank account π
π€Ί While the target was Microsoft M365 Co-Pilot, AI apps leveraging RAG, e.g. RAG RAG-powered AI Agents, could be vulnerable to these attacks. The multi-step attack consists of several MITRE ATLAS techniques, including: LLM Plugin Compromise, Gather RAG-Indexed Targets, and several discovery techniques.
π Good news -> there are countermeasures for these attack techniques, I included some of them in the document (slides 11 & 12).
π However, given the nature of unique app requirements per organization, threat modeling and red teaming exercises are super imperative for detecting, understanding, and mitigating threats.
π More details about the case study are here -> https://atlas.mitre.org/studies/AML.CS0026
β‘ So key take-aways -> Don't assume, conduct threat modeling and red teaming against you AI applications and stay ahead of attackers.
π See how Mitigant helps -> https://www.mitigant.io/en/platform/security-for-genai