[JSAC 2025 LT] Introduction to MITRE ATT&CK utilization tools by multiple LLM agents and RAG

MITRE ATT&CKをLLMやRAGを用いて利活用する方法とその具体的なPoCとしてのdisarmBotの紹介スライドです。JSAC 2025 https://jsac.jpcert.or.jp/ にて発表を行いました。国際会議のためスライドは英語で記述しております。また、ソースコードはOSSとして公開しています。

github：https://github.com/ultra-supara/disarmBot
デモ動画（YouTube）：https://www.youtube.com/watch?v=Ee-JfL17L40
都合のため動画をカットして編集しております。フル尺だと8分以上あります。

disarmBotは、Discord上に導入できるOSSのBotです。ユーザがコマンドを入力することで複数のLLMエージェント(GPT-4)が自動的に立ち上がり、応答します。また、DISARM（Disinformation Analysis and Response Measures）TTP Frameworksに基づいており、DISARMは「理論」にあたるMITRE ATT&CKに基づいています。つまり、理論から公助に向けたLLMによる実践的CTI利活用のための施策です。複数の異なる戦術を学習したLLMエージェントが協力し、attacker、defender、user、skeptics、OSINT specialist、Solution architectが互いに強調しながらフレームワークに基づいた戦術的・技術的な対話を行います。対話を通じてエージェント同士の議論を通じた情報の深堀りを行います。disarmBotは、これらの条件を満たし、ユーザーが多様な意見に触れることができる情報環境を提供します。これにより、ユーザーは自ら考え、情報を消化するクリティカルな能力を高めることができます。

Transcript

1. Introduction to MITRE ATT&CK utilization tools by multiple LLM agents and RAG @4su_para 2025/1/22 1 JSAC 2025 LT
2. $ aws sts get-caller-indentity 2 • atsushi, sada a.k.a @4su_para • Bachelor of Computer Science • Security Engineer（24卒） • PSIRT, Corp IT • OSS dev, LLM lover • Seccamp/SecHack365 OB • Speaker @ BlackHat Arsenal JSAC 2025 LT 2025/1/22
3. Work as PSIRT, Corp IT 3 ◦Incident Response at Cloud Environment⭐⭐ pCSPM, log analysis ◦{Vulnerability, Platform} Analysis⭐ pin house project ◦Device Management (MDM)⭐⭐ pJamf , Intune 2025/1/22 JSAC 2025 LT
4. Hobby 4 ◦Research at Cloud Security⭐⭐ pGitHub Ecosystem ppurple teaming @ AWS ◦OSS Security Tools Development⭐⭐ pSAST, static analysis pLLM Hacker 2025/1/22 JSAC 2025 LT Today’s Topic
5. Overview 5 2025/1/22 JSAC 2025 LT ◦LLMs and AI tech are evolving explosively pSo, it is a priority to consider how LLM can be used in security and incident response as well! ◦AI Agents pAI Agent is a framework based on LLMs that autonomously achieves goals set by humans pintroduce sample PoC : disarmBot ⁍ AutoGen, AutoGPT, crew AI ・・・
6. AutoGen by Microsoft, execution pattern 6 2025/1/22 JSAC 2025 LT Group Chat Two Agent Chat Sequential Chat
7. AutoGen usecases 7 2025/1/22 JSAC 2025 LT AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023) , https://arxiv.org/abs/2308.08155 ◦A1.Math Problem ◦A2.RAG Chat ◦A3.ALF Chat ◦A4.Multi-agent Coding ◦A5.Group Chat ◦A6. Chess The key to innovation is mixed!!
8. RAG（retrieval-augmented generation） 8 2025/1/22 JSAC 2025 LT Create Embeddings Documents Vector Store split into chunks Answer LLM LLM user question Standalone question
9. Main Theme in the Threat Intelligence 9 2025/1/22 JSAC 2025 LT ◦Effective use of data and theory pFramework such as MITRE ATT&CK pThe strength of user-side companies are their ability to utilize vast amounts of internal information Information needs to be transformed into something useful in some way.
10. Sample PoC : disarmBot 10 2025/1/22 JSAC 2025 LT ◦Discord Bot as a proactive approach to enhance critical thinking about disinformation pdisinformation, particularly in digital environments, causes social disruption and underscores the importance of reliable information sources. pOSS, anyone can use in discord platform. pGive answers that are linked to strategy ◦Use vector DB, RAG, AutoGen, API, prompt engineering pRead disarm TTP framework to Chroma DB pAutoGen Group Chat mode & RAG puse Open AI API or Azure Open AI API
11. disarm TTP framework・・・ 11 2025/1/22 JSAC 2025 LT ◦MITRE ATT&CK frameworks designed for describing and understanding disinformation incidents pRed Team framework pBlue Team framework ◦TTP（Tactics, Techniques, Procedures） pPhases: higher-level groupings of tactics, created so we could check we didn't miss anything pTactics: stages that someone running a misinformation incident is likely to use pTechniques: activities that might be seen at each stage pTasks: things that need to be done at each stage. Tasks are things you do . Techniques are how you do them. pCounters: countermeasures to DISARM TTPs. pActor Types: resources needed to run countermeasures pResponse types: the course-of-action categories we used to create counters pMetatechniques: a higher-level grouping for countermeasures pIncidents: incident descriptions used to create the DISARM frameworks format data
12. Insert data to local Embedding DB 12 2025/1/22 JSAC 2025 LT AI native embedding database format data Vector Store
13. Group Chat Characters, 5 AI Agents 13 2025/1/22 JSAC 2025 LT attacker side defender side Solution architect OSINT specialist Skeptics
14. Character1,2 : A/D Agents 14 2025/1/22 JSAC 2025 LT attacker side ◦Read Offensive TTP⭐⭐ pPrompt : You are an expert in disinformation attacks. Your role is to use your expertise in disinformation attacks to find vulnerabilities in the case. Use the searchDisarmFramework function to search for strategies/tactics related to the red framework and discuss them. ◦Read Defensive TTP⭐⭐ pPrompt : You are a disinformation countermeasure/defense expert. It is your role to use your expertise on the disinformation defense side to think about responses to the vulnerabilities in the case. Use the searchDisarmFramework function to search for strategies/tactics related to blue framework and discuss them. defender side
15. Character3：OSINT Specialist Agent 15 2025/1/22 JSAC 2025 LT OSINT specialist ◦Searches the internet and generates answers pPrompt : You are an Internet search expert. Your role is to introduce outside information and stimulate discussion. You must use the searchTheInternet function to search the Internet and summarize the information.
16. Characters4, 5 AI Agents 16 2025/1/22 JSAC 2025 LT Skeptics Solution architect ◦Oppose other agents as a skeptic pPrompt : You are a skeptic. Your role is to act as devil's advocate and provide a critical perspective on what other agents say. Use the searchDisarmFramework function to search for what other agents say and ask your skeptical questions. ◦Solution Architect to bring it all together pPrompt : You are a solution architect. Your role is to provide a solution to the problem using expert's information. Use the searchDisarmFramework functions to provide a solution..
17. Demonstration Snapshot 17 2025/1/22 JSAC 2025 LT Demo Movie
18. Summary・・・ 18 2025/1/22 JSAC 2025 LT ◦Users are responsible for judging the responses generated by the bot. pproactive approach ◦Use Replacing database & prompt, it can also be utilized for Penetration Test, DFIR ・・・ p planning to create a new tool that automates incident response using LLMs and bring it to JSAC 2026! Stay tuned!
19. Thanks! https://github.com/ultra-supara/disarmBot 2025/1/22 19 JSAC 2025 LT