FORENSIA: Local LLM Forensic Harness

sumeshi/forensia
https://github.com/sumeshi/forensia

Do Local LLMs Dream of Forensic Investigators?
https://sumeshi.github.io/posts/works/do-localllms-dream-of-forensic-investigator-en

Transcript

1. YOUR LOCAL AI ASSISTANT FOR WEEKEND FORENSIC WORK. FORENSIA: Local LLM Forensic Harness S.Nakano / @sum3sh1 SA-FOR-001-EN
2. Today's Talk I built and released FORENSIA, a local-LLM tool that assists forensic investigations. I'll share the design principles behind it and the lessons learned during development. github.com/sumeshi/forensia 02
3. Background Generative AI would be invaluable in forensics. But sensitive evidence raises a basic question: how far can we allow the data to be replicated? Is opting out of training enough? Is Bedrock acceptable? What about rented GPUs? Can we still claim full control of the data? For now, local deployment is the clearest answer. 03
4. Goal Build a forensic tool powered by a local LLM that runs completely offline. Ingest Windows Artifacts Output Investigation Tool Investigation Report 04
5. We Can't Compete with Managed LLMs on Their Turf 2B~8B That's the practical limit. At that scale, an LLM can handle only very simple tasks, such as summarizing text. 1T (1,000B) Managed LLMs (5T~) 400B This is about the limit for a typical PC 120B 27B 8B Relative Model Sizes 05
6. Use the LLM as a Cog, Not the Investigator The LLM only generates and validates hypotheses. HARNESS Rule-based Detection LO OP Search Evidence Update Report LLM Detection Results Hypothesis Relevant Evidence Validation Results Generate Hypothesis Validate 06
7. Design Principles 1. Work alone Runs offline during an investigation 2. Don't expect too much from the LLM “Just figure it out” doesn't work Breaking down and supplying the right information is our job 3. Be generous with time A perfect conclusion in one pass is difficult—even for humans Iterate through hypotheses and validation to refine the result 07
8. Architecture Based on these principles, I implemented several mechanisms to support weaker LLMs during investigations. A1 A3 Iterative Hypothesis Validation A2 Structured Memory Strict Reasoning Control A4 Communicate Investigative Intent 08
9. A1. Iterative Hypothesis Validation Checklist-driven investigations can miss case-specific events. Generate hypotheses from questions raised during the investigation, then validate them iteratively. HYPOTHESIS INGEST > My job is to write loops. - Boris Cherny from Anthropic SEARCH UPDATE CHECK 09
10. A2. Structured Memory CASE STATE Established facts TRACE STATE Decision rationale WORKING STATE Current task Too much context makes output inconsistent. Separate memory by role, then reconstruct only what the LLM needs. A decision trace makes each judgment and its rationale traceable. KNOWLEDGE External sources OWASP - Agent Observability Standard https://owasp.org/www-project-agent-observability-standard-2/ 10
11. A3. Strict Reasoning Constraints To keep outputs from small models stable and accurate: use strict prompts, constrain output content and format, and provide only enough context to prevent mix-ups and reasoning drift. This classic approach still works for weaker models. 11
12. A4. Communicate Investigative Intent Detection rules can carry more than match results. Pack in intent, what to investigate next, and search queries; pass the enriched rule to the LLM at inference time. 12 Example Detection Rule
13. Quick Benchmark I tested it on CFReDS questions supported by the current implementation. 8 / 12 About 70% correct with gemma-4-e2b. Hmm, not bad. Failed: 33.3% The RTX 2070 SUPER that did the heavy lifting ($150) NIST - CFReDS Data Leakage Case https://cfreds-archive.nist.gov/data_leakage_case/data-leakage-case.html Correct: 66.7% 13 Correct Failed
14. Where Did It Struggle? 01 Getting Anchored by Words in the Context Seeing “Web Browser,” the model searched only web artifacts and never reached the MFT that actually mattered. → Add pattern filters or use external knowledge to guide the model? 02 Distinguishing Absent, Not Found, and Not Investigated Even when memory correctly separates absent, not found, and not investigated, the model can still conflate them. → Track investigative coverage as explicit internal state? 14
15. Summary FORENSIA decomposes investigations into fine-grained tasks and manages the memory and reasoning around each task. The hard part is not the model's raw capability, but giving it the right task at every step and keeping the investigation consistent. Ultimately, what matters is having a good orchestrator. 15
16. FORENSIA Your local AI assistant for weekend forensic work. Let's be honest: local LLMs are hard mode!!! github.com/sumeshi/forensia
17. Appendix: Dashboard 17
18. Appendix: Report 18