From Hours to Minutes: An AI Case Study with Sheriff

Every developer has seen it happen—a tool with the potential to make a huge difference ends up underused because getting started feels overwhelming. That’s the situation we saw with Sheriff, a tool for enforcing architectural boundaries in TypeScript applications. Its benefits were clear, but in mature codebases, the setup process involved understanding a huge, complex dependency graph and crafting a configuration that could take days to get right.

We’re now exploring how AI, together with the Model Context Protocol (MCP), can change that. The idea is straightforward: use MCP to let the AI access project details, propose an initial configuration, and then iteratively refine it based on Sheriff’s own analysis. Early experiments suggest that this approach can replace a long, manual setup with a faster, guided process.

In this session, I’ll share the problem we set out to solve, how we’re approaching the integration, and the promising results we’ve seen so far. We’ll also discuss how this pattern might be applied to other tools to address real adoption challenges.

Transcript

1. From Hours to Minutes An AI Case Study with Sheriff AI-Poland, 20. November 2025 Murat Sari Rainer Hahnekamp
2. Murat Sari & Rainer Hahnekamp Sheriff: Modularity in TypeScript • Module Encapsulation • Dependency Rules
3. Murat Sari & Rainer Hahnekamp Sheriff: Modularity in TypeScript • Module Encapsulation • Dependency Rules • Lightweight • Convention over Configuration • Zero Dependencies • For all TypeScript Projects
4. Murat Sari & Rainer Hahnekamp From simple configurations…
5. Murat Sari & Rainer Hahnekamp …to something else
6. Murat Sari & Rainer Hahnekamp Motivation • Creating a config could be overwhelming ◦ How to split - Architecture Styles ◦ Transform the desired rules into a config ◦ How to verify the config • Maybe AI can automate it? ◦ Ask to analyze the project ◦ Suggest a style or domain cut ◦ Verify the architecture ◦ Visualize the architecture
7. Murat Sari & Rainer Hahnekamp Features without AI Data Condensation UI
8. Murat Sari & Rainer Hahnekamp Challenges Quality Latency Data Privacy Context Size Token Consumption
9. Murat Sari & Rainer Hahnekamp Approach 1 - ICL Prompting (No MCP / No Tools) • How can we use AI to produce a valid Sheriff configuration as a reliable starting point? • Structured prompting with In-Context Learning (ICL) ◦ The idea is to embedding structured context (documentation, examples and constraints) within the system prompt. ◦ Here is our example https://hackmd.io/@wolfmanfx/SyKDmNveWx ◦ Demo
10. Murat Sari & Rainer Hahnekamp Approach 2 - State Machine (No MCP but tools) • Init - Welcomes the user and guides him • Structure state - Initial list of domains/types • Dependency rules - Rules + updated domain/type list • Done - Generate the sheriff config
11. Murat Sari & Rainer Hahnekamp Approach 2 • State machine controlled by an “ROUTER SYSTEM PROMPT” ◦ acts as a controller that routes requests to sub handler • Each state consist of a short lived context and specific goal ◦ We do not blow up our context as we always include only the data we need in each specific sub state (each state has an isolated context) • DEMO
12. Murat Sari & Rainer Hahnekamp
13. Murat Sari & Rainer Hahnekamp Approach 3 - Own LLM • Current LLMs (Frontier / Local) do not know sheriff ◦ Leads to full hallucination (if no ICL prompting is applied or MCP is used) • Idea train the LLM to be a “Sheriff configuration expert” ◦ Should prevent hallucinate incorrect answers / configs • Traditional Fine-Tuning Problem ◦ Full model: ~1.1 billion parameters (TinyLlama-1.1B-Chat-v1.0) ◦ Training: Update all 1.1B parameters • LoRA - Low-Rank Adaption ◦ Freeze base model: 1.1B parameters (locked) ◦ Train small adapters: 110M parameters (10%)
14. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA • Step 1 - Create Examples (Manual / Automated) • Step 2 - Prepare Training Data • Step 3 - Train Model • Step 4 - Test Model • Step 5 - Convert for Deployment (GGUF / MLX)
15. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA / Step 1 • We have created the examples in markdown ◦ Following this structure: ▪ Question - “Generate a basic Sheriff config for…) ▪ Additional input (Project structure) ▪ Than a requirements section - “Each domain can only access…” ▪ Most important the expected result (Sheriff config) ◦ Why markdown ▪ Easy to use ▪ Human-readable and editable ◦ Created ~15 manual examples and extrapolated to ~250 examples using “Ai”
16. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA / Step 2 • Data Preparation ◦ Input our markdown folder ◦ Output JSONL in ChatML format • JSONL (JSON lines) ◦ It means we store one valid JSON per line in the file ◦ How we store our examples on disk • ChatML (Always check the model how it expects the training data)
17. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA • We use Transformers lib from hugging face (AutoModelForCausalLM)
18. Murat Sari & Rainer Hahnekamp
19. Murat Sari & Rainer Hahnekamp Demo Model • Video
20. Murat Sari & Rainer Hahnekamp Summary • AI makes a significant contribution • AI as a helper ◦ All tasks can also be done without AI ◦ No dependency to AI ◦ Tooling to verify the outcome (non-deterministic behavior) • Where AI can't help us ◦ Specific UI ◦ Raw Import Graph • Mixed approaches ◦ No MCP ◦ MCP with controlled tooling access ◦ Full MCP ◦ State Machine
21. Murat Sari & Rainer Hahnekamp Dziękuję!