What Breaks When You Build AI Systems Under Sovereignty Constraints

A talk at AI Engineer Europe 2026: https://www.ai.engineer/europe

Regulatory and jurisdictional constraints are no longer an edge case in AI system design; they now shape architectural decisions as much as model quality does. From European efforts like “Eurostack” to sovereign cloud offerings by hyperscalers, sovereignty is becoming a practical engineering constraint, pushing teams to design systems that operate within defined boundaries.

What changes when your AI system can’t send data outside a region, rely on external APIs, or depend on infrastructure you don’t control? More importantly, what breaks?

This talk explores sovereign AI as a system design problem, focusing on the hidden assumptions in modern AI architectures that fail under real-world constraints. Many production systems rely on external dependencies, from embedding APIs to evaluation tools, that make them difficult to audit, reproduce, or control.

We’ll examine what breaks in these architectures and how sovereignty requirements reshape core design decisions: where models run, how data flows, and how systems remain observable, auditable, and replaceable.

To make this concrete, we’ll walk through a reference architecture using an open, modular orchestration approach (with Haystack as an example), and show how to:design pipelines that run across cloud, on-prem, and hybrid environmentsswap models without redesigning the systemkeep sensitive data local while integrating external capabilities when allowedmaintain full visibility into data flow and system behaviorThe focus is on building systems that remain flexible under constraints with replaceable components, explicit data flows, and control staying within your boundary.

Transcript

1. What Breaks When You Build AI Systems Under Sovereignty Constraints Bilge Yücel, Sr. DevRel Engineer @ deepset
2. Who is deepset? Company Solving Custom AI challenges since 2018. HQ in Berlin and NYC. Backed by: Leading open source framework & commercial platforms for custom enterprise-grade AI Products Used by 70 Thought leaders
3. Definition Sovereign AI is the ability of an organization to design, deploy, and operate AI systems on its own terms.
4. Definition Sovereign AI is the ability of an organization to design, deploy, and operate AI systems on its own terms. Having explicit control over data flow, model choice, infrastructure, observability and operations.
5. Four Pillars of Sovereign AI Data Sovereignty where do you store it, where does it get processed? Model Sovereignty who controls the running models, origin of the training data? Infrastructure Sovereignty where does compute happen? Operational Sovereignty Is it traceable, who can update it, who owns the incident response? Sovereign AI
6. Data Sovereignty • Data stored and processed within trusted jurisdictions to meet compliance requirements • Access permissions are respected Governing how data is accessed and used in AI systems
7. Infrastructure Sovereignty Where does compute happen? Max control Max convenience Air-gapped On-prem, no egress EU AI Act safe Private cloud VPC, dedicated GDPR safe Sovereign cloud EU-operated infra Depends on provider Hybrid Local + gated APIs Requires data gating SaaS Full vendor CLOUD Act risk
8. Model Sovereignty Who controls the model and origin of the training data? • Freedom to choose and switch models • Swapability without architectural changes • Training data origin
9. Operational Sovereignty Monitoring, evaluating and managing AI systems over time • Monitor how AI systems behave in production, including model outputs. • In high-stake environments, human-in-the-loop is incorporated. • Managing versioning, and updates to models and application layer in a controlled, auditable way.
10. Sovereignty is a spectrum Not everyone needs full sovereignty in all pillars Air gapped finance, healthcare Public Sector Enterprise Startup Know your level of control
11. Engineering Challenges of Sovereignty What you do and what breaks in existing systems Maintenance - Cost - Performance
12. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model → Maintenance - Cost - Performance
13. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model Translate API logic to the new model architecture → Maintenance - Cost - Performance
14. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model You move private data into the required jurisdiction Translate API logic to the new model architecture → Maintenance - Cost - Performance
15. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model You move private data into the required jurisdiction Translate API logic to the new model architecture Managing multiple databases & instances → → Maintenance - Cost - Performance
16. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model You move private data into the required jurisdiction You replace managed infra with on-prem Translate API logic to the new model architecture Managing multiple databases & instances → → Maintenance - Cost - Performance
17. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model You move private data into the required jurisdiction You replace managed infra with on-prem Translate API logic to the new model architecture Managing multiple databases & instances Vendor lock-in, hardware limitations → → → Maintenance - Cost - Performance
18. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model You move private data into the required jurisdiction You replace managed infra with on-prem You incorporate observability & tracing Translate API logic to the new model architecture Managing multiple databases & instances Vendor lock-in, hardware limitations → → → Maintenance - Cost - Performance
19. Engineering Challenges of Sovereignty What you do and what breaks in existing systems You replace the frontier API with a self-hosted model You move private data into the required jurisdiction You replace managed infra with on-prem You incorporate observability & tracing Translate API logic to the new model architecture Managing multiple databases & instances Vendor lock-in, hardware limitations Black-box, version control → → → → Maintenance - Cost - Performance
20. Why Haystack A good orchestration solves some of these problems Serializable to YAML Truly open & extensible Explicit data flow Consistent interface
21. Sovereign Architecture
22. Sovereign Architecture
23. Guardrails Tutorial: AI Guardrails: Content Moderation and Safety with Open Language Models
24. Tools
25. Agent
26. Bring it all together
27. Sovereignty Checklist Can you swap models without changing the application logic? Do you have reproducible run logs, stored in a compliant way? Can your team respond to an incident without calling a vendor?
28. Thank you www.deepset.ai www.haystack.deepset.ai Get the presentation & Ask questions in/bilge-yucel @bilgeyucl