How Application Security Will Change with the Rise of AI

By Riotaro OKADA
Oct 30th, 2024
(Original Japanese version: https://speakerdeck.com/okdt/software-security-with-ai-owasp-llm-top-10 )

As DevOps, CI/CD, and rapid development cycles enhance software quality, security continues to lag, leaving critical systems vulnerable. Leveraging my expertise in the OWASP community, including contributions to projects like the OWASP LLM Top 10 Risks, I will explore how AI transforms application security practices.

This session will explore how AI can augment security efforts, compensate for human limitations, enhance threat detection, and streamline secure design. While AI offers powerful tools for developers, we’ll address its strengths and limitations, discussing why humans still need to guide and validate AI’s work. Attendees will walk away with a clear understanding of how to harness AI to fortify their development pipelines while proactively managing security risks.

Transcript

1. How Application Security Will Change with the Rise of AI Riotaro Okada , the ED of Asterisk Research, Inc OWASP Japan lead An OWASP contributor OWASP distinguished honorable lifetime member awarded(2024)
2. Introduction: AI in Industries • AI’s impact on the real world: sectors like supply chain optimization, medical diagnosis, fraud reduction, and herbicide minimization. Prompt: “If AI is transforming these industries, why not apply it to software development security?” https://www.delveinsight.com/blog/top-applications-of-artificial-intelligence-in-healthcare
3. Challenges in OSS Security • Lack of Security Focus: Many projects prioritize functionality over security. (more than 70%: High-Risk) • Unmonitored Dependencies: Reliance on external libraries can introduce hidden vulnerabilities. • Limited Resources: OSS maintainers often need more resources or expertise to implement strong security measures. OpenSSF 2024: Survey data reveals that nearly one-third of professionals involved in software development —such as system operators, developers, and maintainers— report feeling unfamiliar with secure software development practices, Prompt: ” I have a CVE on my OSS project. Please advise me on how to respond as the developer of this project.”
4. SSDF (NIST SP800-218) …too much references… • BSAFSS: SM.3, DE.1, IA.1, IA.2 • BSIMM: CP1.1, CP1.3, SR1.1, SR2.2, SE1.2, SE2.6 • EO14028: 4e(ix) • IEC62443: SM-7, SM-9 • NISTCSF: ID.GV-3 • OWASP ASVS: 1.1.1 • OWASP MASVS: 1.10 • OWASP SAMM: PC1-A, PC1-B, PC2-A • PCISSLC: 2.1, 2.2 • SCFPSSD: Planning the Implementation and Deployment of Secure Development Practices • SP80053: SA-1, SA-8, SA-15, SR-3 • SP800160: 3.1.2, 3.2.1, 3.2.2, 3.3.1, 3.4.2, 3.4.3 • SP800161: SA-1, SA-8, SA-15, SR-3 • SP800181: T0414; K0003, K0039, K0044, K0157, K0168, K0177, K0211, K0260, K0261, K0262, K0524; S0010, S0357, S0368; A0033, A0123, A0151
5. To get hints shortly: Try OpenCRE.org “The Open Source project “OpenCRE “ links all security standards and guidelines together at the level of requirements into one harmonized resource: threats, weaknesses, what to verify, how to program, how to test, which tool settings, in-depth discussion, training material. Everything organized.”
6. Software Security Practices – OWASP SAMM 2.0 Prompt: ”What is the developers’ dilemma about security practices?”
7. Top 2+1 Fallout Factors in Building Secure Software 1. Lack of High-Resolution Understanding of Threats • Understanding the potential threats to your software is essential to achieving adequate security. 2. Failed to choose the Appropriate Architecture and Implementations for the Business Objectives and equipping the team • Put in place the appropriate defenses against the anticipated threats. Common security functional requirements such as architecture, detailed design, error handling, and authentication are also derived from the threat assumptions. 3. Plus: Lack of governance / to allocate support on the right time and place.
8. AI is here to empower us, as a good design advisor for software security to understand threats and respond • Compensating for Developer Imagination Gaps • AI helps developers identify external threats they might not foresee, expanding their understanding of potential risks. • Broadening Threat Perspectives (unknown things) • AI brings an objective, outside-in view of threats, uncovering vulnerabilities developers may overlook due to limited scope or biases. • Supporting Proactive Security Design (something known) • AI enables the creation of security-first architectures by continuously considering external threats, leading to more resilient systems.
9. Prompt Story – a developer crash idea • I am developing (XXX e.g file uploader) for YYY industry. • List for potential cyber threats, especially for the developing system. • List for potential but industry-specific compliance for this project. • List for general effective countermeasures for designing and implementing (file uploader) • Write an example code of input validations for the file uploader in Python. • Can you find any vulnerable points in your code? • Hmm, I gave up on adding a file uploader to my system. It is too risky. Please explore alternative solutions
10. As for Code generation and testing …AI is here not to replace us, copilots will not think for you with contexts. • AI is a powerful tool for coding assistance but doesn’t know: • Business logic • Compliance rules • Legal risks • Architecture • Rate of accurately finding vulnerabilities from code …. • up to 30-40% by default • (better than general engineers?) • Beneficial to study secure coding fundamentals
11. Legal risk • Legal and IP concerns remains on Copilot and LLMs • What if AI-generated code looks like proprietary code? • What if I get sued because of my code (if it is mostly AI generated) • It's a good idea to use copilots to get hints and learn, and the engineers themselves will know, but it's out of the picture as a coding outsourcing source.
12. Aspect Requirements Definition Design (Architecture Design) Coding Testing Deployment Operations (Monitoring) Maintenance AI's Role • Supporting requirements analysis; • Assisting with threat modeling • Proposing security best practices; • Identifying architecture vulnerabilities • Code generation assistant; • Providing security advice (detecting vulnerabilities) • Generating test cases; • Assisting with security testing • Optimizing automation; • Proposing security deployment guidelines • Threat detection and log analysis; • Supporting incident response • Supporting vulnerability management; • Optimizing patch application Examples • Proposing requirements aligned with project goals; • AI-driven threat modeling tools (predicting potential attack vectors) • Recommendatio ns based on security design; • Discovering potential vulnerabilities during design phase • Copilot suggesting code; • Early detection of vulnerabilities such as input sanitization • Generating unit and automated test cases; • Automatic scans for security holes (e.g., preventing SQL injections or XSS attacks) • Optimizing CI/CD pipelines (automating deployment processes); • Checking configurations for safe deployment • Real-time threat detection via security monitoring tools; • AI-analyzed log data and alerts (e.g., detecting unauthorized access) • Automatically gathering new vulnerability information and proposing patches; • Supporting automated update processes Considerations • AI doesn't fully understand the business or regulatory environment, so developers must make final decisions • AI learns from existing code and designs, so there's a risk of unintentionally generating copyrighted code • The source of AI-generated code is often unclear, increasing the risk of using copyright- protected code unintentionally • Even test code may raise copyright concerns, so AI- generated content must be reviewed thoroughly • While deployment itself carries minimal copyright risks, AI-generated configurations should be audited • AI-generated reports and logs may be subject to legal audits, so ensuring their reliability and transparency is essential • AI-generated patches might involve proprietary technologies, increasing the risk of introducing legal issues Prompt: ” Can you list how AI can work effectively as an advisor in each stage of software development as a reality? Mark those that are high risk for IP or legal risk. Only point out those that are likely to cause problems”
13. Fundamental problem: AI is a Target Too (OWASP LLM Risk Top 10 & LLM Governance Guide) • AI Faces Unique Threats • Data Poisoning: Attackers corrupt training data to manipulate AI behavior. • Adversarial Attacks: Small manipulations trick AI into making incorrect decisions. • Prompt Injection: Complex techniques exploit weaknesses in how AI processes prompts. • How to Protect AI (Just a Part of the Solution) • Rate Limiting & Authentication: Prioritize robust rate limiting and user authentication to control access and usage. • Layered AI Defense: Use multiple AI models to detect and respond to abuse scenarios, leveraging advanced techniques and AI integration. #DefendAI(US Army), AI Cyber Defense Initiative(Google)
14. Key Takeaways – Use AI to Know Your Enemies Proactively and Harden Your Valuable Software • Security becomes proactive and faster. With AI, we don’t just respond to threats—we get ahead of them. AI helps us anticipate, detect, and act on risks before they become more significant problems. • AI helps us think like the attackers. AI gives us insights into attackers' operations, helping us foresee vulnerabilities and potential exploits. It sharpens our ability to anticipate threats and craft more robust defenses. • Humans + AI = the ultimate power duo. AI handles the heavy lifting, but our judgment steers the ship. • We’re responsible for setting boundaries, ensuring ethical use, and validating decisions. • AI is here to empower us, not replace us.
15. We can build more innovative, faster, and more secure software, with our creativity, strategy and collaboration. Thank you Riotaro OKADA riotaro@owasp.org