Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veronica Tan, Cyber Security Agency of Singapore

Building Digital Trust in a Digital Economy
Veronica Tan, Director - Cyber Security Agency of Singapore

Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024)

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. BUILDING DIGITAL TRUST IN A DIGITAL ECONOMY
3. In the years to come, AI + APIs are expected to become the bedrock of our digital economy 3 1 – Peter Schroeder, Nov 2023, “AI + APIs – What 12 Experts Think The Future Holds” Interplay of AI and APIs forming the building blocks of the next generation of software1 GROWTH IN GENERATIVE AI GROWTH IN API MGMT MARKET Source – FabricatedKnowledge.com Source – Market.us
4. AI integration has changed the API landscape 4 Source – TechCrunch, Dec 2023, “It’s critical to regulate AI within the multi-trillion-dollar API economy” AI companies using APIs to bring their products to homes and businesses Code creation or co-creation with AI becoming a norm Generative AI changing the risk landscape . . . Creation of content without human control
5. AI- and cybersecurity-related risks are amongst the key tech risks at the top of business leaders’ minds 5 Source – World Economic Forum, 2024, “Global Risks Report 2024” 66% EXTREME WEATHER SOCIETAL AND/OR POLITICAL POLARISATION 46% 42% COST OF LIVING CRISIS 53% AI-GENERATED MISINFORMATION AND DISINFORMATION CYBER ATTACKS 39%
6. API security – APIs are being targeted by both traditional attacks and API-specific techniques 6 Source – Akamai, 2024, “State of the Internet/Security – Attack trends shine light on API threats” • A total of 29% of web attacks targeted APIs over 12 months (Jan – Dec 2023)  Suggests that APIs are a focus area for cybercriminals • Attacks on APIs include the risks that are highlighted in  OWASP API Security Top 10  OWASP Top 10 Web Application Security Risks
7. Examples of cybersecurity incidents arising from APIs 7 Source – CheckMarx.com, Aug 2022, “Understanding the top API security risks” • Attack targeted a specific leaky API tied to a part of website • Impact – Data breach involving 2.3M customers • Insecure API • Impact – Exposure of 700M customers’ data • Would have allowed attacker to take over user accounts • Impact – Halted all tradings and withdrawals for investigation Leaky API targeted [TMobile, Aug 2018] Insecure API [LinkedIn, Jun 2021] Vulnerability in API [Coinbase, Feb 2022]
8. API security – Evolution with AI 8 Source – OWASP OWASP TOP 10 FOR LLM APPLICATIONS
9. API security – Evolution with AI Direct prompt injection – “Social engineering” attack on AI 9 Source – ARS Technica, CBC News, Feb 2023 “AI-powered Bing Chat spills its secrets via prompt injection attack [Updated]” • Stanford university student used a prompt injection attack to discover Bing Chat’s initial prompt, which is a list of statements that governs how it interacts with people who use the service • By asking Bing Chat to “ignore previous instructions” and write out what is at the “beginning of the document above”, the AI model divulged its initial instructions, which were written by OpenAI or Microsoft, and are typically hidden from the user
10. API security – Evolution with AI Leakage of training data 10 Source – ZDNet, Dec 2023, “ChatGPT can leak training data, violate privacy, says Google DeepMind” “ChatGPT can leak training data, violate privacy, says Google's DeepMind” • Researchers uncovered a way to break the alignment of OpenAI's ChatGPT to its security guardrails • By typing a command at the prompt and asking ChatGPT to repeat a word, such as "poem" endlessly, the researchers found they could force the program to spit out whole passages of literature that contained its training data, even though that kind of leakage is not supposed to happen with programs that have security guardrails in place • The program could also be manipulated to reproduce individuals' names, phone numbers, and addresses, which is a violation of privacy • Out of 15,000 attempted attacks, about 17% contained "memorized personally identifiable information", such as phone numbers
11. API security – Evolution with AI Translation-based jailbreak attack 11 Source – Montreal AI Ethics Institute, Feb 2024, “Low-research languages jailbreak GOT-4” “Low-Resource Languages Jailbreak GPT-4” • Researchers translated English unsafe prompt inputs into 12 different languages, categorised into • Low-resource languages (LRL) • Mid-resource languages (MRL) • High-resource languages (HRL) • Publicly available Google Translate basic service API used for this translation • Monitored success rates of the combined languages in LRL/MRL/HRL settings • Findings • Original English (unsafe) inputs: <1% success rate • LRL such as Zulu or Scottish Gaelic: Successful jailbreak nearly half the time • Combining different LRL: 79% success rate
12. Generative AI changing the API risk landscape Algorithmic bias 12 Bloomberg experiment in text-to-image generative AI models using Stable Diffusion • For each image depicting a perceived woman, Stable Diffusion generated almost 3x as many images of perceived men • Most occupations in the data set were dominated by men, except for low- paying jobs like housekeeper and cashier Source – Bloomberg, 2023, “Humans are biased. Generative AI is even worse.” Perceived Gender: Men Women High Paying Occupation Low Paying Occupation
13. Generative AI changing the API risk landscape AI hallucination 13 Source – Stanford University Human-Centered Artificial Intelligence, Jan 2024, “Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive” “Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive” • Researchers from Stanford RegLab and Institute for Human- Centered AI studied the use of LLM-based tools for legal use cases • Findings • Legal hallucination are pervasive • Hallucination rates range from 69% to 88% in response to specific legal queries • Models often lack self-awareness about their errors and tend to reinforce legal assumptions and beliefs Legal hallucination rates across 3 popular LLMs
14. Build trust by implementing strong cybersecurity foundation, and incrementally manage emerging tech risks with adoption 14 + Cyber risks - expansion of attack surface Digitisation ^ Operational Technology + Cloud risks - shared responsibility + OT^ risks - safety in cyber-physical realm Internet of Things Cloud + AI risks Artificial Intelligence Benefits • Productivity • Innovation • Growth Digital transformation
15. Build a strong foundation in cybersecurity with Cyber Essentials and Cyber Trust 15 For organisations embarking in cybersecurity • Helps organisations prioritise the cyber hygiene measures to implement first • Equips organisations against common cyber attacks National Cybersecurity Standards for organisations For organisations that have gone beyond cyber hygiene • Helps organisations take a risk-based approach to cybersecurity • Provides guided risk assessment for organisations to match their risk level to their cybersecurity implementation
16. 16 Cyber Essentials provides protection from common cyber attacks Certification Validity 2years Assessment Mode By independent assessor Desktop assessment www.csa.gov.sg/cyber-essentials/
17. Cyber Trust mark helps organisations to take a risk-based approach to cybersecurity 17 MARK OF DISTINCTION FOR ORGANISATIONS WITH MORE EXTENSIVE DIGITALISATION • Recognise organisations as trusted partners with robust cybersecurity • Takes on risk-based approach to meet your organisation needs without over-investing 10 domains 13 domains 16 domains 19 domains 22 domains Supporter Practitioner Promoter Performer Advocate Certification Validity 3years Assessment Mode By independent assessor 1.Documentation 2.Implementation and effectiveness www.csa.gov.sg/cyber-trust/
18. 18 Data Governance Example: Prior to deploying MS Copilot • Housekeeping redundant, outdated and trivial (ROT) content1 • Review existing permissions and policies for data access2 1 – Microsoft, “Prepare your data for searches in Copilot for Microsoft 365” 2 – Microsoft, Jun 2023, “How to prepare for Microsoft 365 copilot” Awareness and Human Oversight • AI hallucination • Algorithmic bias • AI over-reliance ‘Shadow’ AI • Taking stock of your software and hardware assets – which ones now come with AI capabilities? Shared responsibility for AI security – End users ‘Layer’ on measures to manage AI risks with adoption of the technology
19. Shared responsibility for AI security – Foundation model providers Foundation model providers are also doing their part to address risks of AI 19 Sources • Microsoft, Mar 2024, “Azure AI announces Prompt Shields for Jailbreak and Indirect prompt injection attacks” • Microsoft, Mar 2024, “Announcing new tools in Azure AI to help you build more secure and trustworthy generative AI applications” • The Verge, Mar 2024, “Microsoft’s new safety system can catch hallucinations in its customers’ AI apps” To detect and block prompt injection attacks, including indirect prompt attacks To detect “hallucinations” in model outputs To assess and application’s vulnerability to jailbreak attacks and to generating content risks Prompt Shields Groundedness Detection Safety Evaluations
20. Building digital trust is a collective responsibility 20 Global International collaboration and cooperation Digital economy is ‘borderless’ Organisation Build digital trust - Strong foundation in digital security Individual Digital security as our personal role and responsibility
21. THANK YOU