Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cryptographic protection of ML models

vixentael
September 03, 2021

Cryptographic protection of ML models

Imagine a system that operates with ML models. These models are unique and work with user-generated content better than anyone else. For various business reasons, instead of running one large sophisticated model on the server, developers have to to run models on mobile devices (_viva TensorFlow!_). Our challenge is to protect these models from leakage and massive accumulation, which leads to reverse engineering of their unique approach.

This talk explains building DRM-like protection with application-level encryption using HPKE-like approach on ephemeral keys. We will discuss risks, threats, dataflow, cryptographic layer, key management and integration with traditional appsec controls for defense-in-depth approach.

vixentael

September 03, 2021
Tweet

More Decks by vixentael

Other Decks in Programming

Transcript

  1. @vixentael head of customer solutions, security software engineer focused on

    applied crypto, building e2ee protocols, and secure software development vixentael.dev
  2. cossacklabs.com Data security tools & solutions @vixentael We make software

    to get data security right – from open-source and proprietary cryptographic tools to custom solutions and consulting. We are cryptographers, security engineers, system engineers, infrastructure engineers.
  3. Working with companies that care about data security Critical infrastructure,

    healthcare, payment processors, ML/AI, popular apps — where data security is a hard requirement.
  4. @vixentael Things we won’t talk about Adversarial networks, adversarial attacks

    Malware inside ML networks Deserialization bug in TensorFlow -> arbitrary code execution https://arxiv.org/abs/2107.08590 https://portswigger.net/daily-swig/deserialization-bug-in-tensor fl ow- machine-learning-framework-allowed-arbitrary-code-execution ML “Unlearning” http://yinzhicao.org/unlearning/UnlearningOakland15.pdf https://arxiv.org/abs/1712.09665
  5. @vixentael What we will talk about Protecting IP Integrating cryptography

    with traditional security controls Application-level encryption cossacklabs.com/case-studies/ai-ml-ip-protection/ TensorFlow (a bit) HPKE-like scheme, DRM-like approach
  6. @vixentael Protecting unique IP (ML models) against leakage and misuse

    Extremely popular AI/ML application. ML models everywhere. They wanted to switch ML execution from backend-side to client-side to decrease load and improve service.
  7. @vixentael IP -> backend Client app sends request. Backend executes

    ML model. Backend sends ready result. Repeat every time. Before
  8. @vixentael Client app sends request. Backend executes ML model. Backend

    sends ready result. Repeat every time. Before After Client app sends request. Backend generates unique Individual ML model (IML). Backend sends IML to client app. 
 Client app stores and executes IML locally. IMLs are unique per user (1 app - 1 user - N models). IP -> backend, client-side
  9. @vixentael Business risks of decision R1 leakage of IP loss

    of IP, competitor advantage, investments into updating ML model. Losing 1 IML is not a problem, losing many IML is. R2 broken apps, clones apps, API fraud abuse of infrastructure, 
 revenue loss, 
 abuse of IP, competitor advantage, 
 reputation risks
  10. @vixentael Tech stack Native mobile apps: iOS – Swift/ObjC +

    CoreML Android – Kotlin + TensorFlow Lite python backend, TensorFlow GCP: workers (GCE), storage, KMS, DBs, Firebase authN
  11. API @vixentael Architecture and data fl ow user data GCE

    worker, TF native iOS app native Android app GCP, storing IMLs training servers main ML infra generating IMLs
  12. API @vixentael IML data fl ow user data GCE worker,

    TF native iOS app native Android app GCP, storing IMLs training servers main ML infra generating IMLs
  13. @vixentael IML lifecycle GCE worker generate IML, send to GCP

    storage memory, transit GCP storage store IML fi le storage, transit API URL on IML fi le transit mobile app download from GCP storage, 
 save locally as fi le, 
 unpack & execute IML transit, storage, memory
  14. @vixentael Threat modelling [simpli fi ed] API leakage via API,

    credential leakage, abuse of IML generation pretending to be a paying user collect from storage, fi nd in backups, fi nd in logs leakage / eavesdropping, client-server passive MitM, client-server active MitM extract IML via RE, crowdsourcing, automation of broken apps, malicious 3rd party libs Cloud storage Transit Mobile app
  15. @vixentael Authenticity Spoo fi ng DoS Tampering Disclosure Integrity Con

    fi dentiality Availability Threat modelling [simpli fi ed] API Cloud storage Transit Mobile app
  16. @vixentael What is ML model ML model – output of

    ML algorithm. A fi le. With model data and procedure/algorithm. Layers with weights. From security perspective – a fi le :)
  17. @vixentael Encryption layer API GCE worker, TF native iOS app

    native Android app GCP, storing IMLs encrypts each IML stores encrypted decrypts IML decrypts IML
  18. @vixentael Encryption layer: requirements 1. Minimize the lifetime of plaintext

    IMLs 2. Minimize the chance of accumulating IMLs 3. Fast, smooth, without complicated crypto 4. Easy key management, without PKI 5. Works across 3+ platforms
  19. @vixentael Encryption layer: solutions 1. Minimize the lifetime of plaintext

    IMLs 2. Minimize the chance of accumulating IMLs 3. Fast, smooth, without complicated crypto 4. Easy key management, without PKI 5. Works across 3+ platforms => encrypt after generation, decrypt before usage => use unique keys per IML => AES-256-GCM + ECDH => ephemeral keys => Themis crypto lib
  20. @vixentael IML encryption & decryption GCE worker, TF 1. Generate

    keypair. Send app.publicKey to backend. 2. Generate keypair. Use server.privateKey and app.publicKey to derive sharedKey (ECDH). 3. Generate random DEK. 4. Encrypt IML using DEK, AES-256-GCM. 5. Encrypt DEK using sharedKey, AES-256-GCM. 6. Send { encryptedIML, encryptedDEK, server.publicKey }. 7. Receive. Use app.privateKey and server.publicKey to derive sharedKey. 8. Decrypt DEK, decrypt IML.
  21. @vixentael IML encryption import pythemis server_keypair = GenerateKeyPair(KEY_PAIR_TYPE.EC) s_private_key =

    server_keypair.export_private_key() s_public_key = server_keypair.export_public_key() secure_message = SMessage(s_private_key, app_public_key) encrypted_DEK = secure_message.wrap(DEK) DEK = GenerateSymmetricKey() cell = SCellSeal(DEK) encrypted_IML = cell.encrypt(IML, userID) send: { encrypted_IML, encrypted_DEK, s_public_key } github.com/cossacklabs/themis
  22. @vixentael IML decryption import themis let keypair = TSKeyGen(algorithm: .EC)!

    let appPrivateKey = keypair.privateKey! let appPublicKey = keypair.publicKey! let cell = TSCellSeal(key: DEK)! let IML = try? cell.decrypt(encryptedIML, userID) let secureMessage = TSMessage(inEncryptModeWithPrivateKey: appPrivateKey, peerPublicKey: serverPublicKey)! let DEK = try? secureMessage.unwrapData(encryptedDEK) github.com/cossacklabs/themis
  23. @vixentael Crypto engine: Themis github.com/cossacklabs/themis same API across 14 platforms

    boring crypto hidden crypto-details recommended by OWASP tons of docs
  24. @vixentael Application-level encryption Encryption process happening within application context, triggered

    by application. ALE could work together with data-at-rest encryption and data- in-transit encryption. ALE could be client-side, server-side, end-to-end, etc. infoq.com/articles/ale-software-architects/
  25. @vixentael encryption controls / events transit (TLS) disk / FS

    TDE / DB encryption ALE E2EE physical access to servers ⛔ ✅ ✅ ✅ ✅ MitM ✅ ⛔ ⛔ ✅ ✅ privileged DB access ⛔ ⛔ ⛔ ✅ ✅ privileged system access ⛔ ⛔ ⛔ Depends ✅ backups, logs, snapshots ⛔ ⛔ Few ✅ ✅ infoq.com/articles/ale-software-architects/
  26. @vixentael Hybrid Public Key Encryption (HPKE) datatracker.ietf.org/doc/draft-irtf-cfrg-hpke/ encrypt data with

    symmetric key using AEAD; encapsulate symmetric key with public key scheme RFC describes approach used before and implies standardization.
  27. @vixentael Lightweight key management 1. Lightweight key management – server

    generates ephemeral keypair each time, no need for PKI. 2. NIST SP 800-57 – sorry, ephemeral keys FTW. 3. Store client-side public key in the user database to “pin” devices, or use ephemeral keypairs too. 4. Server authenticity problem – solve by server attestation, TLS pinning. 5. Mobile app storage problem – use Keychain/KeyStore, encrypt keys by SecureEnclave.
  28. @vixentael Crypto defense in depth: let new_DEK = TSGenerateSymmetricKey() let

    cell = TSCellSeal(key: new_DEK)! let encrypted_IML_ID = try? cell.encrypt(IML) 1. Re-encrypt IML on device on receiving (AES-256-GCM).
  29. @vixentael Crypto defense in depth: 1. Re-encrypt IML on device

    on receiving (AES-256-GCM). => to un-link server keys, to re-encrypt IML purely based on device keys Store re-encryption keys in Keychain/KeyStore. Bonus points for biometrics binding. let new_DEK = TSGenerateSymmetricKey() let cell = TSCellSeal(key: new_DEK)! let encrypted_IML_ID = try? cell.encrypt(IML)
  30. @vixentael Crypto defense in depth: GCE worker, TF IML encryptedIML

    generation encryption storage transfer + TLS transfer + TLS decryption re-encryption & storage execution encryptedIML encryptedIML IML IML encryptedIML 2. IMLs are encrypted after generation for storage, then using TLS for transport, then re-encrypted on device.
  31. @vixentael Crypto defense in depth: GCE worker, TF IML encryptedIML

    generation encryption storage transfer + TLS transfer + TLS decryption re-encryption & storage execution encryptedIML encryptedIML IML IML encryptedIML 😿 2. IMLs are encrypted after generation for storage, then using TLS for transport, then re-encrypted on device.
  32. @vixentael Crypto defense in depth: 3. In-memory encryption. CoreML requires

    plaintext model fi le when loads. => create MLCustomLayer with encrypted weights, decrypt before load to shader (CPU) => create custom shader function to obfuscate weights before execution on shader (GPU) (also see Apple docs on encrypting ML models that are parts of app bundle)
  33. @vixentael Performance considerations 1. GPU shaders have limited cache memory,

    can’t run “normal” ciphers. 2. Use fast crypto: ECC & AES-GCM. 3. Crypto adds performance penalty, but AES-GCM has hardware support everywhere. No noticeable UX penalty. 4. Some Android devices are extremely slow, but if the device can render ML with 50-60 FPS, it can run crypto fast. 5. Generating IMLs and encrypting them might be still faster than executing server-side ML for each request.
  34. @vixentael Overlapped security controls 1. Encryption to protect IMLs globally

    during the whole data fl ow. 2. Whatever is the attack vector, there is a defense layer. 3. For most popular attack vectors, we want as many independent defenses as possible. ✅
  35. @vixentael API GCE worker, TF native iOS app native Android

    app GCP, storing IMLs crypto crypto crypto crypto Integration with other security controls
  36. @vixentael Integration with other security controls API GCE worker, TF

    native iOS app native Android app GCP, storing IMLs AuthN AuthN TTL crypto anti- RE appsec crypto anti- RE appsec API sec anti- fraud crypto crypto ACL logging monitoring appsec
  37. @vixentael Cloud storage security 101 AuthN TTL ACL 1. IMLs

    are stored min time – apps are expected to grab their IML quickly. 2. URL TTL (expire after mins). 3. URL authentication & access control. 4. Clean up IML fi les (every hour). 5. Do not backup IMLs. 6. URLs are not logged. 7. Monitoring of access errors. (also see OWASP WSTG-CONF-11)
  38. @vixentael API protection 101 1. User authN, IMLs are available

    only after successful authN. 2. API limits, requests throttling, fi rewalling. 3. IML request limits – after N model requests, server returns error. (also see OWASP ASVS :) ) AuthN appsec API sec API
  39. @vixentael Anti-fraud system 201 1. Limit access to IML based

    on user behaviour. 2. Gather events from mobile apps and from server side. 3. Calculate user scoring based on events (“stop- factors”, rules). 4. User scoring: OK, suspicious, malicious. 5. Block malicious, limit suspicious. API anti- fraud
  40. @vixentael Anti-fraud system 201 JB detected same public key, different

    device invalid app signature remote device attestation failed 🛑 stop factors } URL download failure app reinstall too many requests keychain not accessible 🤨 implicative rules } wrong API version … honey token deviceID … malicious suspicious OK
  41. @vixentael Remote device attestation developer.apple.com/ documentation/devicecheck Apple DeviceCheck developer.android.com/training/ safetynet/attestation

    Android SafetyNet 1. Use as part of user authN. 2. Use as source for anti-fraud system. 3. Block apps installed not from stores.
  42. @vixentael Special improvements for ML models 1. Watermarks. 2. Custom

    ML layers. 3. Model binding (ML models that work only with custom data -> non- general purpose ML models, no risks to steal).
  43. @vixentael Integration with other security controls API GCE worker, TF

    native iOS app native Android app GCP, storing IMLs AuthN AuthN TTL crypto anti- RE appsec crypto anti- RE appsec API sec anti- fraud crypto crypto ACL logging monitoring appsec
  44. @vixentael Overlapped security controls 1. Encryption to protect IMLs globally

    during the whole data fl ow. 2. Whatever is the attack vector, there is a defense layer. 3. For most popular attack vectors, we want as many independent defenses as possible. ✅ ✅ ✅
  45. Failure of a single security control is a question of

    time. Failure of a security system is a question of design.
  46. @vixentael vixentael.dev/talks/use-crypto-dont-learn-it/ Use cryptography; don’t learn it infoq.com/articles/ale-software-architects/ Application Level

    Encryption for Software Architects, by @9gunpi cossacklabs.com/blog/crypto-signed-audit-logs.html Cryptographically signed audit logs cossacklabs.com/blog/react-native-app-security.html React Native security: things to keep in mind, by @julepka