Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

RailsWorld_2023_-_Everything_We_Learned_the_Har...

Kylie
October 05, 2023

 RailsWorld_2023_-_Everything_We_Learned_the_Hard_Way_Implementing_ActiveRecord_Encryption.pdf

GitHub encrypts your data at rest, as well as specific sensitive database columns. What you may not know is we recently replaced our in-house db column encryption strategy with ActiveRecord::Encryption, in place. While we were able to complete this transition seamlessly for GitHub’s developers, the process was not quite seamless for our team and some of our customers.

Learn why despite making all these mistakes we feel the migration was worth it to our team, GitHub developers and most importantly, GitHub customers.

https://rubyonrails.org/world/agenda/day-1/4b-kylie-stradley-session

Kylie

October 05, 2023
Tweet

More Decks by Kylie

Other Decks in Programming

Transcript

  1. Kylie Stradley Senior Product Security Engineer at GitHub Matt Langlois

    Senior Product Security Engineer at GitHub You may recognize me from “ActiveRecord::Encryption, Stop Hackers from Reading your Data”
  2. Key generation bottleneck Easy to use API Before: bespoke internal

    strategy Maintaining divergent column encryption code
  3. Easy to use API After: ActiveRecord:: Encryption Less bespoke code

    Keys derived at time of writing code Easily upgrade columns to encrypted Centralized key rotation
  4. This talk is for (this will be some of you)

    Converting from an existing column encryption strategy
  5. This talk is for (this will be some of you)

    Those converting from an existing column encryption strategy (likely some more of you) Those working in distributed systems
  6. This talk is for (this will be some of you)

    Those converting from an existing column encryption strategy (this should be all of you) Those who will rotate their encryption keys (likely some more of you) Those working in distributed systems
  7. What We Thought We Knew About Encryption • The hardest

    part of encryption is key management • One byte in, one byte out • The price of “it just works” is performance
  8. What we thought “Key management is the hardest part of

    encryption” meant • FIPS Compliance
  9. What we thought “Key management is the hardest part of

    encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption
  10. What we thought “Key management is the hardest part of

    encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse
  11. What we thought “Key management is the hardest part of

    encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column
  12. What we thought “Key management is the hardest part of

    encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column • Secure Key Storage
  13. What we thought “Key management is the hardest part of

    encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column • Secure Key Storage - we use Hashicorp Vault
  14. Let’s say you have one Rails server Rails Server Database

    Encrypts record with key Encrypted Record
  15. Let’s say you have one Rails server…and you add a

    new key Key Update Mechanism Rails Server Database Append new key Encrypts record with new key Encrypted Record Encrypted Record
  16. Let’s say you have one Rails server…and you add a

    new key Key Update Mechanism Rails Server Database Append new key Encrypted Record Encrypted Record Decrypts with latest key
  17. Let’s say you have one Rails server…and you add a

    new key Key Update Mechanism Rails Server Database Append new key Encrypted Record Encrypted Record Decrypts with next key
  18. Let’s say your app is a bit bigger than just

    one Rails server… • A fleet of app servers • Serving 1000s of requests per second • 400,000 encryptions per day • 75,000,000 decryptions per day
  19. Let’s say your app is a bit bigger than just

    one Rails server… Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with key Encrypts record with key
  20. Let’s say your app is a bit bigger than just

    one Rails server… How will you update the key for all of your app servers? Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with new key Encrypts record with key Key Update Mechanism Encrypted Record Encrypted Record Append new key
  21. Let’s say your app is a bit bigger than just

    one Rails server… How will you update the key for all of your app servers? Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with new key Key Update Mechanism Encrypted Record Encrypted Record Append new key Decrypts with latest key
  22. What we learned “Key management is the hardest part of

    encryption” means • In a distributed system, to rotate keys, we also had to distribute keys
  23. What we learned “Key management is the hardest part of

    encryption” means • In a distributed system, to rotate keys, we also had to distribute keys • We needed a solution that ensured encryption would happen with new keys only once they were propagated to ALL servers
  24. What we learned -A two key strategy allows us to

    deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Append new decryption key Decrypts with latest key
  25. What we learned -A two key strategy allows us to

    deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Append new decryption key Decrypts with next key
  26. What we learned -A two key strategy allows us to

    deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Decrypts with latest key Latest decryption key propagated to all servers Update encryption key to use new value Encrypts record with new key Encrypted Record
  27. What you can learn from our experience - Key Management

    • We learned this during a planned test of our key rotation strategy
  28. What you can learn from our experience - Key Management

    • We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system
  29. What you can learn from our experience - Key Management

    • We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system • Understand your capabilities for updating keys for your production servers
  30. Understand how you will update keys for production servers Can

    you update all keys at once? How can you roll out keys to prevent decryption failure?
  31. Understand how you will update keys for production servers Do

    you update keys via push or pull? What triggers a push? If you pull, do you poll for updates? How often? Can you update all keys at once? How can you roll out keys to prevent decryption failure?
  32. Understand how you will update keys for production servers Do

    you update keys via push or pull? What triggers a push? If you pull, do you poll for updates? How often? Can you update all keys at once? How can you roll out keys to prevent decryption failure? How and when is data migrated? Is your database sharded? Will data ever move between shards? How will you re-encrypt records for key rotation?
  33. What you can learn from our experience - Key Management

    • We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system • Understand your capabilities for updating keys for your production servers - This lets you design rollout with key rotation as top priority
  34. What we thought we knew “One byte in, one byte

    out. All of our ciphertexts are the same size”
  35. What we thought “All of our ciphertexts are the same

    size” meant • AES GCM 256 works like a stream cipher
  36. What we thought “All of our ciphertexts are the same

    size” meant • AES GCM 256 is a block cipher - ciphertexts will be 128 bits
  37. What we thought “All of our ciphertexts are the same

    size” meant • AES GCM 256 is a block cipher - ciphertexts will be 128 bits • Plaintext columns may need to be resized
  38. What we thought “All of our ciphertexts are the same

    size” meant • AES GCM 256 is a block cipher - ciphertexts will be 128 bits • Plaintext columns may need to be resized - just need to be able to hold 128 bits
  39. What we thought “All of our ciphertexts are the same

    size” meant • AES GCM 256 is a block cipher - ciphertexts will always be 128 bits • Plaintext columns may need to be resized - just need to be able to hold 128 bits • Our previous encryption scheme stored some metadata
  40. What we thought “All of our ciphertexts are the same

    size” meant • AES GCM 256 is a block cipher - ciphertexts will always be 256 bits • Plaintext columns may need to be resized - just need to be able to hold 256 bits • Our previous encryption scheme stored some metadata - but not quite the same amount Rails does
  41. What we learned “All of our ciphertexts are the same

    size” means • All of our ciphertext are the same size, however ciphertext is not all that is stored in the database
  42. What we learned “All of our ciphertexts are the same

    size” means • All of our ciphertext are the same size, however ciphertext is not all that is stored in the database • “Anti-nonce-exhaustion-data” is quite a bit of text
  43. Database Record Comparison Previous Strategy "\x02\e\x9FU#\xD3\xA8p\x84l\b\xED\x96\xEA3z,\xDFZ\xC 9-\"\\\xC1\xC7\x86\xA1\xCC\x8A\x8D]\xDB\xC2\x99\xE7`S )#\xCCp\xFD\x81\x9E\x01|$" ActiveRecord::Encryption {

    "p"⇒"nf6EwvS6rsSEErZyjTIRhgy7Gdkj4Wj4iQOtIRphP2p4xeozIL2xdDcO3sA=", "h"⇒ { "iv"⇒"TFXpap3Wpf4OEvtZ", "at"⇒"mbg4Au2ECg2iumLb1WhTsw==", "e"⇒"QVNDSUktOEJJVA==", "i"⇒"MTEwYQ==", "anti_nonce_exhaustion_data"⇒"MjAyMw==" } } Key ID Encrypted message Encrypted message (payload) Message headers
  44. What we learned “All of our ciphertexts are the same

    size” means • All of our ciphertext are the same size, however ciphertext is not all that is stored in the database • “Anti-nonce-exhaustion-data” is quite a bit of text • Recommendation for all columns to be converted to mysql TEXT before conversion to ActiveRecord::Encryption – we do not allow indexing on encrypted columns
  45. What you can learn from our experience - Ciphertext vs

    Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext
  46. What you can learn from our experience - Ciphertext vs

    Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason tags are commonly one character
  47. What you can learn from our experience - Ciphertext vs

    Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason tags are commonly one character • Make it easy for your engineers
  48. What you can learn from our experience - Ciphertext vs

    Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason “i” is commonly used as the tag name for anti-nonce exhaustion data! • Make it easy for your engineers - consider longevity when recommending column length
  49. What we thought “it just works!” meant • When something

    “just works!” you pay a price, we assumed this would be performance
  50. What we thought “it just works!” meant • When something

    “just works!” you pay a price, we assumed this would be performance – moving from one encryption scheme to another so considered negligible
  51. What we thought “it just works!” meant • When something

    “just works!” you pay a price, we assumed this would be performance – moving from one encryption scheme to another so considered negligible • Through monitoring we thought we found out “it just works” was a function of idempotency
  52. Our upgrade strategy relied on a type to feature flag

    Plaintext Encrypt? Model.save Database Ciphertext Type: Text Type: Encrypted Attribute
  53. Our upgrade strategy relied on a type to feature flag

    … but we neglected to delegate changed_in_place, resulting in records that looked like they had changed Model.save Database Ciphertext Type: Encrypted Attribute changed_in_place? ciphertext plaintext != true
  54. We fixed this by delegating the changed_in_place? method to the

    EncryptedAttributeType Model.save Database Ciphertext Type: Encrypted Attribute changed_in_place? decrypted ciphertext plaintext != false
  55. What we learned “It just works” means • It did

    just work for most cases - but we had a special case • Monitoring can help catch special cases
  56. What we learned “It just works” means • It did

    just work for most cases - but we had a special case • Monitoring can help catch special cases - in production
  57. What we learned “It just works” means • It did

    just work for most cases - but we had a special case • Monitoring can help catch special cases - in production • Some data is just special and you need to take extra time and care to get it right
  58. What you can learn from our experience - “It just

    works!” • Encrypted data is being encrypted for a reason - there may be special monitors or side effects associated with these records
  59. We delivered a seamless column encryption strategy • We were

    no longer maintaining divergent column encryption code • New, easy process to upgrade plaintext columns to encrypted • Encryption keys are now derived at time of writing code instead of generated – greatly increasing adoption • Maintained our SLOs
  60. And keeping a few things in mind, you can too

    • Build with Key Rotation and Key Management in mind first • Understand why a column should be encrypted and what side effects there are on those records
  61. Further Reading • https://gh.io/ar-encryption-blog-post • https://gh.io/ar-encryption-blog-post-2 • https://gh.io/ar-encryption-guide • Rails

    Conf Talk - “ActiveRecord::Encryption, Stop Hackers from Reading your Data” • The “Real World Cryptography” book by David Wong • Google’s “Building Secure and Reliable Systems” book Rails World 2023 - Kylie Stradley