Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Wind of Change

Wind of Change

How we perform changes at Auth0 to move fast while reducing risk

Damian Schenkelman

August 23, 2018

Transcript

49 slides
  1. SLOs https://landing.google.com/sre/book/chapters/service- level-objectives.html • Latency: 99th percentile at edge (measured

    every 5 minutes) < 500 ms • Reliability: 99.99% of requests succeed (200-499 status code)
  2. Take me to the magic of the moment On a

    glory night Where the children of tomorrow dream away (dream away) In the wind of change
  3. duration[s] * requests impacted[%] * (period weight) / 100 (60*60*24*30)[s]

    * (error budget[%]/100[%]) * 100 req during period length avg req period length during month period weight = Consumption Real Model
  4. Schedule Region/Time Start End US 12 AM UTC 10 AM

    UTC EU 9 PM UTC 9 AM UTC AU 11 AM UTC 9 PM UTC
  5. Usage const belgrano = require('belgrano'); belgrano.init({ /* config */ });

    const getFlag = belgrano.entities.getFlag; if (await getFlag(entityId, flag)) { // enabled case } else { // disabled case }
  6. Rate limiting limitd-leveldb Client Client Client { key space 2}

    { key space 3} { key space 1} level level level limitd limitd limitd
  7. feature-change const feature_change = require('feature-change'); var options = { expected:

    function(cb){ search_v2(query, cb); }, actual: function(cb){ search_v3(query, cb); }, logAction: function(current_result, new_result){ // invoked when there is a difference in the results // (useful for logging) } }; feature_change(options, function(err, result){ // this is the original callback you were using for search v2 // err and result always come from search_v2 });
  8. Platform • Pick resources (CPU, memory, etc.) • Pick deployment

    method • Control routed traffic • Spinnaker and automatic anomaly detection
  9. Feature flags • % based flags • User Interface for

    managing • New stores (not just mongodb)

More Decks by Damian Schenkelman

Other Decks in Technology