Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Wind of Change

Wind of Change

How we perform changes at Auth0 to move fast while reducing risk

Damian Schenkelman

August 23, 2018
Tweet

More Decks by Damian Schenkelman

Other Decks in Technology

Transcript

  1. SLOs https://landing.google.com/sre/book/chapters/service- level-objectives.html • Latency: 99th percentile at edge (measured

    every 5 minutes) < 500 ms • Reliability: 99.99% of requests succeed (200-499 status code)
  2. Take me to the magic of the moment On a

    glory night Where the children of tomorrow dream away (dream away) In the wind of change
  3. duration[s] * requests impacted[%] * (period weight) / 100 (60*60*24*30)[s]

    * (error budget[%]/100[%]) * 100 req during period length avg req period length during month period weight = Consumption Real Model
  4. Schedule Region/Time Start End US 12 AM UTC 10 AM

    UTC EU 9 PM UTC 9 AM UTC AU 11 AM UTC 9 PM UTC
  5. Usage const belgrano = require('belgrano'); belgrano.init({ /* config */ });

    const getFlag = belgrano.entities.getFlag; if (await getFlag(entityId, flag)) { // enabled case } else { // disabled case }
  6. Rate limiting limitd-leveldb Client Client Client { key space 2}

    { key space 3} { key space 1} level level level limitd limitd limitd
  7. feature-change const feature_change = require('feature-change'); var options = { expected:

    function(cb){ search_v2(query, cb); }, actual: function(cb){ search_v3(query, cb); }, logAction: function(current_result, new_result){ // invoked when there is a difference in the results // (useful for logging) } }; feature_change(options, function(err, result){ // this is the original callback you were using for search v2 // err and result always come from search_v2 });
  8. Platform • Pick resources (CPU, memory, etc.) • Pick deployment

    method • Control routed traffic • Spinnaker and automatic anomaly detection
  9. Feature flags • % based flags • User Interface for

    managing • New stores (not just mongodb)