hours) (during business hours) ‒ First-line support: “I see that your cluster is red” ‒ “Please give us the output of these API endpoints …” • 2. Escalate to ES team engineers ‒ “We see that one of your nodes needs to be shot” ‒ “We see JVM memory pressure is high, please try to reduce it” ‒ “Can you maybe stop logging so much?” ‒ Wait some more • 3. Expedite, option 1: call the TAM ‒ Eventually started going directly through TAM to engineers, who knew the routine • 4. Expedite, option 2: roll the cluster ‒ Trivial change to IAM role ⇒ get an entirely new cluster (blue/green deploy) ‒ Would often get stuck “between” deploys, old nodes sticking around ‒ Still requires manual intervention by AWS support You have opened a new Support case 13