Scenario: You’ve just arrived at work and elect for the side door to avoid detection. You beeline it to the break room to initiate the day with the familiar hum of the local Keurig machine. As you saunter back to the relative comfort of your desk, coffee in tow, you sense the buzz of a message from your phone: it’s your colleague. The message reads:
Unbeknownst to you, one of your key systems just decided it’s had enough and your SaaS is down. In the background, the chatter becomes increasingly uneasy. Slack messages begin to fly. Pigs begin to fly. Panic ensues.
You reach for the top shelf and manage to fish down a binder labeled “Disaster Recovery Plan”; it’s weight creates a jarring thwop as it falls on your desk. You dust off the cover, scoffing at the apparent age of the relic, and begin nervously thumbing through it, scouring the content for answers. The pages begin to stick to the sweat formulating on your fingertips.
In the world of software, something inevitably breaks; SaaS is not immune. Bugs slip through our defenses, data centers have outages, and the sheer number of software dependencies continues to increase. With all of these variables in play, the footprint for disaster recovery continues to swell. However, all is not doom and gloom. DevOps teams, IT teams, and even marketing teams build in safety nets to reduce the possibility of things going awry.
Here are a few strategies we turn to when our SaaS is down. So take a deep breath, remove yourself from the fetal position, and take comfort in a few tips from those who have come before you.
Be “Customer Centric”
When your strategy revolves around your customers, you can avoid some common traps that teams fall into when disaster strikes. Sometimes employees can get caught up in soothing internal stakeholders rather than focusing efforts on mitigation. Stop drafting that internal status report that few will read and start a dialogue with your customers in the channels where they communicate. If you don’t know those channels, you’ll soon find out. Your customers hired you for their “jobs to be done.” Now is your chance to earn the privilege of calling them a customer — even at the end of a tough day.
At Brandfolder, we’re a tight-knit group. We use enterprise tools to keep everyone accessible. Because we’re able to act small, we move quickly and fluidly. And because communications cross so few barriers, we avoid reliance on out-of-sync mediums. This is especially useful when sh#t hits the fan and your SaaS is down.
Knowing your internal team is half the battle. Nothing is worse than communicating with a mysterious colleague for the first time during a situation that’s difficult and stressful for you both. Take some time to get to know the people you’ll be working with during an outage, and understand their working styles and motivations. You’ll likely find that your small investment will yield huge results.
Learn From It
Once you’ve recovered, the instinct to put the past behind you can be alluring: don’t be tempted. Just as in software development, your product nor your processes should ever stop evolving. Use the trials and tribulations you just suffered to proactively bolster your disaster recovery plan. Conduct a brief retrospective on the experience and note the things you need to improve upon. Using agile software methodologies as a resource, the simple “start, stop, continue” retrospective can be powerful. And make sure that you have a shared understanding of who’s responsible for each incremental improvement.
This happens. You are not the first person to experience a SaaS outage, and unfortunately, you are not the last. By keeping your customers the focus, maintaining close communications with your team, and conducting a productive review of what went wrong, you’ll emerge from an undoubtedly bad situation as a stronger team with a better product. And you might just impress your customers by how well you and your team handled yourselves throughout.