Gremlin

“We developed our team's expertise to handle issues that other teams couldn't test. At the same time, we championed Chaos Engineering, demonstrating its value in rapidly evaluating various scenarios.””

“Given how chaotic the systems are, this was why it was so important to us to test this chaos instead of testing it in front of the customer.”

“How do you know you won? Part of it is you have to know it’s still working the next day. It’s not that it’s working 10 minutes after you released it—it still works a month, 10 months later. It just works and it self heals, and the only way you know that is to go create the chaos.”

““I haven’t seen anything that competes,” Chawla says of Gremlin. “Gremlin has a focus and you nailed the focus. I’m looking for tools that do what they do well and better than everybody else.””

““We identified a few defects in an application, and we were impressed with how easy it was for us to identify unknowns. We were quite excited, so we started to share this with other teams.””

““Gremlin checked a lot of the boxes for us—it has a control plane and simple design which means the teams don’t have to spend time building the experiments and can focus on running them and improving reliability based on results,””

“Rather than resilience being a place you’ve got to get to, we've shifted to, 'How do we bake it into everything?'" the Lead Software Engineer explains. "Put the cookies on the lower shelf and make it accessible, and then track who opens the cookie jar and extrapolate how much value they got.”

“Everyone within the organization knows what it is," the Lead Software Engineer says. "They're planning their own experiments. They're thinking of ways to fail [infrastructure and applications]. They have the mindset now and the expectation.”

“They said, 'This fix has gotta work next holiday. This is a super important service,'" the Lead Software Engineer explains. "So we said, okay, we'll definitely run a game day this time.”

“Tech stack diversification has only increased," the Lead Software Engineer explains. "So there’s still a lot more work to be done. Turnover has increased. Teams get really good at resilience, but then there’s rotation- the maturity model goes up and then back down.””

“It was because of this bizarre issue with a live front-end library that you would have never guessed caused this particular behavior," their Lead Software Engineer says. "Basically, it would have reoccurred- and that woke everybody up.””

“Even though people don't plan for failure, it happens all the time,” explains the Lead Software Engineer, who had been with the chaos engineering program since its inception. “And when you hold the pager and you're responsible for the thing working in production, they're not gonna ask your boss or your boss's boss. They're gonna ask you.”

“Kolton [Gremlin’s CEO] was always saying, 'It's all about application fault injection. That's the ultimate goal. And now with Failure Flags, it’s possible — that's the best product out there. It's fantastic.”

“Gremlin has been so much more than just a tool. It’s been a lot of collaboration, a lot of sharing of ideas. It transcends the tool itself.”

“I liked Gremlin’s user interface, just how the tool itself is put together. It makes it very easy to achieve different goals. And I hadn’t seen anything in the market that was quite as easily used that would not require a lot of investment for me to establish a reliability practice. It has definitely lived up to expectations from the very beginning.”