KubeCon + CloudNativeCon Europe 2023

This is a real life story of how our company had an operational incident caused by misconfiguration and the most reliable way to get everything working again was to rebuild our entire cluster from scratch. This was only possible because of the investments we had made into GitOps, ArgoCD, kOps and keeping our infrastructure as code. Out of nowhere our cluster was failing in a way we had never seen before. Our standard backup and recovery methods were not working, etcd was inaccessible, customers were down. Our last card was to recreate the entire cluster and re-install all our services. We had practiced it many times but this would be the first real disaster recovery. Would all our planning and migration to GitOps and ArgoCD pay off, could things be brought back to a healthy state? How fast could we do it? In the end we managed to recreate the cluster in 51 minutes and we learned a lot along the way. Many of the tools we invested in did not work as expected, disaster recovery guides were outdated and things we had never planned for occurred. We talk about the workarounds we had to employ, the work we had to do afterwards and how we plan to improve on this in the future (what we learned).