Don’t let the fear of breaking prod stop you
You will break prod at some point in your career. You should try not too, but you probably will – and if you don’t, someone or something else will. The main point is that it’s OK to accidentally break prod in most software companies (e.g. if you work for an ads business).
One thing that is important is to not let the fear of breaking prod stop you from moving forward. A few things that may help:
- If you work in a large company, the simplest bug (not even outage) may result in more money lost than you make in a year – but you shouldn’t compare this potential loss with your income but more for the company overall. We are not paid a percentage of the income we bring to the company, so it’s unfair to compare a company loss to our income.
- If you are scared of shipping features because something may break, you have to realize that your company must ship new/better products to survive. So there’s no choice but to move forward.
- The best way for you, your peers and the company to become better is to learn from experience, including outages. Google didn’t build such great software at once. As long as you are careful and keep learning from your mistakes, things will be OK.
- If you can make your change safer (e.g. by adding more tests, having better monitoring infrastructure etc.), you should do it if reasonable – if not, you should argue with your leadership to staff such efforts.
Fun fact: during my tenure at Google, I did break all YouTube Ads in a 1% experiment once (or a few times). It doesn’t quite translate to 1% of YouTube Ads revenue, but it was still a non trivial number.
With that being said, if you work on software whose people’s life depends on it, please be extra careful 😅