Had a strange thought last year, that if a system runs too smoothly, eventually the knowledge of dealing with breakage will disappear, and when it inevitably breaks, everyone will be unprepared.
So a little bit of breakage is like a healthy "exercise."
Presumably, a functional system is supposed to have some kind of drills to fill in for that. But I have seen ones that don't!
Very true. If things are too reliable, systems can come to depend on them always being so so reliable... So it can actually pay off to inject transient issues deliberately.
Had a strange thought last year, that if a system runs too smoothly, eventually the knowledge of dealing with breakage will disappear, and when it inevitably breaks, everyone will be unprepared.
So a little bit of breakage is like a healthy "exercise."
Presumably, a functional system is supposed to have some kind of drills to fill in for that. But I have seen ones that don't!
Very true. If things are too reliable, systems can come to depend on them always being so so reliable... So it can actually pay off to inject transient issues deliberately.
https://netflix.github.io/chaosmonkey/ and similar can help.
That said ... most instability is introduced with normal changes, so every engineer can be a chaos monkey. ;-)
Oh, the legendary backup restore drill? I'm sure it will happen someday ...
Nice that it is cc-commons.
Unfortunately site would need some refurbishment for mobile… slide show doesn’t show, and page is floating.