Nobody expects a double disk failure

[a_gs]

Yesterday I was doing some remote maintenance on the company's servers while sitting at a coffee shop. A combination of OpenVPN and ssh allows me to access everything from outside, serial consoles don't let me lose the connection to servers in the case of some severe networking misconfiguration, or installing a bad kernel, so I didn't have to be in

Comments

[mackys]

But there is always a possibility that the only way to fix the problem is to restore from backups.

I think this is one of the lessons that Evi hammered into me, along with "always check your return values from system calls." There's always some way for the system to fail such that backups are the only way to get your data back. The moral? Always have backups.

Uh... what was my point? Oh yeah: ME TOO! ;]

[dagbrown]

Did you buy the drives as a batch? Nice and close serial numbers, manufactured at the same plant on the same day, like?

When building a RAID, always mix and match brands. At least then the failures won't cluster together as much.

[a_gs]

I have invented a physical object replication technology, made drives using it and placed those drives into a RAID1 array for closest conditions possible, specifically to see how many of them will end up failing simultaneously.

(Of course, I know. Even identical firmware bugs can defeat redundancy.)