Comments on: Handling Human Error In the Datacenter http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/ Thu, 26 Aug 2010 16:46:48 -0600 http://wordpress.org/?v=2.8.4 hourly 1 By: Tom http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-4400 Tom Sun, 01 Mar 2009 18:06:09 +0000 http://www.spiteful.com/?p=64#comment-4400 @Mathew Duafala -- agreed, that is a really good idea. @Mathew Duafala — agreed, that is a really good idea.

]]> By: Mathew Duafala http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-4392 Mathew Duafala Sun, 01 Mar 2009 04:12:00 +0000 http://www.spiteful.com/?p=64#comment-4392 I just saw him give this talk at Amazon last week. It was really really good. The best thing that I got out of it was the concept of a 'canary host'. That's a host that you run hotter than the rest (2x or more) so that you can see it brown out and fail before the rest of the hosts. It lets you see what the failure behavior is and gives you warning before your entire fleet of boxes are failing at once. I just saw him give this talk at Amazon last week. It was really really good. The best thing that I got out of it was the concept of a ‘canary host’. That’s a host that you run hotter than the rest (2x or more) so that you can see it brown out and fail before the rest of the hosts. It lets you see what the failure behavior is and gives you warning before your entire fleet of boxes are failing at once.

]]> By: website designs http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-1843 website designs Fri, 22 Aug 2008 18:26:52 +0000 http://www.spiteful.com/?p=64#comment-1843 I'm not so sure about his `$machineType-$number` scheme. When dealing with the physical boxen, I find it a lot easier to just say "power down lucy" or "attach that FireWire drive to charlie". CNAMEs exist so that I can map these physical names to their roles -- my home directories will always live on "users", even after I migrate them to a SAN. "www" will always be my webserver, even after I migrate it to a virtual machine. I’m not so sure about his `$machineType-$number` scheme. When dealing with the physical boxen, I find it a lot easier to just say “power down lucy” or “attach that FireWire drive to charlie”. CNAMEs exist so that I can map these physical names to their roles — my home directories will always live on “users”, even after I migrate them to a SAN. “www” will always be my webserver, even after I migrate it to a virtual machine.

]]> By: samson http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-1751 samson Wed, 13 Aug 2008 18:32:16 +0000 http://www.spiteful.com/?p=64#comment-1751 A minor tool, but molly-guard has saved me more than once. Basically if you call shutdown/reboot from an ssh session, it prompts you for the hostname that you wish to shutdown, hopefully preventing a shutdown of a server that isn't within physical reach in the middle of the night. http://packages.debian.org/etch/molly-guard Capistrano looks interesting though, I'll have figure out what it gets me if I'm not using RoR. A minor tool, but molly-guard has saved me more than once.
Basically if you call shutdown/reboot from an ssh session, it prompts you for the hostname that you wish to shutdown, hopefully preventing a shutdown of a server that isn’t within physical reach in the middle of the night.

http://packages.debian.org/etch/molly-guard

Capistrano looks interesting though, I’ll have figure out what it gets me if I’m not using RoR.

]]> By: Adam http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-1741 Adam Tue, 12 Aug 2008 18:47:22 +0000 http://www.spiteful.com/?p=64#comment-1741 This is really more about automation, but tools like Capistrano (http://capify.org) can be really helpful for managing a number of servers in different roles. I don't have a lot of experience with this tool (it's written in Ruby and used heavily by the Rails community), but it seems useful. To the extent that this reduces the number of steps required to do something (even if those steps are simple things like opening an ssh session), you can reduce mental fatigue and stay focused on getting right the things that do require some mental effort. Are there any other tools like this that people recommend? This is really more about automation, but tools like Capistrano (http://capify.org) can be really helpful for managing a number of servers in different roles. I don’t have a lot of experience with this tool (it’s written in Ruby and used heavily by the Rails community), but it seems useful.

To the extent that this reduces the number of steps required to do something (even if those steps are simple things like opening an ssh session), you can reduce mental fatigue and stay focused on getting right the things that do require some mental effort.

Are there any other tools like this that people recommend?

]]> By: Trevor http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-1731 Trevor Tue, 12 Aug 2008 02:46:52 +0000 http://www.spiteful.com/?p=64#comment-1731 That's why I use Slicehost. RAID10, nightly backups and someone else getting paged when stuff breaks. That’s why I use Slicehost. RAID10, nightly backups and someone else getting paged when stuff breaks.

]]> By: Ammon Lauritzen http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-1730 Ammon Lauritzen Tue, 12 Aug 2008 01:50:13 +0000 http://www.spiteful.com/?p=64#comment-1730 I don't think you can really stress the importance of renaming files (rather than deleting them) enough. Whenever I'm making a huge configuration change to a live system, I copy the entire directory about to be affected... and leave the copy around. Most of the time, that copy remains untouched until I'm making another big change (at which point I'll generally finally delete it in favor of a backup of the current live version). But maybe 1 in 3 or 1 in 4 times... that backup will save you hours of work, or more. One general rule of thumb of server maintenance that you didn't mention is to always operate as a normal user until you're absolutely sure you need to escalate privileges. And then make sure to relinquish root when you're done with that individual task. It requires more typing, but if you can resist the urge to leave root prompts open on 5 different terminals at once... ;) I don’t think you can really stress the importance of renaming files (rather than deleting them) enough.

Whenever I’m making a huge configuration change to a live system, I copy the entire directory about to be affected… and leave the copy around. Most of the time, that copy remains untouched until I’m making another big change (at which point I’ll generally finally delete it in favor of a backup of the current live version). But maybe 1 in 3 or 1 in 4 times… that backup will save you hours of work, or more.

One general rule of thumb of server maintenance that you didn’t mention is to always operate as a normal user until you’re absolutely sure you need to escalate privileges. And then make sure to relinquish root when you’re done with that individual task. It requires more typing, but if you can resist the urge to leave root prompts open on 5 different terminals at once… ;)

]]> By: Nick http://www.tomkleinpeter.com/2008/08/11/handling-human-error-in-the-datacenter/comment-page-1/#comment-1726 Nick Mon, 11 Aug 2008 23:18:09 +0000 http://www.spiteful.com/?p=64#comment-1726 I rock 3 monitors at work and always have screens divided mentally by importance. Screen one is live / check it all twice before hitting enter type of data. Screen two is usually secondary, test or backup servers to pay attention to what was done so it can be duplicated. Screen three is email and web research... Or depending on the situation each screen is a different geographic location. Biggest Trip Ups I have to watch out for are the "Are you in the right directory? Are you on the right machine? Right Data? Right Dates?" type stuff Side Note: The instructor to a CCNA class I was taking walked around during our practice tests with a cheep cell phone refrigerator magnet and simulated the "real world" by making it ring while we took the test. Damn thing stressed us out, but didn't even come close to the owner calling you wanting to know when the database will be back up and you are sitting in the data center with a server in your lap replacing a mother board because your contacted repair man can not get there for a few hours.. I love this job... I rock 3 monitors at work and always have screens divided mentally by importance. Screen one is live / check it all twice before hitting enter type of data. Screen two is usually secondary, test or backup servers to pay attention to what was done so it can be duplicated. Screen three is email and web research… Or depending on the situation each screen is a different geographic location. Biggest Trip Ups I have to watch out for are the “Are you in the right directory? Are you on the right machine? Right Data? Right Dates?” type stuff

Side Note: The instructor to a CCNA class I was taking walked around during our practice tests with a cheep cell phone refrigerator magnet and simulated the “real world” by making it ring while we took the test. Damn thing stressed us out, but didn’t even come close to the owner calling you wanting to know when the database will be back up and you are sitting in the data center with a server in your lap replacing a mother board because your contacted repair man can not get there for a few hours..

I love this job…

]]>