Is this because:

• Nobody needs anything other than Google Website Optimizer?
• Startups don’t actually do AB testing, possibly because they don’t get enough traffic to get meaningful results, or maybe because they don’t have time?
• AB testing (including the statistical analysis to determine if results are valid) is so simple that everyone just bangs out their own?
• As a largely theoretical issue for most startups, scalability is more fun to talk about on the Internet?
• Everyone that is using AB testing is so happy that they are trying to suppress information about it so their competitors don’t start doing it too?

If everyone is secretly using some great framework please shoot me an email and let me know.

If you haven’t thought much about it before, here is a short paper on AB testing from some folks that made Amazon a ton of money.

• A clever proposal from Google: Shared Dictionary Compression over HTTP
• Cacheset – a tool for clearing the windows disk cache (useful for testing cold starts).
• Fun fact: the Tesla Roadster carries 3 milligrams of electrons when fully charged.
• The ultimate Airplane on a Treadmill debate resource: www.airplaneonatreadmill.com
• A 728-ton tuned mass damper in a skyscraper

But, I don’t think I’ve reached the critical mass of followers necessary to really unlock the Q&A potential of the site. Having a few hundred technical folks all following each other would be a tremendously useful resource for everyone involved. For example, I’m considering upgrading my desktop to 8 or 16GB of RAM. I’m going to need a new motherboard, processor, and RAM. My normal approach for this would be to spend a few hours on Newegg and the hardware review sites trying to figure out where the price/performance curve is and making sure I’m not getting ripped off. If someone else has done this same research it would be nice to use their information as a starting point, and twitter provides the kind of free-form conversation necessary for that kind of sharing.

To really make this work, you need to run one of the desktop apps so you don’t have to constantly reload the website (I use Twhirl).

Design the system to never need human interaction, but understand that rare events will occur where combined failures or unanticipated failures require human interaction.

Yes, designing the system to never need human interaction is a great ideal to shoot for, but when you are working for a startup with three guys and a dozen servers, you don’t have the resources or the justification to do it from Day 1. It is entirely likely that your business model will fail before you lose a single disk. And since backend refinements don’t pay the bills at a small scale, something with a pair of hands is going to be interacting with your system until you get enough people and servers to justify more automation.

These events will happen and operator error under these circumstances is a common source of catastrophic data loss.

That is a wonderfully simple and accurate summary of How Bad Things Happen in your datacenter. It starts when you lose a hard drive or MySQL crashes, and you have to promote your slave until you can check the master tables, or anything painful but routine happens. But then, as you are trying to fix things, you notice, for example, that you are almost out of disk space. When you start trying to fix more than one problem under pressure, you are entering a world of pain.

The big issue here as James points out is that you are going to do something wrong. You’ll probably use a much stronger word than “wrong” once it is all over, but let’s settle on “stupid” for right now.

It won’t feel stupid until after you hit “enter,” but when you are making unfamiliar decisions quickly under pressure, you are extremely likely to overlook something. Maybe you won’t shut down mysql before you start a myisamchk from the shell, or maybe you’ll reverse the arguments to “tar -cvzf” and wipe out something important. Or perhaps you’ll screw up a firewall rule and block ssh access to the machine you are frantically trying to fix. Accidently killing the ssh daemon is another favorite. The point is that during a stressful situation in the datacenter, the human operator is the biggest potential source of more downtime or “catastrophic data loss.”

Assuming you can’t automate everything, what can you do? Well, the absolute best thing you can do is practice. Corrupt some data on your dev master db, and see how long it takes you to get it restored from backups or a slave copy. Practice what would happen if you lost a database slave and had to activate a spare machine to take its place (I hope you have at least one spare machine). But of course, no one at a small startup has time to practice. Maybe once you hire a full time ops guy, it would be good to make sure he is practicing this sort of thing occasionally. But when practicing is going to take away from writing code, you aren’t going to practice.

Since you aren’t going to practice, what else can you do? The next best thing is to cultivate the attitude that you are the most likely source of problems. Don’t worry about hard drives, worry about bad decisions. Develop some humility about how you expect to behave when you get woken up at 4am to fix a database the morning of your launch or when a switch fails an hour before your big demo. From that mindset, here are a few things to do:

Script what you can
Off the top of my head, a good place to start would be writing scripts for some of the steps in setting up master/slave replication and manipulating firewall traffic (allowing or blocking external traffic, for instance).

Use the buddy system
It is not a bad idea to have somebody else there looking at what you are typing, or at least on the phone confirming things verbally.

Take your fingers off the keyboard before you hit enter
Are you in the right directory? Are you on the right machine? Are those arguments in the right order? Can you just rename this old stuff instead of deleting it? All of these are excellent questions to ask yourself or your coworker while you have your hands in your lap. This is also a good idea when you are doing something scary with SQL, like running any query that doesn’t have a where clause.

Slow things down
As soon as you make one mistake, no matter how minor, it is time to slow things down. Beyond the fact that making a mistake will fluster you, making one mistake demonstrates that right now, you are likely to make mistakes. That is a huge red flag. At this point, the safest thing may be to accept a slightly longer downtime just so you can slow things down, get some water, and relax. Trying to compensate for a little mistake by doing things faster can result in a much, much worse mistake. Unless you’ve just rolled a server cage down the stairs, there is always a worse mistake you can make.

Make it hard for people you work with to make mistakes
A quality server naming scheme is the easiest thing you can do here. No colors, deities, countries, snack foods, snakes, etc. I like $machineType-$number myself, but with distinct number ranges, even between different machine types. So, don’t have SQL-001 and Web-001. One day, some very sleepy datacenter employee may get things mixed up when you call and ask him to reboot Web-001. I’m sure you’ll get an apology, but you won’t get your uptime back. So make it harder for him to screw up: if your web machines start at Web-201, he’ll have to make 2 mistakes before he accidently reboots your primary database.

Talk about this stuff ahead of time
You probably have plenty of stuff to talk about at lunch with your coworkers, but here are a few convers ation starters if you want to sharpen your disaster recovery skills:

• “What happens if we lose power to one of our racks?”
• “How many of our switches could we lose and get the site back up?”
• “What is the smallest amount of hardware we could lose that would knock us 100% offline?”

This stuff isn’t theoretical. I woke up at 2am one weekend during FolderShare with a ton of text messages from our cluster. The kindly folks at the datacenter had been doing power supply maintenance. At some point, they powered down 2 of our racks. Then, they powered them right back up. It wasn’t tough to fix, but it was so unexpected that it took me a few minutes to even realize what had happened.

Use tricks to deal with the general class of “running a command on the wrong machine” problem.
Typing the right command on the wrong machine is obviously something to avoid. But when you have a sea of ssh windows open, what can you do?

• Use a different color background for your terminal to machines hosting master databases versus slaves
• Make sure the machine name shows up in the command prompt

Does anyone else have any good ideas or horror stories to tell? Post a comment and share your wisdom and/or pain.

I originally fell in love with assertions after reading Steve Maguire’s Writing Solid Code many years ago. After I saw how helpful they could be, I started to structure my code to make it more “assertable.” For example, I like state machines that use a table of valid transitions combined with assertions. This prevents anything I don’t explicitly anticipate from happening and is really helpful in a networked, asynchronous world.

But after developing a few long running services, I’ve started to perceive the need for a new type of assertion that is a bit higher level than a single conditional. I want something that will let me crash a service (and save a memory dump) when something “feels” wrong. For the moment, I call these “probabilistic assertions,” and I would have slept better while running FolderShare if I’d implemented them then.

Like all synchronization software, FolderShare had a few nightmare scenarios that I worried about all the time. If, due to a bug, the service told every client that its files were all deleted, I’d probably have to read a lot of nasty blog posts, and I would feel like crap for a few years. I would much rather crash all my servers and debug the problem with the system offline than take a chance on pissing off so many people.

Anyway, a normal assertion that looks at a single conditional wouldn’t help in this case. Asserting that no files have been deleted when a client connects doesn’t make any sense. But for a large enough sample size, I could assert that 80% of clients that connect shouldn’t have any files deleted. And I could probably assert that 95% of clients that connect shouldn’t have all their files deleted.

Functions like these cover most of what I’m thinking about:

Depending on your application, these functions could be implemented either as assertions or as triggers to alert an administrator. For alerts, you could really take this to another level by checking that incoming events are following a certain distribution for example. Granted that it’s a bit over the top, but I can’t help but imagine checking if a stream of incoming events obeys a Poisson distribution or if the sizes of certain data are following a Gaussian distribution.

Has anyone seen anything like this in the wild? I’d love to hear about it.

I’ve played a big part in two successful startups, and my two had very different flavors. Audiogalaxy was a rocket-ship ride that didn’t let up for three years until it all came crashing down due to a lawsuit. We had traffic from the minute we turned on the Satellite, and all we had to do was scale it up as quickly as we possibly could. FolderShare, on the other hand, sometimes felt like a continuous series of failures until we had success all at once–a glowing review from Walt Mossberg while we’re negotiating our acquisition by Microsoft. That was nice.

Now that I’ve got some free time, one of the goals for this blog is to reflect a little bit on my experiences and what I might change next time. Stay tuned.