I’m sort of lazy, so I really like the idea of code that continually checks itself by using assertions. I even like running production services with assertions turned on. To be clear, I’m talking about assertions that check for actual bugs in your code – not assertions that socket() didn’t fail. Still, crashing production servers is a contentious issue, but sometimes (hopefully rarely) it is the best thing to do. For something like FolderShare, crashing a server as soon as there is any hint of an error is vastly safer than possibly deleting someone’s files due to a bug. Of course, this introduces the risk that you could have multiple servers fail in a short amount of time, but you need to design for that case anyway.
I don’t plan on having many non-technical posts here, but I’m breaking my rule today for a good reason. I’ve got a kid now! My first child, Margot Lee Kleinpeter, was born about 10 days ago. Between a long, drawn out labor, a few nights on a hospital couch, and fatherhood in general, I’ve fallen a bit behind on publishing. Much to my surprise, Margot prefers clean diapers and songs to essays on startups and programming. But, I’ve got a new post for today and I’ll hopefully be back on a more normal schedule soon. In the meantime, enjoy this picture of her sleeping:
When you are running a distributed service in a datacenter, you encounter a lot of interesting problems. At Audiogalaxy, I ran into all the standard application level bugs, crashes, and race conditions. Once we had a certain number of machines, we even had to deal with flaky memory, disks, and networking cards. But all of that was pretty typical compared to the weirdest bug I ever had to deal with – the one that was caused by Quake III Arena.
Next up in the toolbox series is an idea so good it deserves an entire article all to itself: consistent hashing.
Let’s say you’re a hot startup and your database is starting to slow down. You decide to cache some results so that you can render web pages more quickly. If you want your cache to use multiple servers (scale horizontally, in the biz), you’ll need some way of picking the right server for a particular key. If you only have 5 to 10 minutes allocated for this problem on your development schedule, you’ll end up using what is known as the naïve solution: put your N server IPs in an array and pick one using key % N.
I kid, I kid — I know you don’t have a development schedule. That’s OK. You’re a startup.
Continue reading ‘Programmer’s Toolbox Part 3: Consistent Hashing’
This is the final article in my three part series about building Audiogalaxy. If you haven’t already, you may want to read part one and part two first.
2002
By 2002, things were going along quite nicely from a technology standpoint. Running the C backend on the hardware we had purchased for the Java version meant we were very much below our capacity, and we made it past 1 million simultaneous clients. I was pleased that I had learned the right lessons from V1 and designed a solid architecture.
I was also feeling good about the other major service I looked after — search. Audiogalaxy had sponsored my senior engineering project for UT in 2000 — a high performance MySQL indexing service. That, combined with a caching service I built in 2001, had solved the search problem. We were handling 80 million page views every day, most of them searches.
This is the second article in my three part history about building Audiogalaxy.com. You should probably read the first one first.
2000
I came back from my Christmas break feeling less burnt out. I focused on designing a backend that could handle 100,000 simultaneous Satellites and then started building it. To free me up from working on the client, Michael bought a copy of the Steven’s networking book and started working on a C version of the Satellite core. And David hired Kennon Ballou to help build the next-generation web interface.
The new backend and client went live in April, using my humble website from V1. Traffic started growing steadily, and by the end of May, we had about 3,000 clients connected at peak times. Sometime around the end of July, there was a Napster injunction scare, which pushed us over the 8,000 mark. We released version 0.6 of the client and David’s beautiful new website in September. At that point, our peak load started increasing by thousands of users every week.
Reading all the SXSW press always makes me nostalgic for my time in Austin and the 3 years I spent working at Audiogalaxy.com, a music startup. Here is a history of the company from my perspective as the first full-time programmer. This is the first of three parts.
Getting the Job
My career at Audiogalaxy got started by chance one spring afternoon in 1999. At the time, I was a computer-engineering student with just a few more classes to finish before I graduated that fall. Like most of my friends, I was planning on taking an entry-level engineering job at one of the big companies in Austin, but things didn’t end up turning out that way.
I ran into Michael Merhej on the steps of the Electrical and Computer Engineering building and stopped to chat about the business I had heard he had started. Michael and I had been lab partners in an electrical engineering class a few semesters before, and he had struck me as a high energy guy with an amazing capacity to focus on the things he was interested in. We talked a little bit about Audiogalaxy, and he told me that they really needed some web developers. A music website sounded a lot more fun than working for the engineering career center like I had done the previous year, so I told him I was interested.
Following up on my first post, What should be in your programming toolbox, here are a few more ideas from my list:
Wikipedia has more details, but essentially, an unrolled linked list is great because it will give you better performance than a regular linked list, is more cache friendly, and will probably have much less overhead. The basic idea is to store an array of elements at each node rather than a single one. This keeps your pointers closer together, which will make your cache happy when you are iterating through your items.
Continue reading ’5 more essentials for your programming toolbox’
I went to a meeting run by the Seattle Tech Startup folks a few weeks ago. Even though I’m not thinking about doing another startup right now, I was glad to see the enthusiasm of all the other people who are. Because I love seeing the new ideas that come out of startups, I really hate seeing them fail as a result of them making the same silly mistakes. So, the collaboration that the STS meetings and the associated mailing list promote really put a smile on my face.
I’ve played a big part in two successful startups, and my two had very different flavors. Audiogalaxy was a rocket-ship ride that didn’t let up for three years until it all came crashing down due to a lawsuit. We had traffic from the minute we turned on the Satellite, and all we had to do was scale it up as quickly as we possibly could. FolderShare, on the other hand, sometimes felt like a continuous series of failures until we had success all at once–a glowing review from Walt Mossberg while we’re negotiating our acquisition by Microsoft. That was nice.
Now that I’ve got some free time, one of the goals for this blog is to reflect a little bit on my experiences and what I might change next time. Stay tuned.