When you are running a distributed service in a datacenter, you encounter a lot of interesting problems. At Audiogalaxy, I ran into all the standard application level bugs, crashes, and race conditions. Once we had a certain number of machines, we even had to deal with flaky memory, disks, and networking cards. But all of that was pretty typical compared to the weirdest bug I ever had to deal with – the one that was caused by Quake III Arena.
This is the final article in my three part series about building Audiogalaxy. If you haven’t already, you may want to read part one and part two first.
2002
By 2002, things were going along quite nicely from a technology standpoint. Running the C backend on the hardware we had purchased for the Java version meant we were very much below our capacity, and we made it past 1 million simultaneous clients. I was pleased that I had learned the right lessons from V1 and designed a solid architecture.
I was also feeling good about the other major service I looked after — search. Audiogalaxy had sponsored my senior engineering project for UT in 2000 — a high performance MySQL indexing service. That, combined with a caching service I built in 2001, had solved the search problem. We were handling 80 million page views every day, most of them searches.
This is the second article in my three part history about building Audiogalaxy.com. You should probably read the first one first.
2000
I came back from my Christmas break feeling less burnt out. I focused on designing a backend that could handle 100,000 simultaneous Satellites and then started building it. To free me up from working on the client, Michael bought a copy of the Steven’s networking book and started working on a C version of the Satellite core. And David hired Kennon Ballou to help build the next-generation web interface.
The new backend and client went live in April, using my humble website from V1. Traffic started growing steadily, and by the end of May, we had about 3,000 clients connected at peak times. Sometime around the end of July, there was a Napster injunction scare, which pushed us over the 8,000 mark. We released version 0.6 of the client and David’s beautiful new website in September. At that point, our peak load started increasing by thousands of users every week.
Reading all the SXSW press always makes me nostalgic for my time in Austin and the 3 years I spent working at Audiogalaxy.com, a music startup. Here is a history of the company from my perspective as the first full-time programmer. This is the first of three parts.
Getting the Job
My career at Audiogalaxy got started by chance one spring afternoon in 1999. At the time, I was a computer-engineering student with just a few more classes to finish before I graduated that fall. Like most of my friends, I was planning on taking an entry-level engineering job at one of the big companies in Austin, but things didn’t end up turning out that way.
I ran into Michael Merhej on the steps of the Electrical and Computer Engineering building and stopped to chat about the business I had heard he had started. Michael and I had been lab partners in an electrical engineering class a few semesters before, and he had struck me as a high energy guy with an amazing capacity to focus on the things he was interested in. We talked a little bit about Audiogalaxy, and he told me that they really needed some web developers. A music website sounded a lot more fun than working for the engineering career center like I had done the previous year, so I told him I was interested.
I went to a meeting run by the Seattle Tech Startup folks a few weeks ago. Even though I’m not thinking about doing another startup right now, I was glad to see the enthusiasm of all the other people who are. Because I love seeing the new ideas that come out of startups, I really hate seeing them fail as a result of them making the same silly mistakes. So, the collaboration that the STS meetings and the associated mailing list promote really put a smile on my face.
I’ve played a big part in two successful startups, and my two had very different flavors. Audiogalaxy was a rocket-ship ride that didn’t let up for three years until it all came crashing down due to a lawsuit. We had traffic from the minute we turned on the Satellite, and all we had to do was scale it up as quickly as we possibly could. FolderShare, on the other hand, sometimes felt like a continuous series of failures until we had success all at once–a glowing review from Walt Mossberg while we’re negotiating our acquisition by Microsoft. That was nice.
Now that I’ve got some free time, one of the goals for this blog is to reflect a little bit on my experiences and what I might change next time. Stay tuned.