Audiogalaxy had a small client that simply handled the P2P transfers and a complicated website for everything else, including account settings. One of the adjustable account settings on the website was the “max number of transfers.” To encourage users to send as much as they received, we only gave them a single number for this setting. With a value of 1, a Satellite would only send a single file at a time, but it could only download one file a time as well.

Things were not so simple on the back-end. For better or for worse, I had designed some flexibility into the system. The max transfers value was actually stored in two columns in the Users table – MaxSend and MaxRecv. The back-end – the part that actually looked at these values when it was setting up transfers–had no idea these columns were linked. The front-end enforced what went into the database, and the back-end obeyed it. Whenever the Satellite reconnected to the cloud, our server would read the value out of the database and store it in memory for the duration of that connection.

Of course, somewhere between the frontend and the backend is mysqlclient, but I’ll get to that in a moment.

Quake III Arena was my game of choice at the time I worked for AG. We had a few developers that also enjoyed the game, and it was common to find people staying late on the weekend to take advantage of our nice internet connection. Unfortunately, our nice internet connection had a dozen people running our p2p music sharing client on one side, so it would periodically slow down when someone’s computer started blasting a file out at high speed. These slowdowns drove us crazy, particularly when they prevented us from using the game’s rail gun effectively.

Good developers like to fix problems, and developers at startups also tend to have access to the database. So, you can probably imagine what a developer might do. And if you know a little bit about SQL, you can also imagine what might go horribly wrong. I never found out who issued the bad query, but I can just imagine how it played out:

Hey, I’ve got an idea about how we can keep the games from lagging tonight. I’ll just block everyone in the office from sending files. One simple ‘Update Users set MaxSend = 0′ and we should be good to go for the evening… Why is that query taking so long? Uh oh…

SQL is good for a lot of things, but I’ve always marveled at how easy it is to destroy an entire table simply by forgetting a where clause. And thus, in a few short minutes, every one of our 30 million users had a subtle change applied to their accounts. Did I mention that the single value we displayed on the website for this setting came from the MaxRecv column? Whoops…

Monitoring the health of the system was one of my jobs, so I kept a close eye on my graph of the “current transfer rate.” Ultimately, most problems in the system resulted in less files getting transferred, so the global transfer rate was a good proxy for the health of the system.

Every day of the week plotted a unique and predictable curve that I knew by heart, and so it didn’t take me long to realize that something was wrong. Transfer rates were dropping. But why? I called our ISP and asked if they knew of any problems with the Internet. Nope. We had exactly the right number of clients connected. No one had trenched over a fiber optic cable in the middle of nowhere. Requests were coming into the system at the normal rate; they just weren’t getting fulfilled. Microsoft hadn’t pushed any patches out that might have firewalled off half the world.

Clients generally stayed connected for days or weeks at a time. As they gradually reconnected, more and more of the network got their new MaxSend setting and dutifully started not sending anything. Users weren’t complaining – it was perfectly normal for rare songs to be inaccessible, and nobody noticed if his client just wasn’t sending anything.

After tearing my hair out for a day or so about this, I finally realized I was seeing a lot more “client busy – no free slots” type messages than I usually did while tail –f’ing the log files. Digging into that, I noticed some other funny messages, and eventually I was staring in shock at the results of a “select MaxSend, MaxRecv from Users limit 1000.”

Fixing the problem was easy enough: “Update Users set MaxSend = MaxRecv,” but you can imagine I spent quite some time staring at that query before issuing it.

So what’s the moral of the story? Don’t let your developers have access to the production database? Maybe, but that isn’t practical for a small startup. Better logging? That certainly could help. Force everyone to access the database using the –i-am-a-dummy flag for MySQL? That is not a bad idea and will get you some of the way there, but a shoddily written script can do exactly the same kind of damage. Backups? Sure, we had backups, but we were adding customers so quickly that restoring data more than a few hours old would have pissed off many thousands of people. An Admin class of users, with configurable policy that prevented them from sending files between 7pm and 3am on weekends? Yeah, right.

If you run a big and complicated system, problems you will never predict are going to happen and cause your system to do impossibly weird things you don’t expect. You must invest in tools to give you visibility into your system. My transfer rate graph was the only reason I was even able to go looking for a problem. I knew something was wrong, and it was just a matter of digging until I found it. Let your admins see into the system (specifically – how the system is behaving right now) so that they can develop intuition about what it should look like. Finding a bug in production is never fun. But it is going to happen, and it is always better if you find it before your users do.

2002
By 2002, things were going along quite nicely from a technology standpoint. Running the C backend on the hardware we had purchased for the Java version meant we were very much below our capacity, and we made it past 1 million simultaneous clients. I was pleased that I had learned the right lessons from V1 and designed a solid architecture.

I was also feeling good about the other major service I looked after — search. Audiogalaxy had sponsored my senior engineering project for UT in 2000 — a high performance MySQL indexing service. That, combined with a caching service I built in 2001, had solved the search problem. We were handling 80 million page views every day, most of them searches.

But having some of the scalability problems taken care of didn’t mean I was taking it easy. We were getting more and more DMCA takedown notices, and I’d been working on tools to identify all variations of matching songs for a long time. Michael went to Washington, DC to talk to a group of RIAA folks, and I ended up on a conference call. Their tech guy, Dan Farmer (yes, that Dan Farmer), grilled me on the details of the filtering system.

The filtering system was the most frustrating piece of tech I’ve ever worked on. Audiogalaxy was amazingly effective at distributing rare music, and so no matter how many variations on artist and song names we blocked, things kept slipping by. I tried every trick that I could think of and blocked huge numbers of songs, but it was like trying to hold back the tide.

Eventually, the RIAA decided we weren’t doing enough. On May 24th, a gruff and tattooed messenger walked into the office with The Envelope, and that was that. Michael called in the lawyers and spent weeks trying to find a solution. In the end, he decided that settling it as quickly as possible was the best strategy.

I shut Audiogalaxy down from my apartment on the afternoon of June 17, 2002. It was a beautiful, typical Austin summer day. I knew what was coming, so I had stayed home to enjoy the sunshine. Mid afternoon, my cell phone rang, and David gave me the word to shut it down.

I typed out the command, paused for just a moment, and hit Enter.

For the amazing amount of effort that we put into building the system, destroying it was anti-climatic. The script I ran pushed a new configuration setting to several hundred servers and returned seconds later. The transfers that were currently active were allowed to finish, and then everything stopped. It was a moment I won’t forget, and I regret that I didn’t do it with the rest of team in the same room.

Over the next few weeks, I tried to enjoy being free from the stress of maintaining such a big operation 24/7, but I couldn’t shake the feeling that at 24, my career had already peaked. I didn’t really appreciate the experience I had gained at the time, and I was mostly consumed with the thought that I would never again have so many users love my software. I had no idea what I would do next.

Most of the staff was laid off shortly after that. After our traffic plummeted, Geoff and I spent a week unracking hundreds of servers from the data center in north Austin and driving them down I35 to the office. The company partnered with Rhapsody to offer a music service, and we tried to think of other things we could do with Audiogalaxy. But eventually, we decided it was time to move on.

2000
I came back from my Christmas break feeling less burnt out. I focused on designing a backend that could handle 100,000 simultaneous Satellites and then started building it. To free me up from working on the client, Michael bought a copy of the Steven’s networking book and started working on a C version of the Satellite core. And David hired Kennon Ballou to help build the next-generation web interface.

The new backend and client went live in April, using my humble website from V1. Traffic started growing steadily, and by the end of May, we had about 3,000 clients connected at peak times. Sometime around the end of July, there was a Napster injunction scare, which pushed us over the 8,000 mark. We released version 0.6 of the client and David’s beautiful new website in September. At that point, our peak load started increasing by thousands of users every week.

In the way that only 22-year-old unattached guys can do, I revolved my life around work. The kitchen was full of free food, and if I didn’t eat there, I ate lunch and dinner with Michael or anyone else that happened to be around. I went to the gym in the evening with Michael and Geoff and then back to the office. When things got too intense, the distractions of 6th Street were a short walk away. I remember going back to the office to work on low-priority stuff while I sobered up before driving home. At my apartment, my furniture was limited to a twin mattress, a chair, and a folding table for my computer. Every week we had more users, and every time something in the cloud melted down and we had to scramble to fix it, our crew became a little bit more tightly knit.

This was the year that we figured out how to manage a huge website with a tiny team. Geoff handled the hardware, I handled any services I had written and did a little bit of MySQL work, and Michael did everything else: he built Linux kernels, handled Apache and the load balancers, and dealt with any tricky aspects of MySQL. We had a set of internal pages for monitoring the site, and we checked them as soon as we got out of bed, reflexively throughout the day, and right before going to sleep. We all had weird schedules so we managed to cover a good chunk of every day, and it was always fair game to wake someone else up at 4am to diagnose a problem. This wasn’t really designed or planned – it just happened because it needed to be done.

The excitement about how quickly we were scaling up was tempered in December. We had an amazing advertising contract for a flat $5 CPM. Eventually, they realized how much money they owed us and decided to just not pay us. Michael had to let a few reviewers go and borrow $75,000 to make payroll. We all got a lesson about how low the startup roller coaster can go.

2001
Thanks to the lessons we learned from V1, the hardware, rather than the software, became the bottleneck for V2 of the Satellite service. We quickly surpassed our goal of 100,000 simultaneous users and realized that target was missing at least one zero. We watched graphs of how many users were online, but instead of seeing beautiful diurnal curves, we saw harsh, flat caps as we bumped into capacity limits. Due to limitations of Java at the time, each server could only handle 2000 simultaneous users, and our users let us know this was a problem. Michael kept a fax some user had sent in with “Buy A New Server!” scrawled across it hanging up in his office.

We definitely tried. And we definitely bought a lot of servers. Every few weeks we would get a batch of parts delivered to the datacenter. Geoff would drive out and spend 24 highly caffeinated hours assembling and racking 20 new servers with a manual screwdriver that often left his hands bloody. But it seemed like we could never keep up. We filled out one row at the datacenter and started working on a second:

In the fall of 2001 I read about /dev/epoll and decided it was the key to a vastly more scalable backend written in C. We hired Steven Hazel, a talented and practical developer from the FreeNet project to help with the project. I knew we had the right guy when he asked how we were going to avoid Second System Syndrome during the port during our first meeting.

Despite the pain of converting a lot of the string processing we did from Java to C, testing the new version was pretty easy. We loaded the service down with assertions and rigid state machines, and ran a single instance in the cluster. The flood of traffic would initially drive the process into an assertion within seconds.

As we worked through the bugs, the uptime turned into minutes, then hours, and eventually the service virtually never crashed. With hundreds of instances deployed, we got so much traffic that we were able to remove all the bugs we were likely to run into. We had one or two machines that would crash every month or two with inscrutable core files. Because it was always the same machine, I eventually attributed this to faulty memory. The idea that you could write software that was more reliable than hardware was fascinating to me.

In fact, almost everything about the scale of the software fascinated me. I found that a system with hundreds of thousands of clients and thousands of events per seconds behaved like a physical machine built out of springs and weights. If one server process stalled for a moment, effects could ripple throughout the cluster. Sometimes, it seemed like there were physical oscillations – huge bursts of traffic would cause everything to slow down and back off, and then things would recover enough to trigger another burst of load. I had never even imagined that these sorts of problems existed in the software world and I found myself wishing I had taken control theory more seriously in college.

Keeping up with the traffic at this time was difficult, but in retrospect, it was really a lot of fun. I had graduated from UT in December of 2000 and moved downtown within walking distance of both 6th Street and the office. I spent the summer on a completely nocturnal cycle, partially because of the Texas heat, but mainly because restarting services was easier at 3 in the morning. I was tired of staying up late to deploy new code, so I just changed my schedule. Audiogalaxy users had led me to a set of live trance mixes from clubs in Europe which turned me into a diehard electronica fan, and driving around Texas to catch DJs on the weekend was much easier if staying up until 8am was normal. I bought some turntables and a lot of vinyl. And a couch. The light in my apartment when I got home in the morning was very lovely.

Audiogalaxy had gotten big enough at this point that it was not uncommon to overhear strangers talking about it, which was one of the best perks of working at the company. The community-based features of the site had also really taken off and eighty thousand messages were posted per day at our peak. A couple met on our message boards and got married shortly after that. A community of several hundred spinning instructors formed to share ideas about music selection during class. Several years later, a die-hard message board fan got the AG logo tattooed onto his leg:

Life was hectic but good. We were having so much success it was almost like something bad was about to happen.

Continued in Part 3

Getting the Job
My career at Audiogalaxy got started by chance one spring afternoon in 1999. At the time, I was a computer-engineering student with just a few more classes to finish before I graduated that fall. Like most of my friends, I was planning on taking an entry-level engineering job at one of the big companies in Austin, but things didn’t end up turning out that way.

I ran into Michael Merhej on the steps of the Electrical and Computer Engineering building and stopped to chat about the business I had heard he had started. Michael and I had been lab partners in an electrical engineering class a few semesters before, and he had struck me as a high energy guy with an amazing capacity to focus on the things he was interested in. We talked a little bit about Audiogalaxy, and he told me that they really needed some web developers. A music website sounded a lot more fun than working for the engineering career center like I had done the previous year, so I told him I was interested.

My “interview” took place at Michael’s low-budget apartment that he shared with three other guys. He had just finished shipping some new servers off to the datacenter, and a mix of computer component packaging and laundry were strewn about the apartment. David McArthur, the other co-founder of AG, asked me a few questions about HTML, and then they offered me a part time contracting job as a web developer.

At that time, the company had two areas of interest. The first was an FTP search engine that Michael had written the previous year. Back in the stone age of MP3 distribution, FTP sites were the preferred technology. FTP servers allowed admins to enforce upload/download ratios, and so running an FTP site was a fine way to increase the size of your own music collection. Sites had different ratios, were up during different times of day, and were just plain hard to find. This created demand for information about them, and AG filled it. Michael had written one of the leading FTP search engines the year before and was getting an impressive amount of traffic.

The second part of the company targeted unsigned artists trying to get exposure. Artists could upload their MP3s, and the staff would listen to the music, categorize it, and write a few blurbs. Then, the music went on the site for anyone to download. Originally, bands had to use FTP to upload their music, which was too difficult. One of my first projects was “Web-FTP,” which provided an HTTP based system for uploading content and allowed for integration with the tools the music staff used to handle the music.

Summer, 1999
I started working when finals ended. During the summer, I worked part-time because I also had a full-time internship at Tivoli. Tivoli, which had recently been acquired by IBM, had started a boot camp program to try to develop engineering talent. It was a fun program, but more importantly, it wasn’t a very high stress job. This left me with lots of time in the evenings and weekends to get my Audiogalaxy work done.

For my first month or two at AG, the company had no offices. I worked from home for the first few weeks, and then David gave everyone a key to his house and set up some computers in the back. Later in the summer, we moved into offices located on the less distracting part of 6th street. By the end of the summer, I was signed up for the startup life. I finished my internship at Tivoli, quit my campus job at UT, dropped 2 of the classes I was scheduled to take that fall, and started working for AG full time.

The salary was just OK, but that didn’t really matter because I had 55,000 stock options! Once we hit it big and had our IPO, I knew I would make a lot of money. That seems insane and naïve now, but with all the other crazy startups out there at the time, it made complete sense to 21-year-old Tom. Overall, it seemed like an interesting opportunity, and the job market was so good back then that I knew I would have other options if the company folded. Besides, working on a music website with friends in downtown Austin seemed like a pretty cool gig.

At this point, there were 4 other full time employees who stayed with the company until the very end:

• Michael Merhej – Founder/CEO
• David McArthur – Founder/handled the web site design and managed its development. He and Michael had met working together at a computer lab on campus.
• Eric Zappa – Managed all the music reviewers and had previously worked for a label.
• Geoff Midler –Did everything from tech support to handling servers. He had known Michael since high school.

After I started working full-time, I wrapped up Web-FTP and ported the FTP search to PHP (Michael had originally written it as a CGI app in C). Around that time, we were frustrated with how few people were running FTP servers, so we started kicking around ideas about trying to distribute our own. We wanted something that would take just a few clicks to install and would ping our servers so that we could index them. We had some notes for this sketched out on the whiteboard when Michael downloaded a little program called Napster.

Fall, 1999
Michael immediately knew that Napster was the future and that FTP sites were about to become a footnote. Since there weren’t any other programmers on staff at the time, I was given the task of coding a competing system. Without knowing it, I also signed up to be on call for data center problems 24/7 for the next few years. I hadn’t planned on a career that kept me on call, but the defining characteristic of startup life is doing whatever it takes to get the job done. Michael drove me to the Sprint store and I picked out my very first cell phone the next month.

Napster was impressive, but we wanted to do things a little bit differently. First, we wanted to keep users coming back to our website. This was back in the day of high CPM deals, so the more website traffic we could get, the better. Second, we wanted to avoid partitioning our network. One problem with Napster was that you could only search for files shared by people connected to the same server as your client, which greatly reduced the availability of the music.

The project got started around September 24th, 1999. I remember the date because that evening I drove out to Southpark Meadows with my friend John to catch a Phish concert. I spent the time in the car talking about some of my ideas for the service, and I started hacking on it that weekend.

It took a lot of 100-hour weeks, but by the end of the year we had shipped a flawed, early version of the Satellite. I had created a piece of software that only scaled to a few thousand users, but I had also learned a lot of lessons. More importantly, I had gotten a taste of how amazing it was to work on networked software that connected thousands of people and computers. With 2 classes left, I had also screwed up my GPA by getting a C in a silly linear algebra class, but I had become so obsessed with the Satellite that I didn’t mind. I had bigger problems at work.

The client, written in Java and using JNI to show a native WIN32 UI, was flaky. The server was massively bottlenecked on the database. It took hours to start a transfer for a requested file, and we would get bug reports that said things like, “My computer’s clock runs faster when I’m using the Satellite.” Sigh. You really do have to plan to throw one away.

Before I went home for Christmas that year, Michael and I sat down. We talked through what a service that didn’t use the database as much would look like, and we set a goal of building a service that could handle 100,000 simultaneous users. A cluster of servers that passed messages between each other to coordinate file transfers sounded like a good idea. I printed out a list of all the queries that the current service did and took them home to think.

Continued in Part 2