Let’s say you’re a hot startup and your database is starting to slow down. You decide to cache some results so that you can render web pages more quickly. If you want your cache to use multiple servers (scale horizontally, in the biz), you’ll need some way of picking the right server for a particular key. If you only have 5 to 10 minutes allocated for this problem on your development schedule, you’ll end up using what is known as the naïve solution: put your N server IPs in an array and pick one using key % N.

I kid, I kid — I know you don’t have a development schedule. That’s OK. You’re a startup.

Anyway, this ultra simple solution has some nice characteristics and may be the right thing to do. But your first major problem with it is that as soon as you add a server and change N, most of your cache will become invalid. Your databases will wail and gnash their teeth as practically everything has to be pulled out of the DB and stuck back into the cache. If you’ve got a popular site, what this really means is that someone is going to have to wait until 3am to add servers because that is the only time you can handle having a busted cache. Poor Asia and Europe — always getting screwed by late night server administration.

You’ll have a second problem if your cache is read-through or you have some sort of processing occurring alongside your cached data. What happens if one of your cache servers fails? Do you just fail the requests that should have used that server? Do you dynamically change N? In either case, I recommend you save the angriest twitters about your site being down. One day you’ll look back and laugh. One day.

As I said, though, that might be OK. You may be trying to crank this whole project out over the weekend and simply not have time for a better solution. That is how I wrote the caching layer for Audiogalaxy searches, and that turned out OK. The caching part, at least. But if had known about it at the time, I would have started with a simple version of consistent hashing. It isn’t that much more complicated to implement and it gives you a lot of flexibility down the road.

The technical aspects of consistent hashing have been well explained in other places, and you’re crazy and negligent if you use this as your only reference. But, I’ll try to do my best. Consistent hashing is a technique that lets you smoothly handle these problems:

• Given a resource key and a list of servers, how do you find a primary, second, tertiary (and on down the line) server for the resource?
• If you have different size servers, how do you assign each of them an amount of work that corresponds to their capacity?
• How do you smoothly add capacity to the system without downtime? Specifically, this means solving two problems:
  • How do you avoid dumping 1/N of the total load on a new server as soon as you turn it on?
  • How do you avoid rehashing more existing keys than necessary?

In a nutshell, here is how it works. Imagine a 64-bit space. For bonus points, visualize it as a ring, or a clock face. Sure, this will make it more complicated when you try to explain it to your boss, but bear with me:

That part isn’t very complicated.

Now imagine hashing resources into points on the circle. They could be URLs, GUIDs, integer IDs, or any arbitrary sequence of bytes. Just run them through MD5 or SHA and shave off everything but 8 bytes (and if anyone tells you that you shouldn’t use MD5 for this because it isn’t secure, just nod and back away slowly. You have identified someone not worth arguing with). Now, take those freshly minted 64-bit numbers and stick them onto the circle:

Finally, imagine your servers. Imagine that you take your first server and create a string by appending the number 1 to its IP. Let’s call that string IP1-1. Next, imagine you have a second server that has twice as much memory as server 1. Start with server #2’s IP, and create 2 strings from it by appending 1 for the first one and 2 for the second one. Call those strings IP2-1 and IP2-2. Finally, imagine you have a third server that is exactly the same as your first server, and create the string IP3-1. Now, take all those strings, hash them into 64-bit numbers, and stick them on the circle with your resources:

Can you see where this is headed? You have just solved the problem of which server to use for resource A. You start where resource A is and head clockwise on the ring until you hit a server. If that server is down, you go to the next one, and so on and so forth. In practice, you’ll want to use more than 1 or 2 points for each server, but I’ll leave those details as an exercise for you, dear reader.

Now, allow me to use bullet points to explain how cool this is:

• Assuming you’ve used a lot more than 1 point per server, when one server goes down, every other server will get a share of the new load. In the case above, imagine what happens when server #2 goes down. Resource A shifts to server #1, and resource B shifts to server #3 (Note that this won’t help if all of your servers are already at 100% capacity. Call your VC and ask for more funding).
• You can tune the amount of load you send to each server based on that server’s capacity. Imagine this spatially – more points for a server means it covers more of the ring and is more likely to get more resources assigned to it.

  You could have a process try to tune this load dynamically, but be aware that you’ll be stepping close to problems that control theory was built to solve. Control theory is more complicated than consistent hashing.

• If you store your server list in a database (2 columns: IP address and number of points), you can bring servers online slowly by gradually increasing the number of points they use. This is particularly important for services that are disk bound and need time for the kernel to fill up its caches. This is one way to deal with the datacenter variant of the Thundering Herd Problem.

  Here I go again with the control theory — you could do this automatically. But adding capacity usually happens so rarely that just having somebody sitting there watching top and running SQL updates is probably fine. Of course, EC2 changes everything, so maybe you’ll be hitting the books after all.

• If you are really clever, when everything is running smoothly you can go ahead and pay the cost of storing items on both their primary and secondary cache servers. That way, when one server goes down, you’ve probably got a backup cache ready to go.

Pretty cool, eh?

I want to hammer on point #4 for a minute. If you are building a big system, you really need to consider what happens when machines fail. If the answer is “we crush the databases,” congratulations: you will get to observe a cascading failure. I love this stuff, so hearing about cascading failures makes me smile. But it won’t have the same effect on your users.

Finally, you may not know this, but you use consistent hashing every time you put something in your cart at Amazon.com. Their massively scalable data store, Dynamo, uses this technique. Or if you use Last.fm, you’ve used a great combination: consistent hashing + memcached. They were kind enough to release their changes, so if you are using memcached, you can just use their code without dealing with these messy details. But keep in mind that there are more applications to this idea than just simple caching. Consistent hashing is a powerful idea for anyone building services that have to scale across a group of computers.

A few more links:

• Another blog post about consistent hashing
• The original paper

Unrolled Linked Lists

Wikipedia has more details, but essentially, an unrolled linked list is great because it will give you better performance than a regular linked list, is more cache friendly, and will probably have much less overhead. The basic idea is to store an array of elements at each node rather than a single one. This keeps your pointers closer together, which will make your cache happy when you are iterating through your items.

To compare the overhead for an unrolled list, think for a second about a regular linked list node. It probably looks something like this:

Assuming you’ve got 4 byte pointers, each node is going to take 8 bytes. But the allocation overhead for the node could be anywhere between 8 and 16 bytes. Let’s go with the best case and assume it will be 8 bytes. So, if you want to store 1K items in this list, you are going to have 16KB of overhead.

Now, let’s think about an unrolled linked list node. It will look something like this:

Therefore, allocating a single node (12 bytes + 8 bytes of overhead) with an array of 100 elements (400 bytes + 8 bytes of overhead) will now cost 428 bytes, or 4.28 bytes per element. Thinking about our 1K items from above, it would take about 4.2KB of overhead, ,which is close to 4x better than our original list. Even if the list becomes severely fragmented and the item arrays are only 1/2 full on average, this is still an improvement. Also, note that you can tune the array size to whatever gets you the best overhead for your application.

Van Emde Boas Trees

I’ve used unrolled link lists before, but I’ve never used this slightly more exotic structure. I’m just very fond of low overhead data structures that are built for storing large amounts of information, and this is a clever one. VEB trees have good overhead characteristics, but they are also particularly fast — they implement all operations in O(log m), where m is the key length. So, if you use an optimal key length for your number of items, you can do everything in O(log log n), which is obviously an improvement over your standard O(log n) binary tree. The Wikipedia page has more details, but sadly, I haven’t been able to find any sort of library implementation.

Soft Deletes

If you want to be extra cautious about removing items from a database or you just want to support undelete, consider using Soft Deletes for your databases. Just add a “TimeDeleted” column and consider the row deleted when the time is non-zero. You can use a clean up script to actually remove the item after some amount of time has passed.

This technique isn’t without some hassles, though — you may have to add some extra logic or include the TimeDeleted column as part of a unique index if you want to allow the user to insert a duplicate item back into the table.

Skip Lists

I’m not sure if I would ever actually use skip lists because it seems like a cache unfriendly data structure. However, I still like the idea. In a nutshell, you start with a normal, sorted linked list. Then, on some nodes you add pointers that let you skip forward more than one node at a time. With a proper distribution, this allows for much better than O(N) when searching, inserting, and deleting items.

Berkeley DB

If you need to store some structured information on a disk, you need to have a really good reason to implement your own. Using Berkeley DB gets you a simple, well-known and well-debugged disk-backed store without a lot of overhead. Plus, you get things like hot backups and locking.

Note that Amazon is using Berkeley DB as one of the underlying stores for Dynamo. If you haven’t read the Dynamo paper (or for that matter, the BigTable and GFS papers), you really should. I didn’t mention it before, but BigTable uses one of the ideas from my last post, Bloom Filters.

Something I’ve found very helpful at this phase is a “programming toolbox” — a simple list of good ideas and approaches to different problems. When I’m stuck on a problem, or trying to generate a new approach to something, it can be helpful to flip through the list. Most of the ideas won’t apply, but sometimes it will spark something novel. To keep this list from getting too unwieldy, I’ll post a few at a time as I write them up.

Bloom Filters
Let’s start with one of my favorite data structures. Wikipedia has all the details, but here are the key points you need to know. Bloom filters are a “space-efficient probabilistic data structure” that:

• Will answer either “maybe” or “no” to the question, “Have you seen item X?”
• Don’t store the actual item
• Are incredibly space efficient — it takes about 10 bits per stored item to have a 1% error rate
• Are tunable — the error rate drops 10x for every extra 5 bits you use

Just as an example, let’s imagine you want to want to cache copies of web pages on your disk. Before taking the hit to access the disk to check if something is cached, you could use a Bloom Filter to keep track of what you have. Storing 100K URLs in a bloom filter would use just 125KB of memory. By contrast, if you stored UTF-8 versions of them in a hash table, assuming URLs are about 50 characters long, you would need about 4.7MB of memory, not even counting hash table overhead. What is the downside? About 1% of the time, you will get an incorrect “Maybe” from the Bloom Filter and would go to disk looking for something that does not exist. If 1% is too much, you can double the memory usage to 250KB and the error rate drops to 0.01%. Note that if the filter tells you “No”, there is a 0% chance that the URL is in the system.

Generic Headers
Are you defining any sort of message passing layer in your system? One thing you might want to consider is allowing unspecified header/value pairs to be added to the messages at any point later. Anything reading a message should simply ignore any headers it doesn’t recognize. This flexibility is baked into HTTP and has allowed all sorts of wonderful extensions beyond the basic HTTP spec. Adding this flexibility to your message layer from the beginning is trivial and it might let you solve a difficult problem later without completely upgrading all of your services.

MS Manners
Officially “Progress-based regulation of low-importance processes”, but known as MS Manners in this paper out of MS Research. The basic idea is that a background process can easily detect its impact on the user (or a more important process) if a few conditions are met. For processes with some degree of contention, the background process can establish a baseline of how long work should take. If it detects that recent work is taking longer, it can infer that it is impacting the more important process and wait until later to proceed. It sounds pretty simple, but the researchers have done the hard work in the paper and shown how to squeeze out the most gains. I think this is a powerful idea worth incorporating into any sort of maintenance job that runs against a production database.