Last year, Facebook, with 150 million active users, passed MySpace to become the leading social networking site in terms of unique worldwide visitors, according to comScore. Today, the site is ranked fifth in worldwide traffic per Alexa.
Each month, Facebook users upload more than 800 million photos (as of last summer, Facebook was the number one photo site on the Web, with 6.5 billion images) and more than 5 million videos to the site, as well as other content (some 20 million pieces of it, from Web links to news stories, blog posts, status reports and comments), totaling several petabytes of data, all of which need to be safely stored — yet retrievable at a moment’s notice. It is an extremely daunting task.
Indeed, as ubiquitous as the site has become, it is hard to believe that Facebook, the brainchild of former Harvard sophomore Mark Zuckerberg, is barely five years old — and could have amassed such a large following and amount of data in such a relatively short period of time. But that is indeed the case.
In fact, Facebook’s growth has been so quick and spectacular that in December 2007, the company’s California data center reached full capacity. Fortunately, Facebook’s storage team saw it coming and was already in the process of establishing an East Coast data center in Virginia, which went online just a few weeks earlier. Still, managing petabytes of active data is no easy task, and one the Facebook storage team is constantly looking at.
“In terms of capacity, I have breakfast with a number of charts every morning to see how we’re doing,” said Anoop Nagwani, Facebook’s manager of storage engineering, who’s been with the company for a year and a half. “We have monthly capacity reviews across the organization. And then my team and I have a weekly review as well.”
MySQL Writ Large
Facebook user data is stored in MySQL (making its MySQL installation, with over 1,800 MySQL servers, one of the largest in the world), which it chose for its openness so the code could be modified as necessary. Facebook also uses a large number of memcached servers, which are faster than the MySQL servers and thus better able to keep up with requests, to store copies of the data. Additionally, the data is replicated from Facebook’s West Coast data center to the East Coast data center, and is subjected to a data dump each night, with the user data sent to de-duplication specialist Data Domain (NASDAQ: DDUP) for long-term storage.
Facebook handles photos and videos differently. For photos, Facebook uses NFS and NetApp (NASDAQ: NTAP) SATA drives, because of NetApp’s ability to store a large amount of objects within a single container. Increasingly, however, as the number of photos stored on Facebook has grown, the NetApp system has been overwhelmed with metadata, causing a bandwidth bottleneck.
So later this year, Facebook plans to roll out its own objectstore called “Haystack,” which the company describes as “a user level abstraction where lots of data is stored in a single file” (similar to a LUN).
Facebook also uses NFS for videos, “because we have many machines that need to access the same data within the tier, and that was the most cost-effective and high-performance way to do it,” said Nagwani, as well as Isilon’s (NASDAQ: ISLN) clustered storage system. But Nagwani and his team are actively looking into other platforms, like the Sun (NASDAQ: JAVA) Storage 7410 Unified Storage System, which can scale up to 576 TB.
Power and Performance
Another big concern of Nagwani and his team is power consumption.
“We’re always concerned about power,” he said. “And we’re concerned about cabinet positions within our data centers. That said, when we devise a solution, for example, like when we moved to NetApp, and specifically when we moved to running photos on NetApp SATA drives, we looked at the power consumption. We looked at the amount of I/O the system could do, the capacity it provided and other features that it had. And we also looked at creating, basically, a cookie cutter design. So we knew exactly how much capacity was available from that system and how many users that could support.”
As with its MySQL data, Facebook replicates photos and videos from its West Coast data center to its East Coast one, using snapshots. That way, “if there’s any sort of application or administrative error, we can always rollback the snapshot, or, in the worst case, fail over to the East Coast and do a data recovery to repopulate the West Coast from there,” said Nagwani.
Another big issue for the Facebook team is scalability, of course. With approximately 25 TB of data being uploaded by Facebook users each week, “that is our number one requirement,” he said, “to scale ahead of demand, and to scale horizontally.”
— Anoop Nagwani
As part of its long-term strategy to always store its data on the best possible platform available, Facebook is currently looking into flash, specifically PCI-based high-capacity flash devices, joining the ranks of users moving toward solid state storage (SSD).
“We’re looking at dropping high-capacity flash devices into our MySQL databases to give us the I/O we need to do very fast database reads, to do very fast ETL processing, data warehousing, and get that data out of there as soon as it’s available,” said Nagwani. “We’re also looking at the Sun ZFS platform as a system we can use hand-in-hand with the NetApps. We like that because it’s got a ton of flash, so we can cache a lot of data. So basically flash is going to be huge for us this year.”