The Google platform refers to the computer software and large hardware resources Google uses to provide their services. This article describes the technological infrastructure behind Google's websites as presented in the company's public announcements.
2 × 300 MHz dual Pentium II servers donated by Intel, they included 512 MB of RAM and 10 × 9 GB hard drives between the two. It was on these that the main search ran.
F50 IBM RS/6000 donated by IBM, included 4 processors, 512 MB of memory and 8 × 9 GB hard disk drives.
Two additional boxes included 3 × 9 GB hard drives and 6 x 4 GB hard disk drives respectively (the original storage for Backrub). These were attached to the Sun Ultra II.
IBM disk expansion box with another 8 × 9 GB hard disk drives donated by IBM.
Homemade disk box which contained 10 × 9 GB SCSI hard disk drives.
This article is outdated. Please update this article to reflect recent events or newly available information.(November 2010)
Google uses commodity-classx86server computers running customized versions of Linux. The goal is to purchase CPU generations that offer the best performance per dollar, not absolute performance. How this is measured is unclear, but it is likely to incorporate running costs of the entire server, and CPU power consumption could be a significant factor. Servers as of 2009–2010 consisted of custom-made open-top systems containing two processors (each with two cores), a considerable amount of RAM spread over 8 DIMM slots housing double-height DIMMs, and two SATA hard disk drives connected through a non-standard ATX-sized power supply unit. According to CNET and to a book by John Hennessy, each server had a novel 12-volt battery to reduce costs and improve power efficiency.
According to Google their global data center operation electrical power ranges between 500 and 681 megawatts. The combined processing power of these servers might have reached from 20 to 100 petaflops in 2008.
Details of the Google worldwide private networks are not publicly available but Google publications make references to the "Atlas Top 10" report that ranks Google as the third largest ISP behind Level 3.
In order to run such a large network with direct connections to as many ISP as possible at the lowest possible cost Google has a very open peering policy.
From this site we can see that the Google network can be accessed from 67 public exchange points and 69 different locations across the world. As of May 2012 Google had 882 Gbit/s of public connectivity (not counting private peering agreements that Google has with the largest ISPs). This public network is used to distribute content to Google users as well as to crawl the Internet to build its search indexes.
The private side of the network is a secret but recent disclosure from Google indicate that they use custom built high-radix switch-routers (with a capacity of 128 × 10 Gigabit Ethernet port) for the wide area network. Running no less than two routers per datacenter (for redundancy) we can conclude that the Google network scales in the terabit per second range (with two fully loaded routers the bi-sectional bandwidth amount to 1,280 Gbit/s).
From a datacenter view, the network starts at the rack level, where 19-inch racks are custom-made and contain 40 to 80 servers (20 to 40 1U servers on either side, while new servers are 2U rackmount systems. Each rack has a switch). Servers are connected via a 1 Gbit/s Ethernet link to the top of rack switch (TOR). TOR switches are then connected to a gigabit cluster switch using multiple gigabit or ten gigabit uplinks. The cluster switches themselves are interconnected and form the datacenter interconnect fabric (most likely using a dragonfly design rather than a classic butterfly or flattened butterfly layout).
From an operation standpoint, when a client computer attempts to connect to Google, several DNS servers resolve www.google.com into multiple IP addresses via Round Robin policy. Furthermore, this acts as the first level of load balancing and directs the client to different Google clusters. A Google cluster has thousands of servers and once the client has connected to the server additional load balancing is done to send the queries to the least loaded web server. This makes Google one of the largest and most complex content delivery networks.
In February 2009, Stora Enso announced that they had sold the Summa paper mill in Hamina, Finland to Google for 40 million Euros. Google plans to invest 200 million euros on the site to build a data center. Google chose this location due to the availability and proximity of renewable energy sources.
Modular container data centers
Since 2005, Google has been moving to a containerized modular data center. Google filed a patent application for this technology in 2003.
Most of the software stack that Google uses on their servers was developed in-house. According to a well known Google employee, C++, Java, Python and (more recently) Go are favored over other programming languages. For example, the back end of Gmail is written in Java and the back end of Google Search is written in C++. Google has acknowledged that Python has played an important role from the beginning, and that it continues to do so as the system grows and evolves.
The software that runs the Google infrastructure includes:
Google Web Server (GWS) – custom Linux-based Web server that Google uses for its online services.
TeraGoogle – Google's large search index (launched in early 2006), designed by Anna Patterson of Cuil fame.
Caffeine (Percolator) – continuous indexing system (launched in 2010).
Hummingbird – major search index update, including complex search and voice search.
Google has developed several abstractions which it uses for storing most of its data:
Protocol Buffers – "Google's lingua franca for data", a binary serialization format which is widely used within the company.
SSTable (Sorted Strings Table) – a persistent, ordered, immutable map from keys to values, where both keys and values are arbitrary byte strings. It is also used as one of the building blocks of BigTable.
RecordIO – a sequence of variable sized records.
Software development practices
Most operations are read-only. When an update is required, queries are redirected to other servers, so as to simplify consistency issues. Queries are divided into sub-queries, where those sub-queries may be sent to different ducts in parallel, thus reducing the latency time.
To lessen the effects of unavoidable hardware failure, software is designed to be fault tolerant. Thus, when a system goes down, data is still available on other servers, which increases reliability.
Like most search engines, Google indexes documents by building a data structure known as inverted index. Such an index allows obtaining a list of documents by a query word. The index is very large due to the number of documents stored in the servers.
The index is partitioned by document IDs into many pieces called shards. Each shard is replicated onto multiple servers. Initially, the index was being served from hard disk drives, as is done in traditional information retrieval (IR) systems. Google dealt with the increasing query volume by increasing number of replicas of each shard and thus increasing number of servers. Soon they found that they had enough servers to keep a copy of the whole index in main memory (although with low replication or no replication at all), and in early 2001 Google switched to an in-memory index system. This switch "radically changed many design parameters" of their search system, and allowed for a significant increase in throughput and a large decrease in latency of queries.
In June 2010, Google rolled out a next-generation indexing and serving system called "Caffeine" which can continuously crawl and update the search index. Previously, Google updated its search index in batches using a series of MapReduce jobs. The index was separated into several layers, some of which were updated faster than the others, and the main layer wouldn't be updated for as long as two weeks. With Caffeine the entire index is updated incrementally on a continuous basis. Later Google revealed a distributed data processing system called "Percolator" which is said to be the basis of Caffeine indexing system.
Google's server infrastructure is divided into several types, each assigned to a different purpose:
Web servers coordinate the execution of queries sent by users, then format the result into an HTML page. The execution consists of sending queries to index servers, merging the results, computing their rank, retrieving a summary for each hit (using the document server), asking for suggestions from the spelling servers, and finally getting a list of advertisements from the ad server.
Data-gathering servers are permanently dedicated to spidering the Web. Google's web crawler is known as GoogleBot. They update the index and document databases and apply Google's algorithms to assign ranks to pages.
Each index server contains a set of index shards. They return a list of document IDs ("docid"), such that documents corresponding to a certain docid contain the query word. These servers need less disk space, but suffer the greatest CPU workload.
Document servers store documents. Each document is stored on dozens of document servers. When performing a search, a document server returns a summary for the document based on query words. They can also fetch the complete document when asked. These servers need more disk space.
Ad servers manage advertisements offered by services like AdWords and AdSense.
Spelling servers make suggestions about the spelling of queries.
^Tawfik Jelassi and Albrecht Enders (2004). "Case study 16 — Google". Strategies for E-business. Pearson Education. p. 424. ISBN978-0-273-68840-2.
^ abComputer Architecture, Fifth Edition: A Quantitative Approach, ISBN 978-0123838728; Chapter Six; 6.7 "A Google Warehouse-Scale Computer" page 471 "Designing motherboards that only need a single 12-volt supply so that the UPS function could be supplied by standard batteries associated with each server"