Urs Hölzle(Google) Talk at CERN

Anon-068c403e2d · Jul 19, 2005

Petabyte processing made easy
7-Jul-2005 CERN Computing Colloquium

The video

Agenda

Having to process petabytes of data sounds hard, but it's easier than you may think, and more affordable too. In this talk I'll describe how we deal with large amounts of data at Google. It starts with compute clusters that are optimized for throughput rather than peak performance, and a software layer that turns these clusters of relatively unreliable machines into a reliable computing platform. To simplify storage management, the Google File System (GFS) organizes a sea of local IDE disks into a convenient and reliable file system optimized for very large files. To simplify programming, the MapReduce framework allows programmers to focus on just the transformations they want to accomplish, freeing them from worrying about parallelization, scalability, and machine failures. And finally, to allow for effective sharing of a large clusters, a workqueue management system schedules the combined workloads onto the available servers (yes, batch computing is back!).

I hope that this talk will encourage you to demand similar facilities for your own work--00 buys you roughly two Terabytes of raw disk space these days, so why shouldn't *you* have a few? Similarly, PC-based servers provide substantial performance for little money. All that's missing is a few pieces of software to turn this cheap hardware into a reliable tool for researchers who need to process large datasets. If you encourage the cluster software community to provide this software, it might just happen.

Bio: Urs joined Google as its first VP Engineering after having been an Associate Professor of CS at UCSB and currently serves as the VP of Operations and as a Google Fellow. In his previous life he contributed to Sun's Hotspot JVM which is based in part on compiler research he performed at Stanford and Sun Labs as part of the Self project. With a good search engine you can find out much more about him.
Click to expand...

Its long but easy.If you have the time and the bandwidth do watch.

*Requires real player/alternative.

Log in or Sign up

Urs Hölzle(Google) Talk at CERN

Anon-068c403e2d Anonymized