The document summarizes the Unix philosophy of building systems out of small, single-purpose programs and how this approach can be applied to big data problems. It describes how Joyent's Manta object storage system brings this philosophy to big data by combining ZFS for scalable storage with OS-level virtualization using zones to allow Unix tools and approaches to be used on large datasets. Manta allows computations to be run directly on stored data rather than requiring data movement, enabling terse Unix-style one-liners to solve problems like word counting on big data.

What's hot

Viewers also liked

Similar to Bringing the Unix Philosophy to Big Data

More from bcantrill

Recently uploaded

Bringing the Unix Philosophy to Big Data

• 1.
  Bringing the UnixPhilosophyto Big DataBryan CantrillSVP, Engineeringbryan@joyent.com@bcantrill
• 2.
  Unix•When Unix appearedin the early 1970s, it was not just anew system, but a new way of thinking about systems•Instead of a sealed monolith, the operating system wasa collection of small, easily understood programs•First Edition Unix (1971) contained many programs thatwe still use today (ls, rm, cat, mv)•Its very name conveyed this minimalist aesthetic: Unix isa homophone of “eunuchs” — a castrated MulticsWe were a bit oppressed by the big system mentality. Kenwanted to do something simple. — Dennis Ritchie
• 3.
  Unix: Let therebe light•In 1969, Doug McIlroy had the idea of connectingdifferent components:At the same time that Thompson and Ritchie were sketchingout a file system, I was sketching out how to do dataprocessing on the blackboard by connecting togethercascades of processes•This was the primordial pipe, but it took three years topersuade Thompson to adopt it:And one day I came up with a syntax for the shell that wentalong with the piping, and Ken said, “I’m going to do it!”
• 4.
  Unix: ...and therewas lightAnd the next morning we had thisorgy of one-liners. — Doug McIlroy
• 5.
  The Unix philosophy•Thepipe — coupled with the small-system aesthetic —gave rise to the Unix philosophy, as articulated by DougMcIlroy:••Write programs to work together••Write programs that do one thing and do it wellWrite programs that handle text streams, becausethat is a universal interfaceFour decades later, this philosophy remains the singlemost important revolution in software systems thinking!
• 6.
  Doug McIlroy v.Don Knuth: FIGHT!•In 1986, Jon Bentley posed the challenge that becamethe Epic Rap Battle of computer science history:Read a file of text, determine the n most frequently usedwords, and print out a sorted list of those words along withtheir frequencies.•Don Knuth’s solution: an elaborate program in WEB, aPascal-like literate programming system of his owninvention, using a purpose-built algorithm•Doug McIlroy’s solution shows the power of the Unixphilosophy:tr -cs A-Za-z 'n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q
• 7.
  Big Data: Historyrepeats itself?•The original Google MapReduce paper (Dean et al.,OSDI ’04) poses a problem disturbingly similar toBentley’s challenge nearly two decades prior:Count of URL Access Frequency: The function processeslogs of web page requests and outputs ⟨URL, 1⟩. Thereduce function adds together all values for the same URLand emits a ⟨URL, total count⟩ pair••But the solutions do not adhere to the Unix philosophy...•e.g., Appendix A of the OSDI ’04 paper has a 71 lineword count in C++ — with nary a wc in sight...and nor do they make use of the substantial Unixfoundation for data processing
• 8.
  Big Data: Challenges•Mustbe able to scale storage to allow for “big data” —quantities of data that dwarf a single machine•••Must allow for massively parallel executionMust allow for multi-tenancyTo make use of both the Unix philosophy and its toolset,must be able to virtualize the operating system
• 9.
  Scaling storage•There areessentially three protocols for scalablestorage: block, file and object•Block (i.e., a SAN) is far too low an abstraction — andnotoriously expensive to scale•File (i.e., NAS) is too permissive an abstraction — itimplies a coherent store for arbitrary (partial) writes,trying (and failing) to be both C and A in CAP•Object (e.g., S3) is similar “enough” to a file-basedabstraction, but by not allowing partial writes, allows forproper CAP tradeoffs
• 10.
  Object storage••Object storagesystems do not allow for partial updates•A different approach is to have a highly reliable local filesystem that erasure encodes across local spindles —with entire objects duplicated across nodes foravailability•ZFS pioneered both reliability and efficiency of thismodel with RAID-Z — and has refined it over the pastdecade of production use•ZFS is one of the four foundational technologies inJoyent’s open source SmartOSFor both durability and availability, objects are generallyerasure encoded across spindles on different nodes
• 11.
  Virtualizing the operatingsystem?•Historically — since the 1960s — systems have beenvirtualized at the level of hardware•Hardware virtualization has its advantages, but it’sheavyweight: operating systems are not designed toshare resources like DRAM, CPU, I/O devices, etc.•One can instead virtualize at the level of the operatingsystem: a single OS kernel that creates lightweightcontainers — on the metal, but securely partitioned•Pioneered by BSD’s jails; taken to a logical extreme byzones found in Joyent’s SmartOS
• 12.
  Idea: ZFS +Zones?•Can we combine the efficiency and reliability of ZFSwith the abstraction provided by zones to develop anobject store that has compute as a first-class citizen?•ZFS rollback allows for zones to be trashed — simplyrollback the zone after compute completes on an object•Add a job scheduling system that allows for both mapand reduce phases of distributed work•Would allow for the Unix toolset to be used on arbitrarylarge amounts of data — unlocking big data one-liners•If it perhaps seems obvious now, it wasn’t at the time...
• 13.
  Idea: ZFS +Zones?
• 14.
  Manta: ZFS +Zones!•Building a sophisticated distributed system on top ofZFS and zones, we have built Manta, an internet-facingobject storage system offering in situ compute•That is, the description of compute can be brought towhere objects reside instead of having to backhaulobjects to transient compute•The abstractions made available for computation areanything that can run on the OS...•...and as a reminder, the OS — Unix — was built aroundthe notion of ad hoc unstructured data processing, andallows for remarkably terse expressions of computation
• 15.
  Manta: Unix forBig Data•Manta allows for an arbitrarily scalable variant ofMcIlroy’s solution to Bentley’s challenge:mfind -t o /bcantrill/public/v7/usr/man | mjob create -o -m "tr -cs A-Za-z 'n' | tr A-Z a-z | sort | uniq -c" -r "awk '{ x[$2] += $1 }END { for (w in x) { print x[w] " " w } }' | sort -rn | sed ${1}q"•This description not only terse, it is high performing: datais left at rest — with the “map” phase doing heavyreduction of the data stream•As such, Manta — like Unix — is not merely syntacticsugar; it converges compute and data in a new way
• 16.
  Manta: CAP tradeoffs•Eventualconsistency represents the wrong CAPtradeoffs for most; we prefer consistency overavailability for writes (but still availability for reads)•Many more details:http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/•Celebrity endorsement:
• 17.
  Manta: Other designprinciples•Hierarchical storage is an excellent idea (ht: Multics);Manta implements proper directories, delimited with aforward slash•Manta implements a snapshot/link hybrid dubbed asnaplink; can be used to effect versioning••Manta has full support for CORS headers••Manta SDKs exist for node.js, Java, Ruby, PythonManta uses SSH-based HTTP auth for client-sidetooling (IETF draft-cavage-http-signatures-00)“npm install manta” for command line interface
• 18.
  Manta and thefuture of big data•We believe compute/data convergence to be the futureof big data: stores of record must support computationas a first-class, in situ operation•We believe that Unix is a natural way of expressing thiscomputation — and that the OS is the right level atwhich to virtualize to support this securely•We believe that ZFS is the only sane storage substrateunderpinning for such a system•Manta will surely not be the only system to represent theconfluence of these — but it is the first•We are actively retooling our software stack in terms ofManta — Manta is changing the way we developsoftware!
• 19.
  Manta: More information•Productpage:http://joyent.com/products/manta•node.js module:https://github.com/joyent/node-manta•Manta documentation:http://apidocs.joyent.com/manta/•IRC, e-mail, Twitter, etc.:#manta on freenode, manta@joyent.com, @mcavage,@dapsays, @yunongx, @joyent•Here’s to the orgy of big data one-liners!