Repository files navigation

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top ofCascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable toPig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

packagecom.twitter.scalding.examplesimportcom.twitter.scalding._importcom.twitter.scalding.source.TypedTextclassWordCountJob(args:Args)extendsJob(args) {TypedPipe.from(TextLine(args("input")))    .flatMap { line=> tokenize(line) }    .groupBy { word=> word }// use each word for a key    .size// in each group, get the size    .write(TypedText.tsv[(String,Long)](args("output")))// Split a piece of text into individual words.deftokenize(text:String):Array[String]= {// Lowercase each word and remove punctuation.    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]","").split("\\s+")  }}

Notice that thetokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code underexamples/. If you're interested in comparing Scalding to other languages, see ourRosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

• Getting Started page on theScalding Wiki
• Scalding Scaladocs provide details beyond the API References. Prefer using this as it's always up to date.
• REPL in Wonderland a hands-on tour of the scalding REPL requiring only git and java installed.
• Runnable tutorials in the source.
• The API Reference, including many example Scalding snippets:
  • Type-safe API Reference
  • Fields-based API Reference
• The Matrix Library provides a way of working with key-attribute-value scalding pipes:
  • TheIntroduction to Matrix Library contains an overview and a "getting started" example
  • TheMatrix API Reference contains the Matrix Library API reference with examples
• Introduction to Scalding Execution contains general rules and examples of calling Scalding from inside another application.

Please feel free to use the beautifulScalding logo artwork anywhere.

For user questions or scalding development (internals, extending, release planning):https://groups.google.com/forum/#!forum/scalding-dev (Google search also works as a first step)

In the remote possibility that there exist bugs in this code, please report them to:https://github.com/twitter/scalding/issues

Follow@Scalding on Twitter for updates.

Chat:

Pull requests and bug reports are always welcome!

We use a lightweight form of project governence inspired by the one used by Apache projects.Please seeContributing and Committership for our code of conduct and our pull request review process.The TL;DR is send us a pull request, iterate on the feedback + discussion, and get a +1 from aCommitter in order to get your PR accepted.

The current list of active committers (who can +1 a pull request) can be found here:Committers

A list of contributors to the project can be found here:Contributors

There is a script (called sbt) in the root that loads the correct sbt version to build:

1. ./sbt update (takes 2 minutes or more)
2. ./sbt test
3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer toFAQ page if you encounter problems when using sbt.

We use Github Actions to verify the build:

We useCoveralls for code coverage results:

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively,"com.twitter" and0.17.2.

Current published artifacts are

• scalding-core_2.11,scalding-core_2.12
• scalding-args_2.11,scalding-args_2.12
• scalding-date_2.11,scalding-date_2.12
• scalding-commons_2.11,scalding-commons_2.12
• scalding-avro_2.11,scalding-avro_2.12
• scalding-parquet_2.11,scalding-parquet_2.12
• scalding-repl_2.11,scalding-repl_2.12

The suffix denotes the scala version.

• Ebay
• Etsy
• Sharethrough
• Snowplow Analytics
• Soundcloud
• Twitter

To see a full list of users or to add yourself, see thewiki

• Avi Bryanthttp://twitter.com/avibryant
• Oscar Boykinhttp://twitter.com/posco
• Argyris Zymnishttp://twitter.com/argyris

Thanks for assistance and contributions:

• Sam Ritchiehttp://twitter.com/sritchie
• Aaron Siegel:http://twitter.com/asiegel
• Ian O'Connellhttp://twitter.com/0x138
• Alex Levensonhttp://twitter.com/THISWILLWORK
• Jonathan Coveneyhttp://twitter.com/jco
• Kevin Linhttp://twitter.com/reconditesea
• Brad Greenlee:http://twitter.com/bgreenlee
• Edwin Chenhttp://twitter.com/edchedch
• Arkajit Dey:http://twitter.com/arkajit
• Krishnan Raman:http://twitter.com/dxbydt_jasq
• Flavian Vasilehttp://twitter.com/flavianv
• Chris Wenselhttp://twitter.com/cwensel
• Ning Lianghttp://twitter.com/ningliang
• Dmitriy Ryaboyhttp://twitter.com/squarecog
• Dong Wanghttp://twitter.com/dongwang218
• Josh Attenberghttp://twitter.com/jattenberg
• Juliet Houglandhttps://twitter.com/j_houg
• Eddie Xiehttps://twitter.com/eddiex

A full list ofcontributors can be found on GitHub.

Copyright 2016 Twitter, Inc.

Licensed under theApache License, Version 2.0