- Notifications
You must be signed in to change notification settings - Fork68
Distributed Stream Processing
License
WallarooLabs/wally
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Wally is a fast stream-processing framework. Wally makes it easy to react to data in real-time. By eliminating infrastructure complexity, going from prototype to production has never been simpler.
When we set out to build Wally, we had several high-level goals in mind:
- Create a dependable and resilient distributed computing framework
- Take care of the complexities of distributed computing "plumbing," allowing developers to focus on their business logic
- Provide high-performance & low-latency data processing
- Be portable and deploy easily (i.e., run on-prem or any cloud)
- Manage in-memory state for the application
- Allow applications to scale as needed, even when they are live and up-and-running
Wally can be installed via our handy Wallaroo Up command. Check out ourinstallation page to learn more.
The primary API for Wally is written inPony. Wally applications are written using this Pony API.
Once you've installed Wally, Take a look at some of our examples. A great place to start are ourword_count ormarket spread examples inPony.
"""Word Count App"""use"assert"use"buffered"use"collections"use"net"use"serialise"use"wallaroo_labs/bytes"use"wallaroo"use"wallaroo_labs/logging"use"wallaroo_labs/mort"use"wallaroo_labs/time"use"wallaroo/core/common"use"wallaroo/core/metrics"use"wallaroo/core/sink/tcp_sink"use"wallaroo/core/source"use"wallaroo/core/source/tcp_source"use"wallaroo/core/state"use"wallaroo/core/topology"actorMainnewcreate(env:Env) =>Log.set_defaults()trylet pipeline =recovervallet lines =Wallaroo.source[String]("Word Count",TCPSourceConfig[String].from_options(StringFrameHandler,TCPSourceConfigCLIParser("Word Count", env.args)?,1)) lines .to[String](Split) .key_by(ExtractWord) .to[RunningTotal](AddCount) .to_sink(TCPSinkConfig[RunningTotal].from_options(RunningTotalEncoder,TCPSinkConfigCLIParser(env.args)?(0)?))endWallaroo.build_application(env,"Word Count", pipeline)else env.err.print("Couldn't build topology")endprimitiveSplit isStatelessComputation[String,String]funname():String =>"Split"funapply(s:String):Array[String]val =>let punctuation =""" !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"""let words =recovertrnArray[String]endfor linein s.split("\n").values()dolet cleaned =recoverval s.clone().>lower().>lstrip(punctuation) .>rstrip(punctuation)endfor wordin cleaned.split(punctuation).values()do words.push(word)endendconsume wordsclassvalRunningTotallet word:Stringlet count:U64newvalcreate(w:String, c:U64) => word = w count = cclassWordTotal isStatevar count:U64newcreate(c:U64) => count = cprimitiveAddCount isStateComputation[String,RunningTotal,WordTotal]funname():String =>"Add Count"funapply(word:String, state:WordTotal):RunningTotal => state.count = state.count +1RunningTotal(word, state.count)funinitial_state():WordTotal =>WordTotal(0)primitiveStringFrameHandler isFramedSourceHandler[String]funheader_length():USize =>4funpayload_length(data:Array[U8]iso):USize ? =>Bytes.to_u32(data(0)?,data(1)?,data(2)?,data(3)?).usize()fundecode(data:Array[U8]val):String =>String.from_array(data)primitiveExtractWordfunapply(input:String):Key => inputprimitiveRunningTotalEncoderfunapply(t:RunningTotal, wb:Writer =Writer):Array[ByteSeq]val =>let result =recovervalString().>append(t.word).>append(",").>append(t.count.string()) .>append("\n")end wb.write(result) wb.done()
Are you the sort who just wants to get going? Dive right into ourdocumentation then! It will get you up and running with Wally.
Wally currently exists as a mono-repo. All the source that is Wally is located in this repo. Seerepo directory structure for more information.
You can also take a look at ourFAQ.
Trying to figure out how to get started? Drop us a line:
We welcome contributions. Please see ourContribution Guide
For your pull request to be accepted you will need to accept ourContributor License Agreement
Wally is licensed under theApache version 2 license.
About
Distributed Stream Processing