Spark Streaming Programming Guide

• Note
• Overview
• A Quick Example
• Basic Concepts
  • Linking
  • Initializing StreamingContext
  • Discretized Streams (DStreams)
  • Input DStreams and Receivers
  • Transformations on DStreams
  • Output Operations on DStreams
  • DataFrame and SQL Operations
  • MLlib Operations
  • Caching / Persistence
  • Checkpointing
  • Accumulators, Broadcast Variables, and Checkpoints
  • Deploying Applications
  • Monitoring Applications
• Performance Tuning
  • Reducing the Batch Processing Times
  • Setting the Right Batch Interval
  • Memory Tuning
• Fault-tolerance Semantics
• Where to Go from Here

Note

Spark Streaming is the previous generation of Spark’s streaming engine. There are no longerupdates to Spark Streaming and it’s a legacy project. There is a newer and easier to usestreaming engine in Spark called Structured Streaming. You should use Spark Structured Streamingfor your streaming applications and pipelines. SeeStructured Streaming Programming Guide.

Overview

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput,fault-tolerant stream processing of live data streams. Data can be ingested from many sourceslike Kafka, Kinesis, or TCP sockets, and can be processed using complexalgorithms expressed with high-level functions likemap,reduce,join andwindow.Finally, processed data can be pushed out to filesystems, databases,and live dashboards. In fact, you can apply Spark’smachine learning andgraph processing algorithms on data streams.

Internally, it works as follows. Spark Streaming receives live input data streams and dividesthe data into batches, which are then processed by the Spark engine to generate the finalstream of results in batches.

Spark Streaming provides a high-level abstraction calleddiscretized stream orDStream,which represents a continuous stream of data. DStreams can be created either from input datastreams from sources such as Kafka, and Kinesis, or by applying high-leveloperations on other DStreams. Internally, a DStream is represented as a sequence ofRDDs.

This guide shows you how to start writing Spark Streaming programs with DStreams. You canwrite Spark Streaming programs in Scala, Java or Python (introduced in Spark 1.2),all of which are presented in this guide.You will find tabs throughout this guide that let you choose between code snippets ofdifferent languages.

Note: There are a few APIs that are either different or not available in Python. Throughout this guide, you will find the tagPython API highlighting these differences.

A Quick Example

Before we go into the details of how to write your own Spark Streaming program,let’s take a quick look at what a simple Spark Streaming program looks like. Let’s say we want tocount the number of words in text data received from a data server listening on a TCPsocket. All you need todo is as follows.

frompysparkimportSparkContextfrompyspark.streamingimportStreamingContext# Create a local StreamingContext with two working thread and batch interval of 1 secondsc=SparkContext("local[2]","NetworkWordCount")ssc=StreamingContext(sc,1)

# Create a DStream that will connect to hostname:port, like localhost:9999lines=ssc.socketTextStream("localhost",9999)

# Split each line into wordswords=lines.flatMap(lambdaline:line.split(""))

# Count each word in each batchpairs=words.map(lambdaword:(word,1))wordCounts=pairs.reduceByKey(lambdax,y:x+y)# Print the first ten elements of each RDD generated in this DStream to the consolewordCounts.pprint()

ssc.start()# Start the computationssc.awaitTermination()# Wait for the computation to terminate

importorg.apache.spark._importorg.apache.spark.streaming._importorg.apache.spark.streaming.StreamingContext._// not necessary since Spark 1.3// Create a local StreamingContext with two working thread and batch interval of 1 second.// The master requires 2 cores to prevent a starvation scenario.valconf=newSparkConf().setMaster("local[2]").setAppName("NetworkWordCount")valssc=newStreamingContext(conf,Seconds(1))

// Create a DStream that will connect to hostname:port, like localhost:9999vallines=ssc.socketTextStream("localhost",9999)

// Split each line into wordsvalwords=lines.flatMap(_.split(" "))

importorg.apache.spark.streaming.StreamingContext._// not necessary since Spark 1.3// Count each word in each batchvalpairs=words.map(word=>(word,1))valwordCounts=pairs.reduceByKey(_+_)// Print the first ten elements of each RDD generated in this DStream to the consolewordCounts.print()

ssc.start()// Start the computationssc.awaitTermination()// Wait for the computation to terminate

importorg.apache.spark.*;importorg.apache.spark.api.java.function.*;importorg.apache.spark.streaming.*;importorg.apache.spark.streaming.api.java.*;importscala.Tuple2;// Create a local StreamingContext with two working thread and batch interval of 1 secondSparkConfconf=newSparkConf().setMaster("local[2]").setAppName("NetworkWordCount");JavaStreamingContextjssc=newJavaStreamingContext(conf,Durations.seconds(1));

// Create a DStream that will connect to hostname:port, like localhost:9999JavaReceiverInputDStream<String>lines=jssc.socketTextStream("localhost",9999);

// Split each line into wordsJavaDStream<String>words=lines.flatMap(x->Arrays.asList(x.split(" ")).iterator());

// Count each word in each batchJavaPairDStream<String,Integer>pairs=words.mapToPair(s->newTuple2<>(s,1));JavaPairDStream<String,Integer>wordCounts=pairs.reduceByKey((i1,i2)->i1+i2);// Print the first ten elements of each RDD generated in this DStream to the consolewordCounts.print();

jssc.start();// Start the computationjssc.awaitTermination();// Wait for the computation to terminate

If you have alreadydownloaded andbuilt Spark,you can run this example as follows. You will first need to run Netcat(a small utility found in most Unix-like systems) as a data server by using

$nc-lk 9999

Then, in a different terminal, you can start the example by using

$./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999

$./bin/run-example streaming.NetworkWordCount localhost 9999

$./bin/run-example streaming.JavaNetworkWordCount localhost 9999

Then, any lines typed in the terminal running the netcat server will be counted and printed onscreen every second. It will look something like the following.

# TERMINAL 1:# Running Netcat$nc-lk 9999hello world...

# TERMINAL 2: RUNNING network_wordcount.py$./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999...-------------------------------------------Time: 2014-10-14 15:25:21-------------------------------------------(hello,1)(world,1)...

# TERMINAL 2: RUNNING NetworkWordCount$./bin/run-example streaming.NetworkWordCount localhost 9999...-------------------------------------------Time: 1357008430000 ms-------------------------------------------(hello,1)(world,1)...

# TERMINAL 2: RUNNING JavaNetworkWordCount$./bin/run-example streaming.JavaNetworkWordCount localhost 9999...-------------------------------------------Time: 1357008430000 ms-------------------------------------------(hello,1)(world,1)...

Basic Concepts

Next, we move beyond the simple example and elaborate on the basics of Spark Streaming.

Linking

Similar to Spark, Spark Streaming is available through Maven Central. To write your own Spark Streaming program, you will have to add the following dependency to your SBT or Maven project.

<dependency>    <groupId>org.apache.spark</groupId>    <artifactId>spark-streaming_2.13</artifactId>    <version>4.0.1</version>    <scope>provided</scope></dependency>

libraryDependencies += "org.apache.spark" % "spark-streaming_2.13" % "4.0.1" % "provided"

For ingesting data from sources like Kafka and Kinesis that are not present in the SparkStreaming core API, you will have to add the correspondingartifactspark-streaming-xyz_2.13 to the dependencies. For example,some of the common ones are as follows.

For an up-to-date list, please refer to theMaven repositoryfor the full list of supported sources and artifacts.

Initializing StreamingContext

To initialize a Spark Streaming program, aStreamingContext object has to be created which is the main entry point of all Spark Streaming functionality.

frompysparkimportSparkContextfrompyspark.streamingimportStreamingContextsc=SparkContext(master,appName)ssc=StreamingContext(sc,1)

importorg.apache.spark._importorg.apache.spark.streaming._valconf=newSparkConf().setAppName(appName).setMaster(master)valssc=newStreamingContext(conf,Seconds(1))

importorg.apache.spark.streaming._valsc=...// existing SparkContextvalssc=newStreamingContext(sc,Seconds(1))

importorg.apache.spark.*;importorg.apache.spark.streaming.api.java.*;SparkConfconf=newSparkConf().setAppName(appName).setMaster(master);JavaStreamingContextssc=newJavaStreamingContext(conf,newDuration(1000));

importorg.apache.spark.streaming.api.java.*;JavaSparkContextsc=...//existing JavaSparkContextJavaStreamingContextssc=newJavaStreamingContext(sc,Durations.seconds(1));

After a context is defined, you have to do the following.

1. Define the input sources by creating input DStreams.
2. Define the streaming computations by applying transformation and output operations to DStreams.
3. Start receiving data and processing it usingstreamingContext.start().
4. Wait for the processing to be stopped (manually or due to any error) usingstreamingContext.awaitTermination().
5. The processing can be manually stopped usingstreamingContext.stop().

Points to remember:

• Once a context has been started, no new streaming computations can be set up or added to it.
• Once a context has been stopped, it cannot be restarted.
• Only one StreamingContext can be active in a JVM at the same time.
• stop() on StreamingContext also stops the SparkContext. To stop only the StreamingContext, set the optional parameter ofstop() calledstopSparkContext to false.
• A SparkContext can be re-used to create multiple StreamingContexts, as long as the previous StreamingContext is stopped (without stopping the SparkContext) before the next StreamingContext is created.

Discretized Streams (DStreams)

Discretized Stream orDStream is the basic abstraction provided by Spark Streaming.It represents a continuous stream of data, either the input data stream received from source,or the processed data stream generated by transforming the input stream. Internally,a DStream is represented by a continuous series of RDDs, which is Spark’s abstraction of an immutable,distributed dataset (seeSpark Programming Guide for more details). Each RDD in a DStream contains data from a certain interval,as shown in the following figure.

Any operation applied on a DStream translates to operations on the underlying RDDs. For example,in theearlier example of converting a stream of lines to words,theflatMap operation is applied on each RDD in thelines DStream to generate the RDDs of thewords DStream. This is shown in the following figure.

These underlying RDD transformations are computed by the Spark engine. The DStream operationshide most of these details and provide the developer with a higher-level API for convenience.These operations are discussed in detail in later sections.

Input DStreams and Receivers

Input DStreams are DStreams representing the stream of input data received from streamingsources. In thequick example,lines was an input DStream as it representedthe stream of data received from the netcat server. Every input DStream(except file stream, discussed later in this section) is associated with aReceiver(Scala doc,Java doc) object which receives thedata from a source and stores it in Spark’s memory for processing.

Spark Streaming provides two categories of built-in streaming sources.

• Basic sources: Sources directly available in the StreamingContext API.Examples: file systems, and socket connections.
• Advanced sources: Sources like Kafka, Kinesis, etc. are available throughextra utility classes. These require linking against extra dependencies as discussed in thelinking section.

We are going to discuss some of the sources present in each category later in this section.

Note that, if you want to receive multiple streams of data in parallel in your streamingapplication, you can create multiple input DStreams (discussedfurther in thePerformance Tuning section). This willcreate multiple receivers which will simultaneously receive multiple data streams. But note that aSpark worker/executor is a long-running task, hence it occupies one of the cores allocated to theSpark Streaming application. Therefore, it is important to remember that a Spark Streaming applicationneeds to be allocated enough cores (or threads, if running locally) to process the received data,as well as to run the receiver(s).

Points to remember

• When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL.Either of these means that only one thread will be used for running tasks locally. If you are usingan input DStream based on a receiver (e.g. sockets, Kafka, etc.), then the single thread willbe used to run the receiver, leaving no thread for processing the received data. Hence, whenrunning locally, always use “local[n]” as the master URL, wheren > number of receivers to run(seeSpark Properties for information on how to setthe master).

• Extending the logic to running on a cluster, the number of cores allocated to the Spark Streamingapplication must be more than the number of receivers. Otherwise the system will receive data, butnot be able to process it.

Basic Sources

We have already taken a look at thessc.socketTextStream(...) in thequick examplewhich creates a DStream from textdata received over a TCP socket connection. Besides sockets, the StreamingContext API providesmethods for creating DStreams from files as input sources.

File Streams

For reading data from files on any file system compatible with the HDFS API (that is, HDFS, S3, NFS, etc.), a DStream can be created asviaStreamingContext.fileStream[KeyClass, ValueClass, InputFormatClass].

File streams do not require running a receiver so there is no need to allocate any cores for receiving file data.

For simple text files, the easiest method isStreamingContext.textFileStream(dataDirectory).

streamingContext.textFileStream(dataDirectory)

streamingContext.fileStream[KeyClass,ValueClass,InputFormatClass](dataDirectory)

streamingContext.textFileStream(dataDirectory)

streamingContext.fileStream<KeyClass,ValueClass,InputFormatClass>(dataDirectory);

streamingContext.textFileStream(dataDirectory);

How Directories are Monitored

Spark Streaming will monitor the directorydataDirectory and process any files created in that directory.

• A simple directory can be monitored, such as"hdfs://namenode:8040/logs/".All files directly under such a path will be processed as they are discovered.
• APOSIX glob pattern can be supplied, such as"hdfs://namenode:8040/logs/2017/*".Here, the DStream will consist of all files in the directoriesmatching the pattern.That is: it is a pattern of directories, not of files in directories.
• All files must be in the same data format.
• A file is considered part of a time period based on its modification time,not its creation time.
• Once processed, changes to a file within the current window will not cause the file to be reread.That is:updates are ignored.
• The more files under a directory, the longer it will take toscan for changes — even if no files have been modified.
• If a wildcard is used to identify directories, such as"hdfs://namenode:8040/logs/2016-*",renaming an entire directory to match the path will add the directory to the list ofmonitored directories. Only the files in the directory whose modification time iswithin the current window will be included in the stream.
• CallingFileSystem.setTimes()to fix the timestamp is a way to have the file picked up in a later window, even if its contents have not changed.

Using Object Stores as a source of data

“Full” Filesystems such as HDFS tend to set the modification time on their files as soonas the output stream is created.When a file is opened, even before data has been completely written,it may be included in theDStream - after which updates to the file within the same windowwill be ignored. That is: changes may be missed, and data omitted from the stream.

To guarantee that changes are picked up in a window, write the fileto an unmonitored directory, then, immediately after the output stream is closed,rename it into the destination directory.Provided the renamed file appears in the scanned destination directory during the windowof its creation, the new data will be picked up.

In contrast, Object Stores such as Amazon S3 and Azure Storage usually have slow rename operations, as thedata is actually copied.Furthermore, a renamed object may have the time of therename() operation as its modification time, somay not be considered part of the window which the original create time implied they were.

Careful testing is needed against the target object store to verify that the timestamp behaviorof the store is consistent with that expected by Spark Streaming. It may bethat writing directly into a destination directory is the appropriate strategy forstreaming data via the chosen object store.

For more details on this topic, consult theHadoop Filesystem Specification.

Streams based on Custom Receivers

DStreams can be created with data streams received through custom receivers. See theCustom Receiver Guide for more details.

Queue of RDDs as a Stream

For testing a Spark Streaming application with test data, one can also create a DStream based on a queue of RDDs, usingstreamingContext.queueStream(queueOfRDDs). Each RDD pushed into the queue will be treated as a batch of data in the DStream, and processed like a stream.

For more details on streams from sockets and files, see the API documentations of the relevant functions inStreamingContext for Python,StreamingContext for Scala,andJavaStreamingContext for Java.

Advanced Sources

Python API As of Spark 4.0.1,out of these sources, Kafka and Kinesis are available in the Python API.

This category of sources requires interfacing with external non-Spark libraries, some of them withcomplex dependencies (e.g., Kafka). Hence, to minimize issues related to version conflictsof dependencies, the functionality to create DStreams from these sources has been moved to separatelibraries that can belinked to explicitly when necessary.

Note that these advanced sources are not available in the Spark shell, hence applications based onthese advanced sources cannot be tested in the shell. If you really want to use them in the Sparkshell you will have to download the corresponding Maven artifact’s JAR along with its dependenciesand add it to the classpath.

Some of these advanced sources are as follows.

• Kafka: Spark Streaming 4.0.1 is compatible with Kafka broker versions 0.10 or higher. See theKafka Integration Guide for more details.

• Kinesis: Spark Streaming 4.0.1 is compatible with Kinesis Client Library 1.2.1. See theKinesis Integration Guide for more details.

Custom Sources

Python API This is not yet supported in Python.

Input DStreams can also be created out of custom data sources. All you have to do is implement auser-definedreceiver (see next section to understand what that is) that can receive data fromthe custom sources and push it into Spark. See theCustom ReceiverGuide for details.

Receiver Reliability

There can be two kinds of data sources based on theirreliability. Sources(like Kafka) allow the transferred data to be acknowledged. If the system receivingdata from thesereliable sources acknowledges the received data correctly, it can be ensuredthat no data will be lost due to any kind of failure. This leads to two kinds of receivers:

1. Reliable Receiver - Areliable receiver correctly sends acknowledgment to a reliable source when the data has been received and stored in Spark with replication.
2. Unreliable Receiver - Anunreliable receiver doesnot send acknowledgment to a source. This can be used for sources that do not support acknowledgment, or even for reliable sources when one does not want or need to go into the complexity of acknowledgment.

The details of how to write a reliable receiver are discussed in theCustom Receiver Guide.

Transformations on DStreams

Similar to that of RDDs, transformations allow the data from the input DStream to be modified.DStreams support many of the transformations available on normal Spark RDD’s.Some of the common ones are as follows.

A few of these transformations are worth discussing in more detail.

UpdateStateByKey Operation

TheupdateStateByKey operation allows you to maintain arbitrary state while continuously updatingit with new information. To use this, you will have to do two steps.

1. Define the state - The state can be an arbitrary data type.
2. Define the state update function - Specify with a function how to update the state using theprevious state and the new values from an input stream.

In every batch, Spark will apply the state update function for all existing keys, regardless of whether they have new data in a batch or not. If the update function returnsNone then the key-value pair will be eliminated.

Let’s illustrate this with an example. Say you want to maintain a running count of each wordseen in a text data stream. Here, the running count is the state and it is an integer. Wedefine the update function as:

defupdateFunction(newValues,runningCount):ifrunningCountisNone:runningCount=0returnsum(newValues,runningCount)# add the new values with the previous running count to get the new count

runningCounts=pairs.updateStateByKey(updateFunction)

defupdateFunction(newValues:Seq[Int],runningCount:Option[Int]):Option[Int]={valnewCount=...// add the new values with the previous running count to get the new countSome(newCount)}

valrunningCounts=pairs.updateStateByKey[Int](updateFunction_)

Function2<List<Integer>,Optional<Integer>,Optional<Integer>>updateFunction=(values,state)->{IntegernewSum=...// add the new values with the previous running count to get the new countreturnOptional.of(newSum);};

JavaPairDStream<String,Integer>runningCounts=pairs.updateStateByKey(updateFunction);

Note that usingupdateStateByKey requires the checkpoint directory to be configured, which isdiscussed in detail in thecheckpointing section.

Transform Operation

Thetransform operation (along with its variations liketransformWith) allowsarbitrary RDD-to-RDD functions to be applied on a DStream. It can be used to apply any RDDoperation that is not exposed in the DStream API.For example, the functionality of joining every batch in a data streamwith another dataset is not directly exposed in the DStream API. However,you can easily usetransform to do this. This enables very powerful possibilities. For example,one can do real-time data cleaning by joining the input data stream with precomputedspam information (maybe generated with Spark as well) and then filtering based on it.

spamInfoRDD=sc.pickleFile(...)# RDD containing spam information# join data stream with spam information to do data cleaningcleanedDStream=wordCounts.transform(lambdardd:rdd.join(spamInfoRDD).filter(...))

valspamInfoRDD=ssc.sparkContext.newAPIHadoopRDD(...)// RDD containing spam informationvalcleanedDStream=wordCounts.transform{rdd=>rdd.join(spamInfoRDD).filter(...)// join data stream with spam information to do data cleaning...}

importorg.apache.spark.streaming.api.java.*;// RDD containing spam informationJavaPairRDD<String,Double>spamInfoRDD=jssc.sparkContext().newAPIHadoopRDD(...);JavaPairDStream<String,Integer>cleanedDStream=wordCounts.transform(rdd->{rdd.join(spamInfoRDD).filter(...);// join data stream with spam information to do data cleaning...});

Note that the supplied function gets called in every batch interval. This allows you to dotime-varying RDD operations, that is, RDD operations, number of partitions, broadcast variables,etc. can be changed between batches.

Window Operations

Spark Streaming also provideswindowed computations, which allow you to applytransformations over a sliding window of data. The following figure illustrates this slidingwindow.

As shown in the figure, every time the windowslides over a source DStream,the source RDDs that fall within the window are combined and operated upon to produce theRDDs of the windowed DStream. In this specific case, the operation is applied over the last 3 timeunits of data, and slides by 2 time units. This shows that any window operation needs tospecify two parameters.

• window length - The duration of the window (3 in the figure).
• sliding interval - The interval at which the window operation is performed (2 in the figure).

These two parameters must be multiples of the batch interval of the source DStream (1 in thefigure).

Let’s illustrate the window operations with an example. Say, you want to extend theearlier example by generating word counts over the last 30 seconds of data,every 10 seconds. To do this, we have to apply thereduceByKey operation on thepairs DStream of(word, 1) pairs over the last 30 seconds of data. This is done using theoperationreduceByKeyAndWindow.

# Reduce last 30 seconds of data, every 10 secondswindowedWordCounts=pairs.reduceByKeyAndWindow(lambdax,y:x+y,lambdax,y:x-y,30,10)

// Reduce last 30 seconds of data, every 10 secondsvalwindowedWordCounts=pairs.reduceByKeyAndWindow((a:Int,b:Int)=>(a+b),Seconds(30),Seconds(10))

// Reduce last 30 seconds of data, every 10 secondsJavaPairDStream<String,Integer>windowedWordCounts=pairs.reduceByKeyAndWindow((i1,i2)->i1+i2,Durations.seconds(30),Durations.seconds(10));

Some of the common window operations are as follows. All of these operations take thesaid two parameters -windowLength andslideInterval.

Join Operations

Finally, it’s worth highlighting how easily you can perform different kinds of joins in Spark Streaming.

Stream-stream joins

Streams can be very easily joined with other streams.

stream1=...stream2=...joinedStream=stream1.join(stream2)

valstream1:DStream[String,String]=...valstream2:DStream[String,String]=...valjoinedStream=stream1.join(stream2)

JavaPairDStream<String,String>stream1=...JavaPairDStream<String,String>stream2=...JavaPairDStream<String,Tuple2<String,String>>joinedStream=stream1.join(stream2);

Here, in each batch interval, the RDD generated bystream1 will be joined with the RDD generated bystream2. You can also doleftOuterJoin,rightOuterJoin,fullOuterJoin. Furthermore, it is often very useful to do joins over windows of the streams. That is pretty easy as well.

windowedStream1=stream1.window(20)windowedStream2=stream2.window(60)joinedStream=windowedStream1.join(windowedStream2)

valwindowedStream1=stream1.window(Seconds(20))valwindowedStream2=stream2.window(Minutes(1))valjoinedStream=windowedStream1.join(windowedStream2)

JavaPairDStream<String,String>windowedStream1=stream1.window(Durations.seconds(20));JavaPairDStream<String,String>windowedStream2=stream2.window(Durations.minutes(1));JavaPairDStream<String,Tuple2<String,String>>joinedStream=windowedStream1.join(windowedStream2);

Stream-dataset joins

This has already been shown earlier while explainDStream.transform operation. Here is yet another example of joining a windowed stream with a dataset.

dataset=...# some RDDwindowedStream=stream.window(20)joinedStream=windowedStream.transform(lambdardd:rdd.join(dataset))

valdataset:RDD[String,String]=...valwindowedStream=stream.window(Seconds(20))...valjoinedStream=windowedStream.transform{rdd=>rdd.join(dataset)}

JavaPairRDD<String,String>dataset=...JavaPairDStream<String,String>windowedStream=stream.window(Durations.seconds(20));JavaPairDStream<String,String>joinedStream=windowedStream.transform(rdd->rdd.join(dataset));

In fact, you can also dynamically change the dataset you want to join against. The function provided totransform is evaluated every batch interval and therefore will use the current dataset thatdataset reference points to.

The complete list of DStream transformations is available in the API documentation. For the Python API,seeDStream.For the Scala API, seeDStreamandPairDStreamFunctions.For the Java API, seeJavaDStreamandJavaPairDStream.

Output Operations on DStreams

Output operations allow DStream’s data to be pushed out to external systems like a database or a file system.Since the output operations actually allow the transformed data to be consumed by external systems,they trigger the actual execution of all the DStream transformations (similar to actions for RDDs).Currently, the following output operations are defined:

Design Patterns for using foreachRDD

dstream.foreachRDD is a powerful primitive that allows data to be sent out to external systems.However, it is important to understand how to use this primitive correctly and efficiently.Some of the common mistakes to avoid are as follows.

Often writing data to external systems requires creating a connection object(e.g. TCP connection to a remote server) and using it to send data to a remote system.For this purpose, a developer may inadvertently try creating a connection object atthe Spark driver, and then try to use it in a Spark worker to save records in the RDDs.For example (in Scala),

defsendRecord(rdd):connection=createNewConnection()# executed at the driverrdd.foreach(lambdarecord:connection.send(record))connection.close()dstream.foreachRDD(sendRecord)

dstream.foreachRDD{rdd=>valconnection=createNewConnection()// executed at the driverrdd.foreach{record=>connection.send(record)// executed at the worker}}

dstream.foreachRDD(rdd->{Connectionconnection=createNewConnection();// executed at the driverrdd.foreach(record->{connection.send(record);//executedattheworker});});

This is incorrect as this requires the connection object to be serialized and sent from thedriver to the worker. Such connection objects are rarely transferable across machines. Thiserror may manifest as serialization errors (connection object not serializable), initializationerrors (connection object needs to be initialized at the workers), etc. The correct solution isto create the connection object at the worker.

However, this can lead to another common mistake - creating a new connection for every record.For example,

defsendRecord(record):connection=createNewConnection()connection.send(record)connection.close()dstream.foreachRDD(lambdardd:rdd.foreach(sendRecord))

dstream.foreachRDD{rdd=>rdd.foreach{record=>valconnection=createNewConnection()connection.send(record)connection.close()}}

dstream.foreachRDD(rdd->{rdd.foreach(record->{Connectionconnection=createNewConnection();connection.send(record);connection.close();});});

Typically, creating a connection object has time and resource overheads. Therefore, creating anddestroying a connection object for each record can incur unnecessarily high overheads and cansignificantly reduce the overall throughput of the system. A better solution is to userdd.foreachPartition - create a single connection object and send all the records in a RDDpartition using that connection.

defsendPartition(iter):connection=createNewConnection()forrecordiniter:connection.send(record)connection.close()dstream.foreachRDD(lambdardd:rdd.foreachPartition(sendPartition))

dstream.foreachRDD{rdd=>rdd.foreachPartition{partitionOfRecords=>valconnection=createNewConnection()partitionOfRecords.foreach(record=>connection.send(record))connection.close()}}

dstream.foreachRDD(rdd->{rdd.foreachPartition(partitionOfRecords->{Connectionconnection=createNewConnection();while(partitionOfRecords.hasNext()){connection.send(partitionOfRecords.next());}connection.close();});});

This amortizes the connection creation overheads over many records.

Finally, this can be further optimized by reusing connection objects across multiple RDDs/batches.One can maintain a static pool of connection objects than can be reused asRDDs of multiple batches are pushed to the external system, thus further reducing the overheads.

defsendPartition(iter):# ConnectionPool is a static, lazily initialized pool of connectionsconnection=ConnectionPool.getConnection()forrecordiniter:connection.send(record)# return to the pool for future reuseConnectionPool.returnConnection(connection)dstream.foreachRDD(lambdardd:rdd.foreachPartition(sendPartition))

dstream.foreachRDD{rdd=>rdd.foreachPartition{partitionOfRecords=>// ConnectionPool is a static, lazily initialized pool of connectionsvalconnection=ConnectionPool.getConnection()partitionOfRecords.foreach(record=>connection.send(record))ConnectionPool.returnConnection(connection)// return to the pool for future reuse}}

dstream.foreachRDD(rdd->{rdd.foreachPartition(partitionOfRecords->{// ConnectionPool is a static, lazily initialized pool of connectionsConnectionconnection=ConnectionPool.getConnection();while(partitionOfRecords.hasNext()){connection.send(partitionOfRecords.next());}ConnectionPool.returnConnection(connection);// return to the pool for future reuse});});

Note that the connections in the pool should be lazily created on demand and timed out if not used for a while. This achieves the most efficient sending of data to external systems.

Other points to remember:

• DStreams are executed lazily by the output operations, just like RDDs are lazily executed by RDD actions. Specifically, RDD actions inside the DStream output operations force the processing of the received data. Hence, if your application does not have any output operation, or has output operations likedstream.foreachRDD() without any RDD action inside them, then nothing will get executed. The system will simply receive the data and discard it.

• By default, output operations are executed one-at-a-time. And they are executed in the order they are defined in the application.

DataFrame and SQL Operations

You can easily useDataFrames and SQL operations on streaming data. You have to create a SparkSession using the SparkContext that the StreamingContext is using. Furthermore, this has to be done such that it can be restarted on driver failures. This is done by creating a lazily instantiated singleton instance of SparkSession. This is shown in the following example. It modifies the earlierword count example to generate word counts using DataFrames and SQL. Each RDD is converted to a DataFrame, registered as a temporary table and then queried using SQL.

# Lazily instantiated global instance of SparkSessiondefgetSparkSessionInstance(sparkConf):if("sparkSessionSingletonInstance"notinglobals()):globals()["sparkSessionSingletonInstance"]=SparkSession \.builder \.config(conf=sparkConf) \.getOrCreate()returnglobals()["sparkSessionSingletonInstance"]...# DataFrame operations inside your streaming programwords=...# DStream of stringsdefprocess(time,rdd):print("========= %s ========="%str(time))try:# Get the singleton instance of SparkSessionspark=getSparkSessionInstance(rdd.context.getConf())# Convert RDD[String] to RDD[Row] to DataFramerowRdd=rdd.map(lambdaw:Row(word=w))wordsDataFrame=spark.createDataFrame(rowRdd)# Creates a temporary view using the DataFramewordsDataFrame.createOrReplaceTempView("words")# Do word count on table using SQL and print itwordCountsDataFrame=spark.sql("select word, count(*) as total from words group by word")wordCountsDataFrame.show()except:passwords.foreachRDD(process)

/** DataFrame operations inside your streaming program */valwords:DStream[String]=...words.foreachRDD{rdd=>// Get the singleton instance of SparkSessionvalspark=SparkSession.builder.config(rdd.sparkContext.getConf).getOrCreate()importspark.implicits._// Convert RDD[String] to DataFramevalwordsDataFrame=rdd.toDF("word")// Create a temporary viewwordsDataFrame.createOrReplaceTempView("words")// Do word count on DataFrame using SQL and print itvalwordCountsDataFrame=spark.sql("select word, count(*) as total from words group by word")wordCountsDataFrame.show()}

/** Java Bean class for converting RDD to DataFrame */publicclassJavaRowimplementsjava.io.Serializable{privateStringword;publicStringgetWord(){returnword;}publicvoidsetWord(Stringword){this.word=word;}}.../** DataFrame operations inside your streaming program */JavaDStream<String>words=...words.foreachRDD((rdd,time)->{// Get the singleton instance of SparkSessionSparkSessionspark=SparkSession.builder().config(rdd.sparkContext().getConf()).getOrCreate();// Convert RDD[String] to RDD[case class] to DataFrameJavaRDD<JavaRow>rowRDD=rdd.map(word->{JavaRowrecord=newJavaRow();record.setWord(word);returnrecord;});DataFramewordsDataFrame=spark.createDataFrame(rowRDD,JavaRow.class);// Creates a temporary view using the DataFramewordsDataFrame.createOrReplaceTempView("words");// Do word count on table using SQL and print itDataFramewordCountsDataFrame=spark.sql("select word, count(*) as total from words group by word");wordCountsDataFrame.show();});

You can also run SQL queries on tables defined on streaming data from a different thread (that is, asynchronous to the running StreamingContext). Just make sure that you set the StreamingContext to remember a sufficient amount of streaming data such that the query can run. Otherwise the StreamingContext, which is unaware of any of the asynchronous SQL queries, will delete off old streaming data before the query can complete. For example, if you want to query the last batch, but your query can take 5 minutes to run, then callstreamingContext.remember(Minutes(5)) (in Scala, or equivalent in other languages).

See theDataFrames and SQL guide to learn more about DataFrames.

MLlib Operations

You can also easily use machine learning algorithms provided byMLlib. First of all, there are streaming machine learning algorithms (e.g.Streaming Linear Regression,Streaming KMeans, etc.) which can simultaneously learn from the streaming data as well as apply the model on the streaming data. Beyond these, for a much larger class of machine learning algorithms, you can learn a learning model offline (i.e. using historical data) and then apply the model online on streaming data. See theMLlib guide for more details.

Caching / Persistence

Similar to RDDs, DStreams also allow developers to persist the stream’s data in memory. That is,using thepersist() method on a DStream will automatically persist every RDD of that DStream inmemory. This is useful if the data in the DStream will be computed multiple times (e.g., multipleoperations on the same data). For window-based operations likereduceByWindow andreduceByKeyAndWindow and state-based operations likeupdateStateByKey, this is implicitly true.Hence, DStreams generated by window-based operations are automatically persisted in memory, withoutthe developer callingpersist().

For input streams that receive data over the network (such as, Kafka, sockets, etc.), thedefault persistence level is set to replicate the data to two nodes for fault-tolerance.

Note that, unlike RDDs, the default persistence level of DStreams keeps the data serialized inmemory. This is further discussed in thePerformance Tuning section. Moreinformation on different persistence levels can be found in theSpark Programming Guide.

Checkpointing

A streaming application must operate 24/7 and hence must be resilient to failures unrelatedto the application logic (e.g., system failures, JVM crashes, etc.). For this to be possible,Spark Streaming needs tocheckpoint enough information to a fault-tolerant storage system such that it can recover from failures. There are two types of datathat are checkpointed.

• Metadata checkpointing - Saving of the information defining the streaming computation tofault-tolerant storage like HDFS. This is used to recover from failure of the node running thedriver of the streaming application (discussed in detail later). Metadata includes:
  • Configuration - The configuration that was used to create the streaming application.
  • DStream operations - The set of DStream operations that define the streaming application.
  • Incomplete batches - Batches whose jobs are queued but have not completed yet.
• Data checkpointing - Saving of the generated RDDs to reliable storage. This is necessaryin somestateful transformations that combine data across multiple batches. In suchtransformations, the generated RDDs depend on RDDs of previous batches, which causes the lengthof the dependency chain to keep increasing with time. To avoid such unbounded increases in recovery time (proportional to dependency chain), intermediate RDDs of stateful transformations are periodicallycheckpointed to reliable storage (e.g. HDFS) to cut off the dependency chains.

To summarize, metadata checkpointing is primarily needed for recovery from driver failures,whereas data or RDD checkpointing is necessary even for basic functioning if statefultransformations are used.

When to enable Checkpointing

Checkpointing must be enabled for applications with any of the following requirements:

• Usage of stateful transformations - If eitherupdateStateByKey orreduceByKeyAndWindow (withinverse function) is used in the application, then the checkpoint directory must be provided toallow for periodic RDD checkpointing.
• Recovering from failures of the driver running the application - Metadata checkpoints are used to recover with progress information.

Note that simple streaming applications without the aforementioned stateful transformations can berun without enabling checkpointing. The recovery from driver failures will also be partial inthat case (some received but unprocessed data may be lost). This is often acceptable and many runSpark Streaming applications in this way. Support for non-Hadoop environments is expectedto improve in the future.

How to configure Checkpointing

Checkpointing can be enabled by setting a directory in a fault-tolerant,reliable file system (e.g., HDFS, S3, etc.) to which the checkpoint information will be saved.This is done by usingstreamingContext.checkpoint(checkpointDirectory). This will allow you touse the aforementioned stateful transformations. Additionally,if you want to make the application recover from driver failures, you should rewrite yourstreaming application to have the following behavior.

• When the program is being started for the first time, it will create a new StreamingContext,set up all the streams and then call start().
• When the program is being restarted after failure, it will re-create a StreamingContextfrom the checkpoint data in the checkpoint directory.

# Function to create and setup a new StreamingContextdeffunctionToCreateContext():sc=SparkContext(...)# new contextssc=StreamingContext(...)lines=ssc.socketTextStream(...)# create DStreams...ssc.checkpoint(checkpointDirectory)# set checkpoint directoryreturnssc# Get StreamingContext from checkpoint data or create a new onecontext=StreamingContext.getOrCreate(checkpointDirectory,functionToCreateContext)# Do additional setup on context that needs to be done,# irrespective of whether it is being started or restartedcontext....# Start the contextcontext.start()context.awaitTermination()

// Function to create and setup a new StreamingContextdeffunctionToCreateContext():StreamingContext={valssc=newStreamingContext(...)// new contextvallines=ssc.socketTextStream(...)// create DStreams...ssc.checkpoint(checkpointDirectory)// set checkpoint directoryssc}// Get StreamingContext from checkpoint data or create a new onevalcontext=StreamingContext.getOrCreate(checkpointDirectory,functionToCreateContext_)// Do additional setup on context that needs to be done,// irrespective of whether it is being started or restartedcontext....// Start the contextcontext.start()context.awaitTermination()

// Create a factory object that can create and setup a new JavaStreamingContextJavaStreamingContextFactorycontextFactory=newJavaStreamingContextFactory(){@OverridepublicJavaStreamingContextcreate(){JavaStreamingContextjssc=newJavaStreamingContext(...);// new contextJavaDStream<String>lines=jssc.socketTextStream(...);// create DStreams...jssc.checkpoint(checkpointDirectory);// set checkpoint directoryreturnjssc;}};// Get JavaStreamingContext from checkpoint data or create a new oneJavaStreamingContextcontext=JavaStreamingContext.getOrCreate(checkpointDirectory,contextFactory);// Do additional setup on context that needs to be done,// irrespective of whether it is being started or restartedcontext....// Start the contextcontext.start();context.awaitTermination();

In addition to usinggetOrCreate one also needs to ensure that the driver process getsrestarted automatically on failure. This can only be done by the deployment infrastructure that isused to run the application. This is further discussed in theDeployment section.

Note that checkpointing of RDDs incurs the cost of saving to reliable storage.This may cause an increase in the processing time of those batches where RDDs get checkpointed.Hence, the interval ofcheckpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing everybatch may significantly reduce operation throughput. Conversely, checkpointing too infrequentlycauses the lineage and task sizes to grow, which may have detrimental effects. For statefultransformations that require RDD checkpointing, the default interval is a multiple of thebatch interval that is at least 10 seconds. It can be set by usingdstream.checkpoint(checkpointInterval). Typically, a checkpoint interval of 5 - 10 sliding intervals of a DStream is a good setting to try.

Accumulators, Broadcast Variables, and Checkpoints

Accumulators andBroadcast variablescannot be recovered from checkpoint in Spark Streaming. If you enable checkpointing and useAccumulators orBroadcast variablesas well, you’ll have to create lazily instantiated singleton instances forAccumulators andBroadcast variablesso that they can be re-instantiated after the driver restarts on failure.This is shown in the following example.

defgetWordExcludeList(sparkContext):if("wordExcludeList"notinglobals()):globals()["wordExcludeList"]=sparkContext.broadcast(["a","b","c"])returnglobals()["wordExcludeList"]defgetDroppedWordsCounter(sparkContext):if("droppedWordsCounter"notinglobals()):globals()["droppedWordsCounter"]=sparkContext.accumulator(0)returnglobals()["droppedWordsCounter"]defecho(time,rdd):# Get or register the excludeList BroadcastexcludeList=getWordExcludeList(rdd.context)# Get or register the droppedWordsCounter AccumulatordroppedWordsCounter=getDroppedWordsCounter(rdd.context)# Use excludeList to drop words and use droppedWordsCounter to count themdeffilterFunc(wordCount):ifwordCount[0]inexcludeList.value:droppedWordsCounter.add(wordCount[1])Falseelse:Truecounts="Counts at time %s %s"%(time,rdd.filter(filterFunc).collect())wordCounts.foreachRDD(echo)

objectWordExcludeList{@volatileprivatevarinstance:Broadcast[Seq[String]]=nulldefgetInstance(sc:SparkContext):Broadcast[Seq[String]]={if(instance==null){synchronized{if(instance==null){valwordExcludeList=Seq("a","b","c")instance=sc.broadcast(wordExcludeList)}}}instance}}objectDroppedWordsCounter{@volatileprivatevarinstance:LongAccumulator=nulldefgetInstance(sc:SparkContext):LongAccumulator={if(instance==null){synchronized{if(instance==null){instance=sc.longAccumulator("DroppedWordsCounter")}}}instance}}wordCounts.foreachRDD{(rdd:RDD[(String,Int)],time:Time)=>// Get or register the excludeList BroadcastvalexcludeList=WordExcludeList.getInstance(rdd.sparkContext)// Get or register the droppedWordsCounter AccumulatorvaldroppedWordsCounter=DroppedWordsCounter.getInstance(rdd.sparkContext)// Use excludeList to drop words and use droppedWordsCounter to count themvalcounts=rdd.filter{case(word,count)=>if(excludeList.value.contains(word)){droppedWordsCounter.add(count)false}else{true}}.collect().mkString("[",", ","]")valoutput="Counts at time "+time+" "+counts})

classJavaWordExcludeList{privatestaticvolatileBroadcast<List<String>>instance=null;publicstaticBroadcast<List<String>>getInstance(JavaSparkContextjsc){if(instance==null){synchronized(JavaWordExcludeList.class){if(instance==null){List<String>wordExcludeList=Arrays.asList("a","b","c");instance=jsc.broadcast(wordExcludeList);}}}returninstance;}}classJavaDroppedWordsCounter{privatestaticvolatileLongAccumulatorinstance=null;publicstaticLongAccumulatorgetInstance(JavaSparkContextjsc){if(instance==null){synchronized(JavaDroppedWordsCounter.class){if(instance==null){instance=jsc.sc().longAccumulator("DroppedWordsCounter");}}}returninstance;}}wordCounts.foreachRDD((rdd,time)->{// Get or register the excludeList BroadcastBroadcast<List<String>>excludeList=JavaWordExcludeList.getInstance(newJavaSparkContext(rdd.context()));// Get or register the droppedWordsCounter AccumulatorLongAccumulatordroppedWordsCounter=JavaDroppedWordsCounter.getInstance(newJavaSparkContext(rdd.context()));// Use excludeList to drop words and use droppedWordsCounter to count themStringcounts=rdd.filter(wordCount->{if(excludeList.value().contains(wordCount._1())){droppedWordsCounter.add(wordCount._2());returnfalse;}else{returntrue;}}).collect().toString();Stringoutput="Counts at time "+time+" "+counts;}

Deploying Applications

This section discusses the steps to deploy a Spark Streaming application.

Requirements

To run Spark Streaming applications, you need to have the following.

• Cluster with a cluster manager - This is the general requirement of any Spark application,and discussed in detail in thedeployment guide.

• Package the application JAR - You have to compile your streaming application into a JAR.If you are usingspark-submit to start theapplication, then you will not need to provide Spark and Spark Streaming in the JAR. However,if your application usesadvanced sources (e.g. Kafka),then you will have to package the extra artifact they link to, along with their dependencies,in the JAR that is used to deploy the application. For example, an application usingKafkaUtilswill have to includespark-streaming-kafka-0-10_2.13 and all itstransitive dependencies in the application JAR.

• Configuring sufficient memory for the executors - Since the received data must be stored inmemory, the executors must be configured with sufficient memory to hold the received data. Notethat if you are doing 10 minute window operations, the system has to keep at least last 10 minutesof data in memory. So the memory requirements for the application depends on the operationsused in it.

• Configuring checkpointing - If the stream application requires it, then a directory in theHadoop API compatible fault-tolerant storage (e.g. HDFS, S3, etc.) must be configured as thecheckpoint directory and the streaming application written in a way that checkpointinformation can be used for failure recovery. See thecheckpointing sectionfor more details.

• Configuring automatic restart of the application driver - To automatically recover from adriver failure, the deployment infrastructure that isused to run the streaming application must monitor the driver process and relaunch the driverif it fails. Differentcluster managershave different tools to achieve this.
  • Spark Standalone - A Spark application driver can be submitted to run within the SparkStandalone cluster (seecluster deploy mode), that is, theapplication driver itself runs on one of the worker nodes. Furthermore, theStandalone cluster manager can be instructed tosupervise the driver,and relaunch it if the driver fails either due to non-zero exit code,or due to failure of the node running the driver. Seecluster mode andsupervise in theSpark Standalone guide for more details.
  • YARN - Yarn supports a similar mechanism for automatically restarting an application.Please refer to YARN documentation for more details.

• Configuring write-ahead logs - Since Spark 1.2,we have introducedwrite-ahead logs for achieving strongfault-tolerance guarantees. If enabled, all the data received from a receiver gets written intoa write-ahead log in the configuration checkpoint directory. This prevents data loss on driverrecovery, thus ensuring zero data loss (discussed in detail in theFault-tolerance Semantics section). This can be enabled by settingtheconfiguration parameterspark.streaming.receiver.writeAheadLog.enable totrue. However, these stronger semantics maycome at the cost of the receiving throughput of individual receivers. This can be corrected byrunningmore receivers in parallelto increase aggregate throughput. Additionally, it is recommended that the replication of thereceived data within Spark be disabled when the write-ahead log is enabled as the log is alreadystored in a replicated storage system. This can be done by setting the storage level for theinput stream toStorageLevel.MEMORY_AND_DISK_SER. While using S3 (or any file system thatdoes not support flushing) forwrite-ahead logs, please remember to enablespark.streaming.driver.writeAheadLog.closeFileAfterWrite andspark.streaming.receiver.writeAheadLog.closeFileAfterWrite. SeeSpark Streaming Configuration for more details.Note that Spark will not encrypt data written to the write-ahead log when I/O encryption isenabled. If encryption of the write-ahead log data is desired, it should be stored in a filesystem that supports encryption natively.

• Setting the max receiving rate - If the cluster resources are not large enough for the streamingapplication to process data as fast as it is being received, the receivers can be rate limitedby setting a maximum rate limit in terms of records / sec.See theconfiguration parametersspark.streaming.receiver.maxRate for receivers andspark.streaming.kafka.maxRatePerPartitionfor Direct Kafka approach. In Spark 1.5, we have introduced a feature calledbackpressure thateliminates the need to set this rate limit, as Spark Streaming automatically figures out therate limits and dynamically adjusts them if the processing conditions change. This backpressurecan be enabled by setting theconfiguration parameterspark.streaming.backpressure.enabled totrue.

Upgrading Application Code

If a running Spark Streaming application needs to be upgraded with newapplication code, then there are two possible mechanisms.

• The upgraded Spark Streaming application is started and run in parallel to the existing application.Once the new one (receiving the same data as the old one) has been warmed up and is readyfor prime time, the old one can be brought down. Note that this can be done for data sources that supportsending the data to two destinations (i.e., the earlier and upgraded applications).

• The existing application is shutdown gracefully (seeStreamingContext.stop(...)orJavaStreamingContext.stop(...)for graceful shutdown options) which ensure data that has been received is completelyprocessed before shutdown. Then theupgraded application can be started, which will start processing from the same point where the earlierapplication left off. Note that this can be done only with input sources that support source-side buffering(like Kafka) as data needs to be buffered while the previous application was down andthe upgraded application is not yet up. And restarting from earlier checkpointinformation of pre-upgrade code cannot be done. The checkpoint information essentiallycontains serialized Scala/Java/Python objects and trying to deserialize objects with new,modified classes may lead to errors. In this case, either start the upgraded app with a differentcheckpoint directory, or delete the previous checkpoint directory.

Monitoring Applications

Beyond Spark’smonitoring capabilities, there are additional capabilitiesspecific to Spark Streaming. When a StreamingContext is used, theSpark web UI showsan additionalStreaming tab which shows statistics about running receivers (whetherreceivers are active, number of records received, receiver error, etc.)and completed batches (batch processing times, queueing delays, etc.). This can be used tomonitor the progress of the streaming application.

The following two metrics in web UI are particularly important:

• Processing Time - The time to process each batch of data.
• Scheduling Delay - the time a batch waits in a queue for the processing of previous batchesto finish.

If the batch processing time is consistently more than the batch interval and/or the queueingdelay keeps increasing, then it indicates that the system isnot able to process the batches as fast they are being generated and is falling behind.In that case, considerreducing the batch processing time.

The progress of a Spark Streaming program can also be monitored using theStreamingListener interface,which allows you to get receiver status and processing times. Note that this is a developer APIand it is likely to be improved upon (i.e., more information reported) in the future.

Performance Tuning

Getting the best performance out of a Spark Streaming application on a cluster requires a bit oftuning. This section explains a number of the parameters and configurations that can be tuned toimprove the performance of your application. At a high level, you need to consider two things:

1. Reducing the processing time of each batch of data by efficiently using cluster resources.

2. Setting the right batch size such that the batches of data can be processed as fast as theyare received (that is, data processing keeps up with the data ingestion).

Reducing the Batch Processing Times

There are a number of optimizations that can be done in Spark to minimize the processing time ofeach batch. These have been discussed in detail in theTuning Guide. This sectionhighlights some of the most important ones.

Level of Parallelism in Data Receiving

Receiving data over the network (like Kafka, socket, etc.) requires the data to be deserializedand stored in Spark. If the data receiving becomes a bottleneck in the system, then considerparallelizing the data receiving. Note that each input DStreamcreates a single receiver (running on a worker machine) that receives a single stream of data.Receiving multiple data streams can therefore be achieved by creating multiple input DStreamsand configuring them to receive different partitions of the data stream from the source(s).For example, a single Kafka input DStream receiving two topics of data can be split into twoKafka input streams, each receiving only one topic. This would run two receivers,allowing data to be received in parallel, thus increasing overall throughput. These multipleDStreams can be unioned together to create a single DStream. Then the transformations that werebeing applied on a single input DStream can be applied on the unified stream. This is done as follows.

numStreams=5kafkaStreams=[KafkaUtils.createStream(...)for_inrange(numStreams)]unifiedStream=streamingContext.union(*kafkaStreams)unifiedStream.pprint()

valnumStreams=5valkafkaStreams=(1tonumStreams).map{i=>KafkaUtils.createStream(...)}valunifiedStream=streamingContext.union(kafkaStreams)unifiedStream.print()

intnumStreams=5;List<JavaPairDStream<String,String>>kafkaStreams=newArrayList<>(numStreams);for(inti=0;i<numStreams;i++){kafkaStreams.add(KafkaUtils.createStream(...));}JavaPairDStream<String,String>unifiedStream=streamingContext.union(kafkaStreams.get(0),kafkaStreams.subList(1,kafkaStreams.size()));unifiedStream.print();

Another parameter that should be considered is the receiver’s block interval,which is determined by theconfiguration parameterspark.streaming.blockInterval. For most receivers, the received data is coalesced together intoblocks of data before storing inside Spark’s memory. The number of blocks in each batchdetermines the number of tasks that will be used to processthe received data in a map-like transformation. The number of tasks per receiver per batch will beapproximately (batch interval / block interval). For example, a block interval of 200 ms willcreate 10 tasks per 2 second batches. If the number of tasks is too low (that is, less than the numberof cores per machine), then it will be inefficient as all available cores will not be used toprocess the data. To increase the number of tasks for a given batch interval, reduce theblock interval. However, the recommended minimum value of block interval is about 50 ms,below which the task launching overheads may be a problem.

An alternative to receiving data with multiple input streams / receivers is to explicitly repartitionthe input data stream (usinginputStream.repartition(<number of partitions>)).This distributes the received batches of data across the specified number of machines in the clusterbefore further processing.

For direct stream, please refer toSpark Streaming + Kafka Integration Guide

Level of Parallelism in Data Processing

Cluster resources can be under-utilized if the number of parallel tasks used in any stage of thecomputation is not high enough. For example, for distributed reduce operations likereduceByKeyandreduceByKeyAndWindow, the default number of parallel tasks is controlled bythespark.default.parallelismconfiguration property. Youcan pass the level of parallelism as an argument (seePairDStreamFunctionsdocumentation), or set thespark.default.parallelismconfiguration property to change the default.

Data Serialization

The overheads of data serialization can be reduced by tuning the serialization formats. In the case of streaming, there are two types of data that are being serialized.

• Input data: By default, the input data received through Receivers is stored in the executors’ memory withStorageLevel.MEMORY_AND_DISK_SER_2. That is, the data is serialized into bytes to reduce GC overheads, and replicated for tolerating executor failures. Also, the data is kept first in memory, and spilled over to disk only if the memory is insufficient to hold all of the input data necessary for the streaming computation. This serialization obviously has overheads – the receiver must deserialize the received data and re-serialize it using Spark’s serialization format.

• Persisted RDDs generated by Streaming Operations: RDDs generated by streaming computations may be persisted in memory. For example, window operations persist data in memory as they would be processed multiple times. However, unlike the Spark Core default ofStorageLevel.MEMORY_ONLY, persisted RDDs generated by streaming computations are persisted withStorageLevel.MEMORY_ONLY_SER (i.e. serialized) by default to minimize GC overheads.

In both cases, using Kryo serialization can reduce both CPU and memory overheads. See theSpark Tuning Guide for more details. For Kryo, consider registering custom classes, and disabling object reference tracking (see Kryo-related configurations in theConfiguration Guide).

In specific cases where the amount of data that needs to be retained for the streaming application is not large, it may be feasible to persist data (both types) as deserialized objects without incurring excessive GC overheads. For example, if you are using batch intervals of a few seconds and no window operations, then you can try disabling serialization in persisted data by explicitly setting the storage level accordingly. This would reduce the CPU overheads due to serialization, potentially improving performance without too much GC overheads.

Setting the Right Batch Interval

For a Spark Streaming application running on a cluster to be stable, the system should be able toprocess data as fast as it is being received. In other words, batches of data should be processedas fast as they are being generated. Whether this is true for an application can be found bymonitoring the processing times in the streaming web UI, where the batchprocessing time should be less than the batch interval.

Depending on the nature of the streamingcomputation, the batch interval used may have significant impact on the data rates that can besustained by the application on a fixed set of cluster resources. For example, let usconsider the earlier WordCountNetwork example. For a particular data rate, the system may be ableto keep up with reporting word counts every 2 seconds (i.e., batch interval of 2 seconds), but notevery 500 milliseconds. So the batch interval needs to be set such that the expected data rate inproduction can be sustained.

A good approach to figure out the right batch size for your application is to test it with aconservative batch interval (say, 5-10 seconds) and a low data rate. To verify whether the systemis able to keep up with the data rate, you can check the value of the end-to-end delay experiencedby each processed batch (either look for “Total delay” in Spark driver log4j logs, or use theStreamingListenerinterface).If the delay is maintained to be comparable to the batch size, then the system is stable. Otherwise,if the delay is continuously increasing, it means that the system is unable to keep up and it istherefore unstable. Once you have an idea of a stable configuration, you can try increasing thedata rate and/or reducing the batch size. Note that a momentary increase in the delay due totemporary data rate increases may be fine as long as the delay reduces back to a low value(i.e., less than batch size).

Memory Tuning

Tuning the memory usage and GC behavior of Spark applications has been discussed in great detailin theTuning Guide. It is strongly recommended that you read that. In this section, we discuss a few tuning parameters specifically in the context of Spark Streaming applications.

The amount of cluster memory required by a Spark Streaming application depends heavily on the type of transformations used. For example, if you want to use a window operation on the last 10 minutes of data, then your cluster should have sufficient memory to hold 10 minutes worth of data in memory. Or if you want to useupdateStateByKey with a large number of keys, then the necessary memory will be high. On the contrary, if you want to do a simple map-filter-store operation, then the necessary memory will be low.

In general, since the data received through receivers is stored with StorageLevel.MEMORY_AND_DISK_SER_2, the data that does not fit in memory will spill over to the disk. This may reduce the performance of the streaming application, and hence it is advised to provide sufficient memory as required by your streaming application. Its best to try and see the memory usage on a small scale and estimate accordingly.

Another aspect of memory tuning is garbage collection. For a streaming application that requires low latency, it is undesirable to have large pauses caused by JVM Garbage Collection.

There are a few parameters that can help you tune the memory usage and GC overheads:

• Persistence Level of DStreams: As mentioned earlier in theData Serialization section, the input data and RDDs are by default persisted as serialized bytes. This reduces both the memory usage and GC overheads, compared to deserialized persistence. Enabling Kryo serialization further reduces serialized sizes and memory usage. Further reduction in memory usage can be achieved with compression (see the Spark configurationspark.rdd.compress), at the cost of CPU time.

• Clearing old data: By default, all input data and persisted RDDs generated by DStream transformations are automatically cleared. Spark Streaming decides when to clear the data based on the transformations that are used. For example, if you are using a window operation of 10 minutes, then Spark Streaming will keep around the last 10 minutes of data, and actively throw away older data.Data can be retained for a longer duration (e.g. interactively querying older data) by settingstreamingContext.remember.

• Other tips: To further reduce GC overheads, here are some more tips to try.

  • Persist RDDs using theOFF_HEAP storage level. See more detail in theSpark Programming Guide.
  • Use more executors with smaller heap sizes. This will reduce the GC pressure within each JVM heap.

Important points to remember:

• A DStream is associated with a single receiver. For attaining read parallelism multiple receivers i.e. multiple DStreams need to be created. A receiver is run within an executor. It occupies one core. Ensure that there are enough cores for processing after receiver slots are booked i.e.spark.cores.max should take the receiver slots into account. The receivers are allocated to executors in a round robin fashion.

• When data is received from a stream source, the receiver creates blocks of data. A new block of data is generated every blockInterval milliseconds. N blocks of data are created during the batchInterval where N = batchInterval/blockInterval. These blocks are distributed by the BlockManager of the current executor to the block managers of other executors. After that, the Network Input Tracker running on the driver is informed about the block locations for further processing.

• An RDD is created on the driver for the blocks created during the batchInterval. The blocks generated during the batchInterval are partitions of the RDD. Each partition is a task in spark. blockInterval== batchinterval would mean that a single partition is created and probably it is processed locally.

• The map tasks on the blocks are processed in the executors (one that received the block, and another where the block was replicated) that has the blocks irrespective of block interval, unless non-local scheduling kicks in.Having a bigger blockinterval means bigger blocks. A high value ofspark.locality.wait increases the chance of processing a block on the local node. A balance needs to be found out between these two parameters to ensure that the bigger blocks are processed locally.

• Instead of relying on batchInterval and blockInterval, you can define the number of partitions by callinginputDstream.repartition(n). This reshuffles the data in RDD randomly to create n number of partitions. Yes, for greater parallelism. Though comes at the cost of a shuffle. An RDD’s processing is scheduled by the driver’s jobscheduler as a job. At a given point of time only one job is active. So, if one job is executing the other jobs are queued.

• If you have two dstreams there will be two RDDs formed and there will be two jobs created which will be scheduled one after the another. To avoid this, you can union two dstreams. This will ensure that a single unionRDD is formed for the two RDDs of the dstreams. This unionRDD is then considered as a single job. However, the partitioning of the RDDs is not impacted.

• If the batch processing time is more than batchinterval then obviously the receiver’s memory will start filling up and will end up in throwing exceptions (most probably BlockNotFoundException). Currently, there is no way to pause the receiver. Using SparkConf configurationspark.streaming.receiver.maxRate, rate of receiver can be limited.

Fault-tolerance Semantics

In this section, we will discuss the behavior of Spark Streaming applications in the eventof failures.

Background

To understand the semantics provided by Spark Streaming, let us remember the basic fault-tolerance semantics of Spark’s RDDs.

1. An RDD is an immutable, deterministically re-computable, distributed dataset. Each RDDremembers the lineage of deterministic operations that were used on a fault-tolerant inputdataset to create it.
2. If any partition of an RDD is lost due to a worker node failure, then that partition can bere-computed from the original fault-tolerant dataset using the lineage of operations.
3. Assuming that all of the RDD transformations are deterministic, the data in the final transformedRDD will always be the same irrespective of failures in the Spark cluster.

Spark operates on data in fault-tolerant file systems like HDFS or S3. Hence,all of the RDDs generated from the fault-tolerant data are also fault-tolerant. However, this is notthe case for Spark Streaming as the data in most cases is received over the network (except whenfileStream is used). To achieve the same fault-tolerance properties for all of the generated RDDs,the received data is replicated among multiple Spark executors in worker nodes in the cluster(default replication factor is 2). This leads to two kinds of data in thesystem that need to be recovered in the event of failures:

1. Data received and replicated - This data survives failure of a single worker node as a copy of it exists on one of the other nodes.
2. Data received but buffered for replication - Since this is not replicated,the only way to recover this data is to get it again from the source.

Furthermore, there are two kinds of failures that we should be concerned about:

1. Failure of a Worker Node - Any of the worker nodes running executors can fail,and all in-memory data on those nodes will be lost. If any receivers were running on failednodes, then their buffered data will be lost.
2. Failure of the Driver Node - If the driver node running the Spark Streaming applicationfails, then obviously the SparkContext is lost, and all executors with their in-memorydata are lost.

With this basic knowledge, let us understand the fault-tolerance semantics of Spark Streaming.

Definitions

The semantics of streaming systems are often captured in terms of how many times each record can be processed by the system. There are three types of guarantees that a system can provide under all possible operating conditions (despite failures, etc.)

1. At most once: Each record will be either processed once or not processed at all.
2. At least once: Each record will be processed one or more times. This is stronger thanat-most once as it ensures that no data will be lost. But there may be duplicates.
3. Exactly once: Each record will be processed exactly once - no data will be lost and no data will be processed multiple times. This is obviously the strongest guarantee of the three.

Basic Semantics

In any stream processing system, broadly speaking, there are three steps in processing the data.

1. Receiving the data: The data is received from sources using Receivers or otherwise.

2. Transforming the data: The received data is transformed using DStream and RDD transformations.

3. Pushing out the data: The final transformed data is pushed out to external systems like file systems, databases, dashboards, etc.

If a streaming application has to achieve end-to-end exactly-once guarantees, then each step has to provide an exactly-once guarantee. That is, each record must be received exactly once, transformed exactly once, and pushed to downstream systems exactly once. Let’s understand the semantics of these steps in the context of Spark Streaming.

1. Receiving the data: Different input sources provide different guarantees. This is discussed in detail in the next subsection.

2. Transforming the data: All data that has been received will be processedexactly once, thanks to the guarantees that RDDs provide. Even if there are failures, as long as the received input data is accessible, the final transformed RDDs will always have the same contents.

3. Pushing out the data: Output operations by default ensureat-least once semantics because it depends on the type of output operation (idempotent, or not) and the semantics of the downstream system (supports transactions or not). But users can implement their own transaction mechanisms to achieveexactly-once semantics. This is discussed in more details later in the section.

Semantics of Received Data

Different input sources provide different guarantees, ranging fromat-least once toexactly once. Read for more details.

With Files

If all of the input data is already present in a fault-tolerant file system likeHDFS, Spark Streaming can always recover from any failure and process all of the data. This givesexactly-once semantics, meaning all of the data will be processed exactly once no matter what fails.

With Receiver-based Sources

For input sources based on receivers, the fault-tolerance semantics depend on both the failurescenario and the type of receiver.As we discussedearlier, there are two types of receivers:

1. Reliable Receiver - These receivers acknowledge reliable sources only after ensuring that the received data has been replicated. If such a receiver fails, the source will not receive acknowledgment for the buffered (unreplicated) data. Therefore, if the receiver is restarted, the source will resend the data, and no data will be lost due to the failure.
2. Unreliable Receiver - Such receivers donot send acknowledgment and thereforecan lose data when they fail due to worker or driver failures.

Depending on what type of receivers are used we achieve the following semantics.If a worker node fails, then there is no data loss with reliable receivers. With unreliablereceivers, data received but not replicated can get lost. If the driver node fails,then besides these losses, all of the past data that was received and replicated in memory will belost. This will affect the results of the stateful transformations.

To avoid this loss of past received data, Spark 1.2 introducedwriteahead logs which save the received data to fault-tolerant storage. With thewrite-ahead logsenabled and reliable receivers, there is zero data loss. In terms of semantics, it provides an at-least once guarantee.

The following table summarizes the semantics under failures:

With Kafka Direct API

In Spark 1.3, we have introduced a new Kafka Direct API, which can ensure that all the Kafka data is received by Spark Streaming exactly once. Along with this, if you implement exactly-once output operation, you can achieve end-to-end exactly-once guarantees. This approach is further discussed in theKafka Integration Guide.

Semantics of output operations

Output operations (likeforeachRDD) haveat-least once semantics, that is,the transformed data may get written to an external entity more than once inthe event of a worker failure. While this is acceptable for saving to file systems using thesaveAs***Files operations (as the file will simply get overwritten with the same data),additional effort may be necessary to achieve exactly-once semantics. There are two approaches.

• Idempotent updates: Multiple attempts always write the same data. For example,saveAs***Files always writes the same data to the generated files.

• Transactional updates: All updates are made transactionally so that updates are made exactly once atomically. One way to do this would be the following.

  • Use the batch time (available inforeachRDD) and the partition index of the RDD to create an identifier. This identifier uniquely identifies a blob data in the streaming application.

  • Update external system with this blob transactionally (that is, exactly once, atomically) using the identifier. That is, if the identifier is not already committed, commit the partition data and the identifier atomically. Else, if this was already committed, skip the update.

    dstream.foreachRDD { (rdd, time) =>  rdd.foreachPartition { partitionIterator =>    val partitionId = TaskContext.get.partitionId()    val uniqueId = generateUniqueId(time.milliseconds, partitionId)    // use this uniqueId to transactionally commit the data in partitionIterator  }}

Where to Go from Here

• Additional guides
  • Kafka Integration Guide
  • Kinesis Integration Guide
  • Custom Receiver Guide
• Third-party DStream data sources can be found inThird Party Projects
• API documentation
  • Python docs
    • StreamingContext andDStream
  • Scala docs
    • StreamingContext andDStream
    • KafkaUtils,KinesisUtils
  • Java docs
    • JavaStreamingContext,JavaDStream andJavaPairDStream
    • KafkaUtils,KinesisUtils
• More examples inPythonandScalaandJava
• Paper andvideo describing Spark Streaming.