Commit5f9071a

Pedro Bernardo

committed

Added advanced/accumulator/*.py

1 parent3aeb5d8 commit5f9071aCopy full SHA for 5f9071a

File tree

+53

-0

lines changed

+53

-0

lines changed

Lines changed: 25 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,25 @@`
	`1`	`+frompysparkimportSparkContext`
	`2`	`+fromcommons.UtilsimportUtils`
	`3`	`+`
	`4`	`+deffilterResponseFromCanada(response,total,missingSalaryMidPoint):`
	`5`	`+splits=Utils.COMMA_DELIMITER.split(response)`
	`6`	`+total.add(1)`
	`7`	`+ifnotsplits[14]:`
	`8`	`+missingSalaryMidPoint.add(1)`
	`9`	`+returnsplits[2]=="Canada"`
	`10`	`+`
	`11`	`+if__name__=="__main__":`
	`12`	`+sc=SparkContext("local","StackOverFlowSurvey")`
	`13`	`+sc.setLogLevel("ERROR")`
	`14`	`+`
	`15`	`+total=sc.accumulator(0)`
	`16`	`+missingSalaryMidPoint=sc.accumulator(0)`
	`17`	`+`
	`18`	`+responseRDD=sc.textFile("in/2016-stack-overflow-survey-responses.csv")`
	`19`	`+`
	`20`	`+responseFromCanada=responseRDD.filter(lambdaresponse: \`
	`21`	`+filterResponseFromCanada(response,total,missingSalaryMidPoint))`
	`22`	`+`
	`23`	`+print("Count of responses from Canada: {}".format(responseFromCanada.count()))`
	`24`	`+print("Total count of responses: {}".format(total.value))`
	`25`	`+print("Count of responses missing salary middle point: {}".format(missingSalaryMidPoint.value))`

Lines changed: 28 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,28 @@`
	`1`	`+frompysparkimportSparkContext`
	`2`	`+fromcommons.UtilsimportUtils`
	`3`	`+`
	`4`	`+deffilterResponseFromCanada(response,total,missingSalaryMidPoint,processedBytes):`
	`5`	`+processedBytes.add(len(response.encode('utf-8')))`
	`6`	`+splits=Utils.COMMA_DELIMITER.split(response)`
	`7`	`+total.add(1)`
	`8`	`+ifnotsplits[14]:`
	`9`	`+missingSalaryMidPoint.add(1)`
	`10`	`+returnsplits[2]=="Canada"`
	`11`	`+`
	`12`	`+if__name__=="__main__":`
	`13`	`+sc=SparkContext("local","StackOverFlowSurvey")`
	`14`	`+sc.setLogLevel("ERROR")`
	`15`	`+`
	`16`	`+total=sc.accumulator(0)`
	`17`	`+missingSalaryMidPoint=sc.accumulator(0)`
	`18`	`+processedBytes=sc.accumulator(0)`
	`19`	`+`
	`20`	`+responseRDD=sc.textFile("in/2016-stack-overflow-survey-responses.csv")`
	`21`	`+`
	`22`	`+responseFromCanada=responseRDD.filter(lambdaresponse: \`
	`23`	`+filterResponseFromCanada(response,total,missingSalaryMidPoint,processedBytes))`
	`24`	`+`
	`25`	`+print("Count of responses from Canada: {}".format(responseFromCanada.count()))`
	`26`	`+print("Number of bytes processed: {}".format(processedBytes.value))`
	`27`	`+print("Total count of responses: {}".format(total.value))`
	`28`	`+print("Count of responses missing salary middle point: {}".format(missingSalaryMidPoint.value))`

Comments

(0)