Commit4d3e13f

committed

Initial Commit

1 parentafa23cc commit4d3e13fCopy full SHA for 4d3e13f

File tree

+941036

-0

lines changed

+941036

-0

lines changed

Lines changed: 23 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,23 @@`
	`1`	`+frompyspark.sqlimportSparkSession`
	`2`	`+`
	`3`	`+fromlib.loggerimportLog4j`
	`4`	`+`
	`5`	`+if__name__=="__main__":`
	`6`	`+spark=SparkSession \`
	`7`	`+ .builder \`
	`8`	`+ .appName("Shuffle Join Demo") \`
	`9`	`+ .master("local[3]") \`
	`10`	`+ .getOrCreate()`
	`11`	`+`
	`12`	`+logger=Log4j(spark)`
	`13`	`+`
	`14`	`+flight_time_df1=spark.read.json("data/d1/")`
	`15`	`+flight_time_df2=spark.read.json("data/d2/")`
	`16`	`+`
	`17`	`+spark.conf.set("spark.sql.shuffle.partitions",3)`
	`18`	`+`
	`19`	`+join_expr=flight_time_df1.id==flight_time_df2.id`
	`20`	`+join_df=flight_time_df1.join(flight_time_df2,join_expr,"inner")`
	`21`	`+`
	`22`	`+join_df.foreach(lambdaf:None)`
	`23`	`+input("press a key to stop...")`

Lines changed: 172660 additions & 0 deletions

Large diffs are not rendered by default.