Commitf5a4329

committed

Initial Commit

1 parent5172e5a commitf5a4329Copy full SHA for f5a4329

File tree

-3

lines changed

-3

lines changed

Lines changed: 7 additions & 3 deletions

Original file line number	Diff line number	Diff line change
`@@ -31,14 +31,18 @@ def parse_gender(gender):`
`31`	`31`	`.option("inferSchema","true") \`
`32`	`32`	`.csv("data/survey.csv")`
`33`	`33`
`34`		`-survey_df.show()`
	`34`	`+survey_df.show(10)`
`35`	`35`
`36`	`36`	`parse_gender_udf=udf(parse_gender,returnType=StringType())`
	`37`	`+logger.info("Catalog Entry:")`
`37`	`38`	`[logger.info(r)forrinspark.catalog.listFunctions()if"parse_gender"inr.name]`
	`39`	`+`
`38`	`40`	`survey_df2=survey_df.withColumn("Gender",parse_gender_udf("Gender"))`
`39`		`-survey_df2.show()`
	`41`	`+survey_df2.show(10)`
`40`	`42`
`41`	`43`	`spark.udf.register("parse_gender_udf",parse_gender,StringType())`
	`44`	`+logger.info("Catalog Entry:")`
`42`	`45`	`[logger.info(r)forrinspark.catalog.listFunctions()if"parse_gender"inr.name]`
	`46`	`+`
`43`	`47`	`survey_df3=survey_df.withColumn("Gender",expr("parse_gender_udf(Gender)"))`
`44`		`-survey_df3.show()`
	`48`	`+survey_df3.show(10)`

Comments

(0)