Loading Data
Viewing Contents of a Parquet Folder
%fs ls /mnt/training/crime-data-20016
Reading Parquet
df = spark.read.parquet ( "/mnt/training/cimr-data-2016/Crime-Data-Boston-2016.parquet" )
Viewing Results
Text Based (Generic)
show(df)
Databricks Native Viewer
display(df)
Manipulating Dataframes
Select
df.select("*","firstName","last_name")
Filter
df.select("*").filter("firstName='Brian'").filter('lastName='Popp')
Combining Dataframes
Union
homicidesBostonDF = homicidesNewYorkDF.union ( homicidesBostonDF )
