Getting Started w/ Python, Spark, and Databricks

Loading Data

Viewing Contents of a Parquet Folder

%fs ls /mnt/training/crime-data-20016

Reading Parquet

df = spark.read.parquet ( "/mnt/training/cimr-data-2016/Crime-Data-Boston-2016.parquet" )

Viewing Results

Text Based (Generic)

show(df)

Databricks Native Viewer

display(df)

Manipulating Dataframes

Select

df.select("*","firstName","last_name")

Filter

df.select("*").filter("firstName='Brian'").filter('lastName='Popp')

== Combining Dataframes ==

=== Union ===
<pre>homicidesBostonDF = homicidesNewYorkDF.union ( homicidesBostonDF )