> For the complete documentation index, see [llms.txt](https://claudiu-stanciu.gitbook.io/hadoop-tools/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://claudiu-stanciu.gitbook.io/hadoop-tools/transformation/spark.md).

# Spark - Scala

## Context

```
spark = Spark SQL Context
sc = SparkContext
```

## Basic blocks

* RDD = resilient distributed dataset = collection of distributed data elements
* DS = dataset = collection of distributed data elements
* DF = dataframe = dataset with named columns (schema)

## RDD

<https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html>

### Creation

* declare data

```
val data = Array(1,3,4,5)
val rddData = sc.parallelize(data)
```

* from file

```
val dataFS = sc.textFile("file:///...")
val dataHDFS = sc.textFile("hdfs:///..")
```

* sql?

### Operations

Operations are lazy on spark, and the more the merrier, so that the optimizer can better tune the DAG

#### Transformations

Create a new RDD by applying an operation. Doesn't trigger an execution

* map(f: (A) ⇒B) = apply the function for each item

```
rdd.map(_ => _ + 2)
```

* flatMap(f: (A) ⇒B) = flatten the data structure(tuple2 or whatever) and apply function for each item

```
rdd.flatMap(_ => _ + 3)
```

* filter(f: (A) ⇒Boolean)  = keep only the data for which the lambda returns true

```
rdd.filter(_ => _ % 2 == 0) // keep
```

* **Actions**

Return the value to the driver after the final computation on the data. Triggers an execution

## DF

<https://spark.apache.org/docs/2.2.0/sql-programming-guide.html>