> For the complete documentation index, see [llms.txt](https://claudiu-stanciu.gitbook.io/hadoop-tools/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://claudiu-stanciu.gitbook.io/hadoop-tools/transformation/pig.md).

# Pig

Your general purpose analytics pig-tool

## Launch CLI

```
pig
```

## Change execution engine

### At CLI launch

```
pig -x <mode>
```

### In script

```
set exectype=tez;
```

mode: local, mapreduce, tez, tez\_local

## Execute Pig script

```
pig script.pig
```

## Load data from HDFS

```
<var> = load 'path/to/file';
```

## Load data from HDFS with schema

```
<var> = load 'path/to/file' as (field1:chararray, field2:int, ...);
```

## Load data from Hive

* start pig with hcatalog

```
$ pig -useHCatalog

var = load '<db>.<table>' using org.apache.hive.hcatalog.pig.HCatLoader();
```

## Group data

```
<grouped_var> = group (<var> by <col>)/ (all), group <var> by <col>/all
```

## Transform schema

```
<var> = foreach <raw_var> generate <col1>, <col2> ...;
```

## Filter data

```
<var> = filter <raw_var> by <filter_query>
```

## Order

```
<var> = order <var> by <col> (asc/desc), <col> ...
```

## Limit the number of rows

```
<var> = limit <var> <size>
```

## Split

```
split <var> into <good_var> if <query>, <bad_var> otherwise;
```

## Remove duplicates

```
<var> = distinct <var>
```

## Inner join

```
<varC> = join <varA> by <col1>, <varB> by <col2>
```

## Right/Left/Full Outer join

```
<varC> = join <varA> by <col1> right/left/full (outer), <varB> by >col2>
```

## Cross join

```
<varC> = cross <varA>, <varB>;
```

## Join options

```
join .... left , .... using 'replicated' | skewed | merge
```

replicated dosn't work on tez. use mapreduce mdoe

## Dump data on console

```
dump <var>;
```

## Store data into HDFS

```
store <var> into <path/to/file> ( using PigStorage(',') )
```

## Store data into Hive

```
store <var> into 'table' using org.apache.hive.hcatalog.pig.HCatStorer()
```

## Specify nb reducers

Add 'parallel ' to any reducer operator: group, distinct, order, join

```
<reducer ops> parallel <nb>
```

## Debug

```
explain <query>

describe <query>

illustrate <query>

set debug on
```

## Register a UDF Jar

Register the jar (eg: PiggyBank.jar).

```
register <jar>
```

## Use an UDF

Like any other function, just it might require the full package name

## Create an alias for function

```
define <alias> <udf/macro>
```

## Import a macro/UDF from another script

```
import <>
```