> For the complete documentation index, see [llms.txt](https://claudiu-stanciu.gitbook.io/software-craft/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://claudiu-stanciu.gitbook.io/software-craft/big-data/introduction.md).

# Introduction

## Defined by the 3 V's

* volume (data size) = TB/PB of data
* velocity (data/event speed)= GB/sec, 1 mil events/sec
* variety (data formats)
  * structured: tabular (SQL, CSV, Excel)
  * semi-structured: JSON, XML, binary formats
  * unstructured: text, pdf, images, videos, binary blobs

## Storing data

* lakehouse
  * data lake ++
  * data is optimistically consistent (transactional)
  * uses iceberg data format
* data swamp
  * raw/all data
  * required to enable reprocessing
  * data is stored to enable reprocessing, or processing in the future
* data lake 
  * all data
  * used in analytics & reporting
  * data is stored in the hope that it could be processed in the future by analytics
  * cleansed & standardised data
  * data governance (data catalogue, lineage, security, metadata)
  * less size than data swap
* data warehouse
  * enterprise-wide data
  * sub-collection of day to day/ historical data, used in reporting
  * less size than data lake
* data mart
  * department-wide data
  * reporting
  * less size than data warehouse

## Processing data

* streaming
  * row/event/micro-batch based processing
  * mostly for preparing data, or short-term decisions/analytics (decision engine)
* batch
  * large chunks of data for processing
  * mostly for long-term analytics (finding connections between data)
* lambda

  * stream + batch in parallel

  kappa

  * stream only