ETL-process

ETL (Extract, Transform and Load) is a process in data warehousing responsible for pulling data out of the source systems and placing it into a data warehouse. ETL involves the following tasks:

- extracting the data from source systems (SAP, ERP, other oprational systems), data from different source systems is converted into one consolidated data warehouse format which is ready for transformation processing.

- transforming the data may involve the following tasks:

•   applying business rules (so-called derivations, e.g., calculating new measures and dimensions),
•   cleaning (e.g., mapping NULL to 0 or "Male" to "M" and "Female" to "F" etc.),
•   filtering (e.g., selecting only certain columns to load),
•   splitting a column into multiple columns and vice versa,
•   joining together data from multiple sources (e.g., lookup, merge),
•   transposing rows and columns,
•   applying any kind of simple or complex data validation (e.g., if the first 3 columns in a row are empty then reject the row from processing)

- loading the data into a data warehouse or data repository other reporting applications