Description

We want to bring open-source technologies to write modular, testable, maintainable SQL transformations and automate documentation of the sql code.

DBT Core is an open-source library for data engineering that helps users build interdependent SQL models for sql code data transformation, code testing, documentation and lineage.

We use Apache Airflow to orchestrate data pipelines therefore this project will be evaluation of dbt-core with Airflow (MWAA).

Goals

• Get more familiar with DBT-core
• Create the transformation data pipeline
• Organize sql code in models
• Create automated testing
• Create documentation for this code
• Demo

Resources

https://github.com/dbt-labs/dbt-core

https://github.com/astronomer/astronomer-cosmos

https://www.astronomer.io/docs/learn/airflow-dbt/