Data Engineering Project Notes DBT DBT Best Practiceshttps://airbyte.com/blog/best-practices-dbt-style-guide https://airbyte.com/blog/sql-data-modeling-with-dbt Secrets ManagementThis is already taken care of by DBT. Account Settings > Credentials > Environment VariablesTo use these environment variables in dbt code https://docs.getdbt.com/reference/dbt-jinja-functions/env_var Unit Testinghttps://servian.dev/unit-testing-in-dbt-part-1-d0cc20fd189ahttps://servian.dev/a-macro-ful-way-to-test-incremental-models-and-snapshots-2a8187919885CI/CDhttps://towardsdatascience.com/how-to-deploy-dbt-to-production-using-github-action-778bf6a1dff6 Look for topics with heading testing with pull request and linting with sqlfluffMonitoring & alertinghttps://medium.com/@oravidov/dbt-observability-101-how-to-monitor-dbt-run-and-test-results-f7e5f270d6b6 DeploymentsA. Schedule a job in dbt: https://docs.getdbt.com/docs/get-started/getting-started/building-your-first-project/schedule-a-job B. Deploy on airflow: https://github.com/merklescience/crypto-batch-etl/pull/35/filesWe are using airflow for dbt instead of only scheduling the jobs is because the dbt jobs aren’t powerful enough or flexible enough to do multiple things that airflow solves right now. ×