Create a new Astronomer / Airflow Github repository

Create an empty repository on github

Clone it into your local repository or make an initial commit from local repo to remote repo

Create placeholder files for all the required astronomer files you need. You can do this by running

cd repo/
astro dev init

Create a doppler project on the Data Engineering team. This doppler project will be linked to all your cloud environments and local environments for astronomer

Modify the Dockerfile which was created to look like this. This modification allows us to inject environment variables into airflow through doppler

FROM quay.io/astronomer/astro-runtime:5.0.6

USER root

# Install Doppler CLI
RUN apt-get update && apt-get install -y apt-transport-https ca-certificates curl gnupg && \
    curl -sLf --retry 3 --tlsv1.2 --proto "=https" 'https://packages.doppler.com/public/cli/gpg.DE2A7741A397C129.key' | apt-key add - && \
    echo "deb https://packages.doppler.com/public/cli/deb/debian any-version main" | tee /etc/apt/sources.list.d/doppler-cli.list && \
    apt-get update && \
    apt-get -y install doppler

USER astro

ENV DOPPLER_TOKEN=${DOPPLER_TOKEN}

ENTRYPOINT ["/entrypoint", "doppler", "run", "--"]

CMD ["airflow", "--help"]

Create two GCP projects into which will contain data assets (GCP files, Bigquery Tables) with this repository. These need to be named as follows

1. data-engg-<projectname>-dev

2. data-engg-<projectname>-prod

Setup 3 custom IAM roles

1. Bigquery Read Permissions

   #!/usr/bin/env bash
   
   # This script creates or updates a bigquery read access role in multiple gcp projects
   
   ROLE_ID=Read_BigqueryReadAccessRole
   DESCRIPTION="Provide Read Access to Bigquery Datasets in this project"
   PERMISSIONS="bigquery.datasets.get,bigquery.datasets.getIamPolicy,bigquery.tables.export,bigquery.tables.get,bigquery.tables.getData,bigquery.tables.getIamPolicy,bigquery.tables.list,resourcemanager.projects.get"
   
   declare -a projects=("data-engg-<projectname>-dev" "data-engg-<projectname>-prod")
   
   for project in "${projects[@]}"
   	do
   		gcloud iam roles create ${ROLE_ID} --project=${project} \
   		    --title=${ROLE_ID} \
   		    --description=${DESCRIPTION} \
   		    --permissions=${PERMISSIONS}
   	done

   CODE

2. Airflow User Permissions

   #!/usr/bin/env bash
   
   ROLE_ID=Write_AirflowUser
   DESCRIPTION="Provide Write Access to Bigquery Datasets and Google Cloud Storage in this project"
   PERMISSIONS="bigquery.datasets.create,bigquery.datasets.createTagBinding,bigquery.datasets.delete,bigquery.datasets.get,bigquery.datasets.getIamPolicy,bigquery.datasets.update,bigquery.jobs.create,bigquery.jobs.delete,bigquery.jobs.get,bigquery.jobs.list,bigquery.jobs.listAll,bigquery.jobs.listExecutionMetadata,bigquery.jobs.update,bigquery.models.export,bigquery.models.getData,bigquery.models.getMetadata,bigquery.models.list,bigquery.routines.get,bigquery.routines.list,bigquery.savedqueries.create,bigquery.savedqueries.delete,bigquery.savedqueries.get,bigquery.savedqueries.list,bigquery.savedqueries.update,bigquery.tables.create,bigquery.tables.delete,bigquery.tables.export,bigquery.tables.get,bigquery.tables.getData,bigquery.tables.getIamPolicy,bigquery.tables.list,bigquery.tables.update,bigquery.tables.updateData,resourcemanager.projects.get,storage.buckets.create,storage.buckets.delete,storage.buckets.get,storage.buckets.list,storage.objects.create,storage.objects.delete,storage.objects.get,storage.objects.getIamPolicy,storage.objects.list,storage.objects.update"
   
   declare -a projects=("data-engg-<projectname>-dev" "data-engg-<projectname>-prod")
   
   for project in "${projects[@]}"
   	do
   		gcloud iam roles create ${ROLE_ID} --project=${project} \
   		    --title=${ROLE_ID} \
   		    --description=${DESCRIPTION} \
   		    --permissions=${PERMISSIONS}
   	done

   CODE

3. Airflow Owner Permissions

   #!/usr/bin/env bash
   
   ROLE_ID=Write_AirflowOwner
   DESCRIPTION="Provide Write & Read Access to Bigquery Datasets, And Google Cloud Storage Admin in this project"
   PERMISSIONS=bigquery.datasets.create,bigquery.datasets.createTagBinding,bigquery.datasets.delete,bigquery.datasets.get,bigquery.datasets.getIamPolicy,bigquery.datasets.update,bigquery.jobs.create,bigquery.jobs.delete,bigquery.jobs.get,bigquery.jobs.list,bigquery.jobs.listAll,bigquery.jobs.listExecutionMetadata,bigquery.jobs.update,bigquery.models.export,bigquery.models.getData,bigquery.models.getMetadata,bigquery.models.list,bigquery.routines.get,bigquery.routines.list,bigquery.savedqueries.create,bigquery.savedqueries.delete,bigquery.savedqueries.get,bigquery.savedqueries.list,bigquery.savedqueries.update,bigquery.tables.create,bigquery.tables.delete,bigquery.tables.export,bigquery.tables.get,bigquery.tables.getData,bigquery.tables.getIamPolicy,bigquery.tables.list,bigquery.tables.update,bigquery.tables.updateData,resourcemanager.projects.get,storage.buckets.create,storage.buckets.delete,storage.buckets.get,storage.buckets.list,storage.objects.create,storage.objects.delete,storage.objects.get,storage.objects.getIamPolicy,storage.objects.list,storage.objects.update,cloudbuild.builds.create,cloudbuild.builds.get,cloudbuild.builds.list,cloudbuild.builds.update,remotebuildexecution.blobs.get
   
   declare -a projects=("data-engg-<projectname>-dev" "data-engg-<projectname>-prod")
   
   for project in "${projects[@]}"
   	do
   		gcloud iam roles create ${ROLE_ID} --project=${project} \
   		    --title=${ROLE_ID} \
   		    --description="Provide Write & Read Access to Bigquery Datasets, And Google Cloud Storage Admin in this project" \
   		    --permissions=${PERMISSIONS}
   	done

   CODE

Assign roles to the following google groups

1. data-science@merklescience.com, intelligence@merklescience.com

   1. Read_BigqueryReadAccessRole on data-engg-<projectname>-dev and data-engg-<projectname>-prod

2. data-engineering@merklescience.com

   1. Write_AirflowOwner on data-engg-<projectname>-dev

   2. Write_AirflowOwner on data-engg-<projectname>-dev can be provided to specific individuals instead of the google group

3. service accounts for both environments will need to be created and provided with Write_AirflowUser access

Setup CI / CD using Cloud Build
Copy the contents below into a folder structure and filename -> infra/deploy/build.yaml
This file describes the sequence of steps to deploy Astronomer to their respective environments

steps:

    # Substitutes values from cloud build env into Dockerfile
  - id: load-envvars
    name: "gcr.io/${PROJECT_ID}/envsubst"
    env:
    - 'DOPPLER_TOKEN=${_DOPPLER_TOKEN}'
    args: ["Dockerfile"]

  - id: deploy-astro
    name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
        [
            "-c",
            "curl -sSL install.astronomer.io | bash -s && astro deploy ${_ASTRONOMER_DEPLOYMENT_ID} -f"
        ]
    env:
        - 'ASTRONOMER_KEY_ID=${_ASTRONOMER_KEY_ID}'
        - 'ASTRONOMER_KEY_SECRET=${_ASTRONOMER_KEY_SECRET}'

Enabled Google Cloud APIs for Cloud Build → This can only be done by the project owner. Contact him / her to enable this API. Without enabling this API, it will not be possible to trigger deployments through Cloud Build.
https://www.loom.com/share/daeb5f48b79346aa9214845e9d1eca44

Create a cloud build trigger in each of the GCP projects - data-engg-<projectname>-dev and data-engg-<projectname>-prod

1. Link this trigger to the github repository → Anyone with Github Owner permissions should be able to do this

2. On the dev environment, set this trigger to manually fire and keep main as the default branch.
   On the prod environment, set this trigger to fire whenever this is a commit / merge made into the main branch

3. Setup the following substitution variables

   1. _ASTRONOMER_KEY_ID

   2. _ASTRONOMER_KEY_SECRET

   3. _DOPPLER_TOKEN

   4. _ASTRONOMER_DEPLOYMENT_ID
      You can retrieve these values for your dev and prod environments with the following tutorial
      https://www.loom.com/share/6e67529112984ee796b826359466bceb

Setup Slack notifications for Cloud build jobs.
Slack notifications are implemented with Cloud Functions + Pub/Sub + Cloud Scheduler. I know, you’re going to say this is unnecessarily complicated, and I dont blame you, I blame GCP for this. Idiots whoever thought this was a good idea. But, https://github.com/aknirmal90/gcp-cloudbuild-slack-notifications is the easiest implementation I’ve found so far.

We have a slack channel called #airflow-dev-events which is where all build notifications and github events

It’s webhook URL is https://hooks.slack.com/services/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

Before you can begin deployments through cloud-build, you must first push a container image for envsubst, an app we use in cloud builds to overwrite variables in application files (substitution variables only work on cloud build files, so we need to resort to envsubst to achieve fuller automation)

How to install envsubst
https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/envsubst

git clone https://github.com/GoogleCloudPlatform/cloud-builders-community.git
cd cloud-builders-community/envsubst
gcloud config set project <projectid>
gcloud builds submit --config=cloudbuild.yaml

And, you have now setup a CI / CD + Cloud Environments for your astro project