Airflow, GKE, and easy autoscaling

This is a short one…GCP has a hosted Airflow service: Composer. There’s a lot to talk about with Airflow, and I’m going to gloss over most of it. For the sake of brevity, I’ll assume you know a bit about what Airflow is, and how you can run jobs in various places using it.

If you’re using Composer, you have easy access to GKE for launching jobs. In fact, Composer actually runs on a GKE cluster! And using GKE (or the more generic KubernetesPodOperator) to launch jobs is great, because then we can easily request more CPU/Memory/Disk for a particular DAG easily:

    KubernetesPodOperator(
        container_resources=k8s_models.V1ResourceRequirements(
            requests={
                "cpu": "1000m",
                "memory": "10G",
            },
            limits={
                "cpu": "1000m",
                "memory": "10G",
            },
        ),
        do_xcom_push=False,

This is particularly nice for ML jobs that may need a lot of resources while running, but we don’t want to keep those resources running…

Anyway…this is getting long…the point of this post:

The Point

KubernetesPodOperator needs a namespace to operate in. The default (default) actually has some permission issues for general workloads, but Composer has a (undocumented as far as I can tell) proper user namespace! I’m going to put this is a unique block so it’s easy to see:

User Namespace for Composer k8s DAGs:

composer-user-workloads

Any jobs submitted to this namespace will be able to request resources, do xcom, and all the other tasks we expect of Airflow. Using this namespace is way easier than trying to provision/manage your own namespaces.