Once you’re done, you’re ready to go! Materials To run Airflow on Kubernetes you need 5 tools: Docker, Docker Compose, KinD, Helm and Kubectl. Without Kubectl, you won’t be able to get the logs of your PODs, debug your errors or check your nodes. As you know, Helm allows you deploy applications on Kubernetes whereas Kubectl allows you to run commands against your Kubernetes Cluster. ![]() In addition to Docker and KinD, you obviously need to install Helm as well as Kubectl. If you know MiniKube, Kind is pretty similar now (It wasn’t the case a few years ago). It was primarily designed for testing Kubernetes but it is perfect to have a quick local development environment to experiment applications that run on top of Kubernetes. KinD is the second tool you need as it allows you to set up and run a Kubernetes cluster using Docker container “nodes”. Now you may ask: “How can I create a Kubernetes cluster with Docker?” Well, say hello to KinD! You are going to have multiple Docker containers where each container will represent a Kubernetes node. Why? Because the Kubernetes cluster will be created in Docker. Therefore, there are some prerequisites before jumping into the practice part.įirst, you need to install Docker and Docker Compose. In this very hands-on focused tutorial you are going to create a local multi-node Kubernetes cluster to deploy and run Airflow on it. Deploy your DAGs on Kubernetes with GitSync.Install dependencies with Airflow on Kubernetes.$ helm install MY-RELEASE oci:///tac-for-tanzu-advanced/charts/airflow -set "global. **If you are using Helm CLI version 3.7.x (please note `export HELM_EXPERIMENTAL_OCI=1` is not needed from 3.8.0 onwards)** $ helm install MY-RELEASE airflow/ -set "global.imagePullSecrets=SECRET-NAME" $ helm chart export /tac-for-tanzu-advanced/charts/airflow:latest $ helm chart pull /tac-for-tanzu-advanced/charts/airflow:latest **If you are using Helm CLI version older than 3.7.0** Replace the SECRET-NAME placeholder with the same Kubernetes ImagePullSecret name used previously. If using VMware Application Catalog for Tanzu Advanced, use the commands below to deploy the chart instead. $ helm install MY-RELEASE appcatalog/airflow The name must start and end with a lower-case alphanumeric character and can only contain alphanumeric character, '-' or '.'. Replace it with the name you want to give to the deployment or add the -generate-name parameter to automatically generate a name. NOTE: Remember that MY-RELEASE is a placeholder. The "release" includes files with Kubernetes-needed resources and files that describe the installation, configuration, usage and license of a chart.Ĭheck that your Kubernetes cluster is running by executing the following command: $ kubectl cluster-infoĭeploy the chart with the following command: $ helm registry login /tac-for-tanzu-advanced/chartsĪ Helm chart describes a specific version of a solution, also known as a "release". $ kubectl create secret docker-registry SECRET-NAME \ Replace the USERNAME and TOKEN placeholders with the correct registry credentials and the SECRET-NAME placeholder with a custom name for the secret. ![]() If using VMware Application Catalog for Tanzu Advanced, it is necessary to first create a Kubernetes ImagePullSecret for /tac-for-tanzu-advanced with the registry credentials, and then log in to the registry with the registry credentials. $ helm repo add appcatalog REPOSITORY -username "USERNAME" -password "PASSWORD" Replace the USERNAME and PASSWORD placeholders with the correct username and token and the REPOSITORY placeholder with a reference to your VMware Application Catalog chart repository. ![]() Follow these steps:Īdd the VMware Application Catalog repository to Helm with the following command. To start using any VMware Application Catalog Helm chart, it is necessary to first add the VMware Application Catalog Helm chart repository to Helm and then run the helm install command to deploy this chart. NOTE: To install a Helm chart repository, it is necessary to have Helm previously installed and configured in your cluster. Apache Airflow packaged by VMware - Install the chart
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |