OpenShift Airflow Cluster

I spent some time playing with Airflow installation in OpenShift. It was a little bit tricky. Python is sensitive to run own application under not root user, as requires OpenShift Dedicated.

Every pod in OpenShift Dedicated is running under not root user.  
Finally, I am happy with the result. You can find it following this link.

Also, made changes into the Puckel Docker image for Airflow. Created own Docker image on DockerHub. You can find the image here.

Comments

Post a Comment

Popular posts from this blog

Install Kubeflow locally

For local installation of Kubeflow you can use Vagrant box, Minikube or install it using Multipass. Here I will describe the multipass installation. The best link for the installation process is here . This installation uses Microk8s orchestrator and has some caveats. First of all you have to create volumes manually, other way you services won't work correctly. Before all I would suggest to install version 1.13 of microk8s, because 1.14 is not compatible now with kubflow. sudo snap install microk8s --channel=1.13 For this purpose please edit kubernetes-tools/setup-microk8s.sh line 10: CHANNEL=${CHANNEL:-1.13/stable} 1.14 where changed on 1.13. Then run  sudo kubernetes-tools/setup-microk8s.sh Step 1: Create storage class. I used next data: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer Just save and apply this data. kubectl apply -f you_file_name.yaml ...
Keep reading

RabbitMQ and OpenShift

RedHat created very good launcher for a RabbitMQ cluster in OpenShift ( can be found here ). The problem I was stacked a little bit was creation of the cluster in an already created project. I found out that it is easy to do, just change slightly a all.yml file. --- openshift_cluster_content: # No need to create a new project #- object: projectrequest # content: # - name: rabbitmq-spaces # file: "{{ inventory_dir }}/../files/projects/projects.yml" # file_action: create - object: imagestream content: - name: rhel7 file: "{{ inventory_dir }}/../files/imagestreams/images.yml" namespace: scouting # Also important to fix deployment config template, which is located in the files/deployments/template.yml . The deployment will hook wrong image, so cluster won't build without the fix. Problem is in the naming of the image used to build rabbits. image: "${APPLICATION_NAME}:${RABBITMQ_VERSION}" Change to: image: ...
Keep reading