puckel Airflow V1 on K8S

• https://github.com/puckel/docker-airflow
  • "에어플로우 V1" 기준으로~ 쉽게 구축을 할 수 있도록, docker-compose 제공.
  • (이를 참고하여 -> 쿠3스 으로 재구축 ㄱ ㄱ ㄱ ...)
• 쿠3스 구성 구조
  • gw
    • Ingress // webserver 및 flower 서비스 연동
  • airflow
    
    • Service (webserver, flower) -> Endpoints & EndpointSlice
    • Deployment (webserver, scheduler, worker, gitsync, flower) -> ReplicaSet -> POD
    • ConfigMap (airflow.cfg, requirements.txt, airflow.env)
    • Secret (...)
  • vol
    • PersistentVolumeClaim (dags, logs)
  • db
    • Service (postgre, redis) -> Endpoints & EndpointSlice
    • Deployment (postgre, redis) -> ReplicaSet -> POD
  • rbac // rule-based access control (역할기반 접근 제어)
    • ServiceAccount -> Secret (token...)
    • ClusterRoleBinding
    • ClusterRole (resrc[dp, pod, pvc, ...] - verbs[*])
  • test
    • pv, pvc, pod
  • ...
• 1. gw
  • ing-airflow.yaml
    • 메타 : name="airflow-ing", annotation(ingress.class=nginx, ...)
    • 스펙 :
      • host="airflow.abc.xyx", http.path -> service(name=webserver-svc, port=8080)
      • host="flower.abc.xyx", http.path -> service(name=flower-svc, port=5555)
• 2. airflow
  • svc-webserver.yaml
    • 메타 : name="webserver-svc"
    • 스펙 :
      • type="ClusterIP", select.app="webserver-app", port(tcp:8080->8080)
  • svc-flower.yaml
    • 메타 : name="flower-svc"
    • 스펙 :
      • type="ClusterIP", select.app="flower-app", port(tcp:5555->5555)
  • dp-webserver.yaml
    • 메타 : name="webserver-dp"
    • 스펙 :
      • replicas=1, selector.matchLabels.app="webserver-app"
      • 템플릿 :
        • 메타 : label="webserver-app"
        • 스펙 :
          • restartPolicy="Always"
          • containers:
            • image="puckel/docker-airflow:1.10.9", imagePullPolicy="IfNotPresent"
            • resource=..., livenessProbe=..., ...
            • envFrom <- configMapRef(name=airflow-env-cm), env=..., args=["..."]
            • volumeMounts = ['airflow-cfg-cm', 'airflow-req-cm', 'dags-pvc', 'logs-pvc']
          • volumes:
            • configMap(name=airflow-cfg-cm), configMap(name=airflow-req-cm),
            • persistentVC(claimName=dags-pvc), persistentVC(claimName=logs-pvc)
  • dp-scheduler.yaml
    • ''
  • dp-worker.yaml
    • 메타 : name="worker-dp"
    • 스펙 :
      • replicas=1, selector.matchLabels.app="worker-app"
      • 템플릿 :
        • 메타 : label="worker-app"
        • 스펙 :
          • serviceAccount="airflow-sa" // ???
          • restartPolicy="Always"
          • containers:
            • image="커스텀-도커-에어플로우:1.10.9", imagePullPolicy="IfNotPresent"
            • resource=...
            • envFrom <- configMapRef(name=airflow-env-cm), env=..., args=["..."]
            • volumeMounts = ['aws-cred-secret', 'google-auth-secret', 'airflow-cfg-cm', 'airflow-req-cm', 'dags-pvc', 'logs-pvc']
          • volumes:
            • secret(secretName=aws-cred-secret, defaultMode=0644),
            • secret(secretName=google-auth-secret, defaultMode=0644),
            • configMap(name=airflow-cfg-cm), configMap(name=airflow-req-cm),
            • persistentVC(claimName=dags-pvc), persistentVC(claimName=logs-pvc)
  • dp-gitsync.yaml
    • 메타 : name="gitsync-dp"
    • 스펙 :
      • replicas=1, selector.matchLabels.app="gitsync-app"
      • 템플릿 :
        • 메타 : label="gitsync-app"
        • 스펙 :
          • restartPolicy="Always"
          • containers:
            • image="git-sync:v3.3.2", imagePullPolicy="IfNotPresent"
            • resource=...
            • env="GIT_SYNC_...", args=["..."]
            • volumeMounts = ['git-key-secret', 'dags-pvc']
          • volumes:
            • secret(secretName=git-key-secret, defaultMode=0600),
            • persistentVC(claimName=dags-pvc)
          • securityContext:
            • fsGroup: 65533 // ???
  • dp-flower.yaml 
    • ''
  • cm-airflow-cfg.yaml
    • 메타 : name="airflow-cfg-cm", label="airflow-cfg-cm"
    • 데이터 :
      • airflow.cfg: |
        • ...
  • cm-airflow-req.yaml
    • 메타 : name="airflow-req-cm", label="airflow-req-cm"
    • 데이터 :
      • requirements.txt: |
        • ...
  • airflow.env
    • ...
  • secret-google-auth.yaml
    • 메타 : name="google-auth-secret", label="google-auth-secret"
    • 타입 : Opaque
    • 데이터 :
      • google.json: <- base64("{...}")
  • secret-git-key.yaml
    • 메타 : name="git-key-secret", label="git-key-secret"
    • 타입 : Opaque
    • 데이터 :
      • id_rsa: <- base64(비밀키)
      • known_hosts: <- base64("... github.com ssh-rsa ...")
• 3. vol
  • pvc-dags.yaml
    • 메타 : name="dags-pvc"
    • 스펙 :
      • accessModes="ReadWriteMany", storageClassName="airflow-strg", ...
• 4. db
• 5. rbac
• ...

-끝-