Deployment

The MLOps system is deployed on Kubernetes using a Helm chart composed of three sub-charts: Beat, Workers, and Flower. All three share the same container image and environment configuration defined in the parent values.yaml.

Helm Chart Repository

The MLOps Helm chart is published in the maintainer's own chart repository (vhidvz.github.io/charts, which packages airflow and mlops) — not in the Wenex org host wenex-org.github.io/charts that the platform charts use, hence the distinct alias:

helm repo add wenex-mlops https://vhidvz.github.io/charts
helm repo update

Verify the chart is available:

helm search repo wenex-mlops/mlops

Chart Structure

mlops/
├── Chart.yaml          # parent chart
├── values.yaml         # global config + sub-chart toggles
└── charts/
    ├── beat/           # Celery Beat scheduler (singleton)
    ├── flower/         # Celery monitoring UI
    └── workers/        # Celery task executors (scalable)

The parent chart also deploys the Collector pods (main.py) that consume Kafka CDC events and write to PostgreSQL.

Sub-chart Roles

Sub-chart	Default Replicas	Command	Notes
beat	1	celery -A tasks beat --loglevel=info --schedule=.data/celerybeat-schedule	Exactly 1 replica — see Scaling Workers.
flower	1	celery -A tasks flower	Monitoring UI on port 5555.
workers	3 (configurable)	celery -A tasks worker --loglevel=info	Stateless; can be scaled horizontally.

Enabling Sub-charts

Sub-charts are disabled by default. Enable them in values.yaml:

beat:
  enabled: true

flower:
  enabled: true

workers:
  enabled: true

Environment Variables Reference

All environment variables are set under global.envs in values.yaml and are injected into every pod (Collector, Beat, Workers, Flower).

LakeFS

Variable	Description
LAKEFS_SERVER_ENDPOINT_URL	LakeFS server base URL (e.g. https://lakefs.example.com)
LAKEFS_CREDENTIALS_ACCESS_KEY_ID	LakeFS access key ID
LAKEFS_CREDENTIALS_SECRET_ACCESS_KEY	LakeFS secret access key

Kafka

Variable	Description
KAFKA_BOOTSTRAP_SERVERS	Comma-separated broker addresses (e.g. kafka-broker:9092)

PostgreSQL

Variable	Description
PG_HOST	Hostname of the PostgreSQL server
PG_DB	Database name (default: lakefs)
PG_USER	Database user
PG_PASSWORD	Database password
PG_PORT	Port (default: 5432)

Redis

Variable	Description
REDIS_HOST	Redis hostname
REDIS_PORT	Redis port (default: 6379)
REDIS_PASSWORD	Redis password (leave empty if none)
REDIS_PREFIX	Key prefix for all MLOps Redis keys (default: mlops)

Celery

Variable	Description
CELERY_BROKER	Celery broker URL (e.g. redis://redis-host:6379/0)
CELERY_BACKEND	Celery result backend URL (same host, different DB index is fine)

Both are required — every component fails at startup if either is unset — and are used as given. Released images up to and including 1.0.6 (the 1.0.0 tag below among them) ignore both values and connect to redis://localhost:6379/0; the fix is newer than 1.0.6. Celery prefixes its broker and result keys with a fixed mlops:, independent of REDIS_PREFIX.

MongoDB

Variable	Description
MONGO_URI	Full MongoDB URI including auth and replica set (e.g. mongodb://user:pass@host/?replicaSet=rs0&authSource=admin)

Example values.yaml

replicaCount: 5        # collector pods

global:
  image:
    repository: registry.wenex.org/vhidvz/mlops
    pullPolicy: IfNotPresent
    tag: "1.0.0"
  envs:
    lakefs:
      serverEndpointUrl: "https://lakefs.wenex.org"
      credentialsAccessKeyId: "AKIAxxx"
      credentialsSecretAccessKey: "xxx"
    kafka:
      bootstrapServers: "wenex-kafka-bootstrap.kafka.svc.cluster.local:9092"
    postgres:
      host: "postgres-cluster-rw.cnpg-system.svc.cluster.local"
      port: "5432"
      db: "lakefs"
      user: "lakefs"
      password: "lakefs"
    redis:
      host: "redis-stack-master.redis.svc.cluster.local"
      port: "6379"
      password: ""
      prefix: "mlops"
    celery:
      broker: "redis://redis-stack-master.redis.svc.cluster.local:6379/0"
      backend: "redis://redis-stack-master.redis.svc.cluster.local:6379/0"
    mongodb:
      uri: "mongodb://lakefs:<LAKEFS_PASS>@psmdb-server-psmdb-db-rs0.mongodb.svc.cluster.local/?replicaSet=rs0&authSource=admin"

beat:
  enabled: true

flower:
  enabled: true

workers:
  enabled: true
  replicaCount: 3    # override per sub-chart

Install or upgrade with your custom values:

helm upgrade --install mlops wenex-mlops/mlops -f values.yaml

Scaling Workers

Increase or decrease Worker replicas by setting workers.replicaCount in values.yaml, or via --set at install time:

helm upgrade mlops wenex-mlops/mlops --set workers.replicaCount=6

Workers are stateless — scaling up adds capacity immediately without any coordination required.

Do not scale Beat. Beat stores its schedule state in .data/celerybeat-schedule and must run as a singleton. Running two Beat instances will result in duplicate task dispatches.

Container Image

registry.wenex.org/vhidvz/mlops:1.0.0

Base image: apache/airflow:3.1.6-python3.12. The image includes all dependencies (celery, lakefs, deltalake, polars, pyarrow, psycopg2, etc.) and the full scripts/ directory.

Accessing Flower

Flower listens on port 5555 in its pod; the mlops-flower Service exposes it on port 80. In a cluster environment, access it via port-forward:

kubectl port-forward svc/mlops-flower 5555:80

Then open http://localhost:5555 to view active workers, task history, queues, and execution times.