MLOps
The Wenex MLOps system is a production-grade data pipeline that continuously captures change events from MongoDB via Kafka (CDC), archives each changed document's latest state in PostgreSQL, and transforms them into versioned Delta Lake tables in LakeFS using configurable Python scripts. Once data is tagged in LakeFS, Airflow DAGs can be triggered automatically to run downstream ML workflows — including batch model training directly from the Delta Lake store.
Key Technologies
| Component | Technology | Version |
|---|---|---|
| Task queue | Celery | 5.6.2 |
| Data lake | LakeFS — Python SDK (lakefs) | 0.14.1 |
| Table format | Delta Lake | 1.3.0 |
| DAG orchestration | Apache Airflow | 3.1.6 |
| Columnar processing | PyArrow | 18.1.0 |
| DataFrame library | Polars | 1.36.1 |
| Runtime | Python | 3.12 |
Pages
→ Quickstart — Add a data source and start collecting in minutes
→ Architecture — Component roles, storage layers, data flow detail, and concurrency model
→ Scripts — config.yaml reference and how to write custom data transformation scripts
→ Airflow DAGs — DAG structure, LakeFS webhook trigger, and writing new DAGs
→ Model Training — Streaming Delta Lake data batch-by-batch for PyTorch + MLflow training
→ Deployment — Helm chart configuration for Beat, Workers, and Flower