At the end of September, Hydra ETL did one thing well: you described a data job in four YAML files, hdrctl validate checked them before any data was read, and the same job ran from the terminal, a REST API or a visual editor.
Six weeks and 14 releases later, a Hydra ETL project can start itself when a file lands in a folder, roll back what it wrote when something fails, aggregate 20 million rows in about half a minute, and call another project as a step. The latest one, 0.19.0, shipped on November 4, which also happens to be my birthday. Not a bad way to celebrate.
This post walks through that evolution in five steps, and what stayed the same.
The starting point: a job you can read and check
A Hydra ETL job is four files: sources.yaml, transformations.yaml, destinations.yaml, pipeline.yaml. Nothing runs until the manifests are valid. That was the promise in September, and every release since has kept it: each new feature comes with its own checks, run before anything touches your data.
1. The project becomes the unit (0.11.4 → 0.12.3)
Teams don't run one job; they run a set of them in a certain order.
-
hdrctl initnow scaffolds a project: jobs, workflows, parameters, environments. -
workflows/thread.yamldeclares the order: stages, sequential or parallel, delays, loops. -
hdrctl run <project>checks the whole project — every workflow, every job, everypipeline.yaml— and lists every problem before starting anything. - Studio stopped overwriting files edited outside it: a change made in your editor or by git is detected, and the latest change wins.
2. Failures stop leaving half-written data (0.13 → 0.14)
This is the part I care most about. A pipeline that fails halfway is worse than one that never starts.
- All-or-nothing load per job: a job writes through one load session. A failure or Ctrl+C leaves the destination as it was (transactions on PostgreSQL, MySQL/MariaDB and SQL Server; temporary file then swap on CSV and Parquet).
- Workflow rollback: with
rollback: true, a failed workflow undoes the steps it already did, last one first. Actions that touch the outside world (webhook, email, scripts) must declare how to undo them, or explicitly say they can't. - Fallback workflows: a backup workflow can take over — before the start, when a condition is true (
not source_available('crm_db')), or after a failure, but only if the failed workflow published nothing. - Timeouts per workflow.
3. Volume (0.15 → 0.16)
pandas stays the default, but above two million estimated rows Hydra ETL can read CSV with pyarrow and run aggregations, sorts and joins in DuckDB. Results are identical to pandas, sums included.
Measured on 20 million rows, 2 CPUs:
- aggregation: 112.8 s / 5.4 GB → 34.3 s / 1.4 GB
- right join then aggregation: 71.8 s / 3.4 GB → 27.7 s / 2.6 GB
No silent fallback: when a step can't run in DuckDB, the log says so. And a new engine: key chooses the engine per job.
4. Secrets and environments (0.17 → 0.18)
- Secrets are masked everywhere Hydra ETL writes text: logs, errors, tracebacks, the output of scripts, stored runs.
- Secrets can be encrypted with SOPS (age, PGP, cloud KMS) and are decrypted in memory only.
- One project, several environments: global variables per machine, parameter values and secrets per environment, managed with
hdrctl env,hdrctl paramandhdrctl secret. A secret value is never passed on the command line.
5. Projects that start themselves (0.19)
The latest release adds project triggers. A triggers.yaml at the project root:
version: "1.0" triggers: - name: nightly type: cron cron: "0 2 * * *" timezone: Europe/Paris - name: new_orders type: file path: inbox/*.csv debounce: 2s - name: after_crm type: after_project project: ../crm "on": success hdrctl trigger validate checks it, then hdrctl watch runs it. Dropping a CSV into inbox/, then finishing the crm project, gives:
Waiting for events. Ctrl+C to stop. 2026-11-04 21:29:36 ▶ sales/new_orders files: inbox/orders_2026-11-04.csv env: none 2026-11-04 21:29:39 ✓ sales/new_orders 2.6s 2026-11-04 21:30:07 ▶ sales/after_crm project crm ended: success env: none 2026-11-04 21:30:12 ✓ sales/after_crm 4.9s Seven trigger types in total: cron, interval, file, marker file, change in a source table, webhook with a token, and the end of another project.
The guarantees are what make this usable:
- a file is only read once it has stopped changing;
- a trigger already running is not started twice;
- a change to
triggers.yamlis picked up without restarting; -
after_projectloops are refused.
The same release adds sub-projects: a workflow step type: project runs another project, with cycle detection and results the next steps can read.
The six weeks at a glance
| Version | Date | What changed |
|---|---|---|
| 0.11.3–0.11.4 | Sep 27 – Oct 17 | Secrets resolve; PostgreSQL upsert fixed; projects |
| 0.12.x | Oct 22–24 | thread.yaml, hdrctl run <project>, loops, Studio ↔ disk sync |
| 0.13–0.14 | Oct 26–28 | All-or-nothing loads, rollback, fallback, timeouts |
| 0.15–0.16 | Oct 28–29 | DuckDB large-volume engine, engine: per job |
| 0.17–0.18 | Nov 1–2 | Masked secrets, SOPS, environments |
| 0.19 | Nov 4 | Project triggers, hdrctl watch, sub-projects |
What didn't change
Still one pip install, no database, no broker, no Node. Still YAML you can review in a pull request. Still checked before it runs.
What's not there yet
Hydra ETL is in beta, with one maintainer. File triggers watch local folders; SFTP and S3 are planned. If you try it and something breaks, an issue helps more than a star.
A Hydra Studio Quick tour
Try Hydra ETL in playground: killercoda.com/hydra-etl
Every release, with its notes: hydraetl.com/changelog
Source: github.com/bejaouibechir/Hydra
Install:
pip install -U hydra-etl
