When you build a test automation framework, configuration is usually an afterthought. You write a config.py file, load a YAML file, and move on.
But as a framework grows, configuration becomes the hardest part to get right. You need to handle YAML files, environment variables, and command-line arguments. You need to merge them in a predictable order, validate the values, and fail gracefully when something is wrong.
I've been building Praman, an open-source QA engine for Python. Here is how I designed the configuration pipeline, and why testing it in isolation matters more than the code itself.
Most frameworks use a chain of if/else statements to handle configuration. If the CLI argument exists, use it. Otherwise, check the environment variable. Otherwise, fall back to the YAML file.
This works until it doesn't. It becomes impossible to test, and changing the precedence order means digging through hundreds of lines of code.
I separated the "what order" from the "how to merge".
The configuration is built as a list of sources: YAML, environment variables, and CLI arguments. A builder function assembles this list. A resolver function takes the list and merges the dictionaries from left to right. The last source in the list wins.
Because the resolver doesn't know what a "CLI" is, it can't accidentally reorder the sources. The precedence is defined entirely by the order of the list returned by the builder. If you want to change precedence, you just change the order of the list.
This makes it incredibly easy to test. We use fake config sources in the tests and assert that a later source always overrides an earlier one.
A configuration pipeline is useless if it accepts invalid data.
Praman uses immutable settings objects. Once the settings are validated, they cannot be changed. This prevents tests from accidentally modifying the configuration mid-run.
The validation layer checks for required fields, ensures the browser name is supported (chromium, firefox, or webkit), verifies timeouts are positive integers, and makes sure the version policy is valid. If a client passes an unknown configuration field, the framework raises a clear error.
We also added a nested browser configuration. Instead of just passing a browser name, the settings object holds a BrowserSettings dataclass. This encapsulates the viewport, locale, timezone, and headless mode. It keeps the configuration clean.
A QA framework is only as good as its failure reporting.
We implemented a JSON evidence report that outputs after every test run. It captures test durations, failure details, full-page screenshots (saved to a configurable directory), and the pytest exit status. The report is timestamped in UTC and generated automatically.
But we are also honest about what isn't finished. The blueprint checklist clearly marks which features are fully tested, which are implemented but need edge-case verification, and which are still pending.
For example, we have a GitHub Actions workflow that runs the test suite. The latest successful run (commit 35e0424) passed. But we haven't yet verified the external client installation from a clean virtual environment using the final wheel. That step is pending.
The configuration pipeline is just one piece of the framework. The full test suite is currently passing, and the code is public on GitHub.
Building a framework isn't just about writing the automation logic. It's about building a reliable, predictable configuration layer, validating it properly, and knowing exactly what you have and haven't proven.
If you want to look at the code, the repository is public: https://github.com/pranromumu/praman
