FastAPI, PostgreSQL, Redis, Celery, and Docker — and the real-world challenges that emerged when everything had to work together.

Every software project starts with an idea. But there is a considerable gap between defining what we want to build and having an application that actually works, is tested, and is ready to evolve.

That is exactly the gap I started bridging with JC SocialFlow AI, a personal software engineering project focused on social media content management and automation across platforms such as Instagram, Facebook, WhatsApp, TikTok, and YouTube.

The goal is to build a platform capable of organizing, preparing, and eventually publishing content across multiple channels, combining automation, artificial intelligence, and human oversight.

But before thinking about social media integrations or AI-powered features, I had a more important question:

How can I build a technical foundation that allows the project to grow without turning every new feature into an infrastructure problem?

That became the mission of Sprint 01.

A quick note: in my current projects, I use Sprint XX to describe development phases. Each phase has a defined scope, validation criteria, and a clear outcome.

First, the foundation

During the previous phase, Sprint 00, I focused on planning: requirements, architecture, security, the roadmap, and component responsibilities.

For the implementation, I chose a modular monolith with independent workers.

It was a deliberate decision.

Instead of starting with multiple microservices, each requiring its own infrastructure and operational considerations, I preferred to keep the backend within a modular application.

Each module has clearly defined responsibilities, while the architecture leaves room to extract components later if there is a genuine need.

This approach reduces initial operational complexity without sacrificing important engineering principles such as separation of concerns, testability, and observability.

The technology stack includes:

  • Python 3.12+ and FastAPI: API development and modular organization.
  • PostgreSQL and SQLAlchemy: data persistence.
  • Alembic: versioned and controlled database migrations.
  • Redis and Celery: asynchronous task processing.
  • Docker Compose: integrated service execution.
  • GNU Make: standardized operational commands.
  • Pytest and GitHub Actions: automated testing and continuous integration.

The goal was not to use as many technologies as possible, but to ensure that each one had a clear purpose.

IMAGE 1 — JC SocialFlow AI architecture diagram

Figure 1 — JC SocialFlow AI architecture after Sprint 01: a modular FastAPI backend, PostgreSQL persistence, and asynchronous processing with Redis and Celery.

A simple API with important responsibilities

The first step was turning the architecture into a working application.

I built the API with FastAPI, organizing it into modules and preparing it to handle identity, auditing, and infrastructure concerns without mixing their responsibilities.

I also implemented two endpoints that might look simple but make a significant difference during operation:

GET /health/live GET /health/ready 

The first checks whether the application process is running.

The second checks whether the application is ready to handle operations that depend on PostgreSQL.

This distinction became especially important during testing.

At one point, the API responded successfully to the liveness endpoint while returning HTTP 503 from the readiness endpoint.

In other words, the process was alive, but the application was not ready.

That behavior helped uncover real configuration and database connectivity issues instead of hiding them behind a simple HTTP 200 response.

I also introduced structured JSON logging, correlation IDs, and sanitized error handling.

These details may not be particularly exciting in a visual demonstration, but they become extremely useful when something goes wrong.

IMAGE 2 — Terminal showing health endpoints and container status

Figure 2 — Local Docker Compose services and FastAPI health checks after completing Sprint 01.

Identity and security from the beginning

Another important goal was establishing the first authentication and authorization mechanisms.

The backend now uses Argon2id for password hashing, along with opaque, revocable sessions managed through the database.

I also implemented protections involving cookies, request origins, CSRF, and role-based and resource-owner authorization.

On the PostgreSQL side, I separated database roles for administration, migrations, and application runtime.

This means the database user used by the API does not need the same permissions as the user responsible for changing the database schema.

Migrations are not executed automatically when the application starts, either.

They must be triggered explicitly and require interactive confirmation when targeting the persistent local development database.

This introduced a little more operational work, but it established an important distinction between running an application and modifying its data structures.

And, interestingly enough, this was where some of the most valuable lessons of the sprint emerged.

When an @ in a password becomes an infrastructure problem

One of the most challenging parts of Sprint 01 involved PostgreSQL configuration and database migrations.

The environment used Docker Compose with a persistent database volume.

After some changes to the .env file, the application stopped passing its readiness check.

The first issue was relatively common: PostgreSQL retained users and passwords from the database's previous initialization.

Changing an environment variable does not automatically update credentials already stored in a persistent database.

I had to reconcile the existing database roles while preserving the data.

Then came a more subtle issue.

One of the passwords used during testing contained the @ character.

In a database connection URL, this character has a special meaning. Without proper encoding, it can be interpreted as part of the URL structure, changing how the connection address is parsed.

The solution involved applying the correct percent-encoding to the URLs and strengthening configuration validation.

But there was still a third issue.

During Alembic execution, the configuration handling duplicated % characters from the encoded password. As a result, PostgreSQL received credentials that differed from the intended values.

I fixed the connection setup so the correctly encoded URL was preserved when creating the database engine.

I also adjusted the migration command to prevent Docker Compose from unnecessarily recreating the database container.

Eventually, Alembic revision 0002 was successfully applied without removing the persistent volume.

The lesson was straightforward: a configuration can pass validation and still fail when it moves through different libraries.

It is not enough to check whether a URL looks correct. You need to test the actual path through which that URL will be consumed.

IMAGE 3 — Sanitized evidence showing Alembic revision raw `0002` endraw applied

Figure 3 — PostgreSQL confirms Alembic revision 0002_identity_security, while the FastAPI readiness endpoint returns HTTP 200.

Asynchronous processing without building features too early

Another part of Sprint 01 involved preparing the asynchronous processing infrastructure.

For this, I used Redis as the message broker and Celery to execute tasks in a worker running independently from the API.

At this stage, the goal was not to publish content to social media platforms.

Instead, I implemented a deterministic internal diagnostic task with no external side effects.

This task allows me to validate the flow between the application, the queue, and the worker, including tracking execution through correlation IDs.

The approach helped verify the infrastructure before introducing external integrations, platform credentials, or more complex publishing rules.

During implementation, I also encountered issues involving dependencies and service startup behavior.

Once again, the challenge was not simply getting each component to work independently.

It was making sure they worked together.

IMAGE 4 — Celery worker logs showing successful diagnostic task execution

Figure 4 — Celery worker logs confirming successful completion of internal diagnostic tasks, with correlation IDs for execution tracking.

Making the environment reproducible

With the main components implemented, the next challenge was simplifying local operations.

Docker Compose now manages the API, PostgreSQL, Redis, and worker.

PostgreSQL and Redis do not need to expose their ports directly to the host, while the API is accessible through the local loopback interface.

To reduce manual commands and make operational procedures more consistent, I introduced a Makefile.

Some examples:

make up make status make check make smoke make down 

The make check command runs code checks and automated tests.

The make smoke command validates the integrated services.

And make down stops the services without removing persistent volumes.

I also prepared a manual acceptance procedure covering authentication, sessions, and authorization.

An unexpected issue appeared in the documentation itself.

A command example using a heredoc could leave the terminal waiting at a continuation prompt when the closing delimiter was not copied correctly.

I replaced that example with a dedicated script that requests credentials interactively and displays only the necessary HTTP results.

It was a useful reminder that operational documentation needs testing, too.

Testing: from my local machine to GitHub Actions

By the end of the sprint, the main components had gone through automated, integrated, and manual testing.

The results included:

  • 20 passing Pytest tests.
  • Five passing migration-validator tests.
  • Regression tests using passwords containing @.
  • Integrated smoke tests involving the API, PostgreSQL, and worker.
  • Alembic revision 0002 successfully applied and verified.
  • Health endpoints returning HTTP 200.
  • Successful manual functional acceptance.

I also published the continuous integration workflow through GitHub Actions.

The final workflow run, triggered by the push of the Sprint 01 closing commit, completed successfully in 1 minute and 26 seconds.

That does not mean the system is production-ready.

It means the foundation has met the validation criteria defined for this development phase and can continue evolving in a controlled way.

IMAGE 5 — Successful Sprint 01 GitHub Actions workflow

What I learned along the way

Sprint 01 reinforced a few principles that I consider important in software engineering.

The first is that architectural simplicity does not mean a lack of engineering discipline.

A modular monolith can include authentication, access control, versioned migrations, independent workers, observability, and integrated testing without starting as a collection of microservices.

The second is that persistent infrastructure requires careful handling.

Containers can be recreated, but volumes retain state. And that state does not necessarily follow changes made to configuration files.

The third is that seemingly small problems can reveal deeper integration issues.

A password containing a reserved character, for example, exposed differences in URL handling across libraries and tools.

Finally, this sprint reinforced something I have learned repeatedly: automated tests do not completely replace operational validation.

Some of the most important issues appeared during manual acceptance, using the actual persistent local development environment.

Those problems eventually became regression tests and improvements to the development process.

What's next?

With Sprint 01 completed, JC SocialFlow AI now has an initial foundation that includes a modular API, persistence, identity, auditing, asynchronous processing, a reproducible local environment, and continuous integration.

There is still plenty of work ahead.

Backup and recovery, data retention, key rotation, production security hardening, and more advanced observability remain areas that need attention.

Social media integrations, artificial intelligence features, media processing, and the user interface are also part of the project's planned evolution.

My intention is to move forward one phase at a time, documenting architectural decisions, real-world problems, and lessons learned along the way.

Because, in the end, building software is not just about implementing features.

It is about creating the conditions for a system to keep working — and evolving — as complexity grows.

References and further reading

The following official resources provide additional technical context for the technologies and engineering practices discussed in this article.

Architecture and API development

  1. FastAPI — Official Documentation. API development, dependency injection, application structure, and testing.

    https://fastapi.tiangolo.com/

  2. Python Software Foundation — Python 3.12 Documentation. Language and standard library reference.

    https://docs.python.org/3.12/

  3. The Twelve-Factor App — Config and Processes. Principles for application configuration and process separation.

    https://12factor.net/

Database, migrations, and security

  1. PostgreSQL — Database Roles. Role management and database privileges.

    https://www.postgresql.org/docs/current/user-manag.html

  2. SQLAlchemy — Engine Configuration. Database URLs, connection configuration, and special characters in passwords.

    https://docs.sqlalchemy.org/en/20/core/engines.html

  3. Alembic — Official Documentation. Database migrations with SQLAlchemy.

    https://alembic.sqlalchemy.org/en/latest/

  4. OWASP — Password Storage Cheat Sheet. Password hashing recommendations, including Argon2id.

    https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html

  5. OWASP — Session Management Cheat Sheet. Secure session management practices.

    https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html

Asynchronous processing and infrastructure

  1. Celery — Official Documentation. Distributed task queues, workers, and asynchronous processing.

    https://docs.celeryq.dev/en/stable/

  2. Redis — Official Documentation. Redis concepts, configuration, and messaging capabilities.

    https://redis.io/docs/latest/

  3. Docker — Compose Documentation. Multi-container applications, services, and lifecycle management.

    https://docs.docker.com/compose/

  4. Docker — Volumes. Persistent storage and volume lifecycle.

    https://docs.docker.com/engine/storage/volumes/

Testing and continuous integration

  1. Pytest — Official Documentation. Automated testing and test organization.

    https://docs.pytest.org/en/stable/

  2. GitHub Actions — Official Documentation. Continuous integration workflows and automation.

    https://docs.github.com/en/actions

  3. GNU Make — Official Manual. Build automation and reproducible commands.

    https://www.gnu.org/software/make/manual/

This article is part of a series documenting the development of JC SocialFlow AI, following the project's evolution from architectural decisions to implementation and operation.