Forward deploying migrations while sharing database schema across projects

How do we at Xelix deploy dozens of Django migrations per month with zero downtime or migration-related errors? Linters are part of the story, but equally we employed a strict database schema separation from code which allows us to forward deploy migrations before any code safely.

At Xelix we are building an accounts payable control centre that is comprised of several modules. For the end user, they access all the modules from a single frontend application written in React. The frontend talks to our primary backend service, the API service, which is written in Python and uses Django as the primary technology for serving HTTP requests. Under the hood, we use PostgreSQL as our database, and we use the django-tenants package to segregate the data of each of our customers in PostgreSQL schemas. Much has been written about this strategy of segregating data, so that part of our setup will not be covered within this blog.

Our SaaS product is made of more than one service. The API service handles all the API requests and performs some of the actions triggered by users, but many parts of the work we do are implemented in separate, independently deployed services. For example, we have a file processing service which handles all of the importing of invoice and vendor data from ERP and Workflow systems. These external services do still connect to the same database.

This setup provides us with a couple of benefits - in technologies used, deployment strategies and development speed. Having the services separate lets us use different technologies as appropriate, for example our API is served by long-running AWS ECS tasks, but the file processing service only runs on-demand in AWS Lambda. We can deploy them independently, a bug in the API would not block a release in file processing and vice versa. And finally, each of our services lives in its own repository, meaning that development speed is improved by only needing to run some tests in the merge queue, fewer conflicts etc.

We have a very active development team that deploys to production every day - new features, bug fixes, and performance improvements, including many schema changes, all while aspiring to a 100% availability at all times and to lowering the amount of bugs as much as possible. In this blog I will cover how we manage database migrations at Xelix, how we make sure that migrations are always safe by forward-deploying them ahead of any service code. This also helps us to use the same DB schema for multiple services connecting to the same database.

Problems we had before the current solution

When we first developed the platform, the database schema and the migrations were defined in our primary API service. On deploy, we would run migrations while also booting new API workers.

Migration Safety

Migration safety is a well known problem in developing Django applications - it boils down to: how do you make sure that you only perform safe database migrations, and how do you ensure that your code only accesses the database columns and tables that actually exist?

For example, if you add a new column to your user table, you need to make sure that:

A read from the table won’t try to read the new column before it definitely exists, i.e. new code works before the migration actually runs

As Xelix signed more customers, more users came onto the platform, and the dev team grew, we hit these issues increasingly often.This is especially where our usage of django-tenants becomes relevant, that package has many benefits, but one of the downsides is increased migration time, as the migrations need to happen for all tenants.

These issues usually resolved themselves quickly, but they still disrupted users whenever they happened, and developers were spending a lot of time putting mitigations in place.

One of them was using Django’s .only() or .defer() on critical paths. That makes Django skip loading certain columns when evaluating querysets. This was useful in both adding new columns (add the new column so new code can handle the column not existing yet), and removing columns (first deploy a defer on deprecated column, so once it’s removed any old code can handle the column already being gone). These were time-costly and didn’t work unless all cases were covered.

Database Schema Drift

Before we introduced the solution described later, we had two services outside the primary API service which were connected to same database, both using Django ORM. The issue was - how do we make sure that changes to the database schema are reflected in those two services? And on top of that, we wanted to add more of these separated services, and with each one the overhead of managing the schema became a headache, occasionally causing bugs. Our original solution was as simple (and dumb) as copy-pasting the relevant Django models to the service repositories whenever it changed in API, pointing to the same database, turning off schema management.

Our database schema and migration setup

Our solution was to extract the entire database schema from the primary API service to an internal Python package (hosted in AWS CodeArtifact, our internal Python package repository), which is now the source of truth for the database schema across all services that talk to the database. This package contains only the database schema (along with Enums for choice fields and Pydantic schemas for any JSON fields), migrations and database/model utils shared across all services.

On each merge to main of that database package, we automatically create a new release and build the Python package (we use CalVer schema, so for example v2026.06.18.0). From there, any service can install the package and use the models from it.

On deployment of the schema to an environment, the migrations run from a small AWS ECS task, which only runs the migrations and exits - it basically just runs python manage.py migrate_schemas (django-tenants’ equivalent of migrate).

How do we make sure migrations are safe

There are two levels of making sure that database migrations are safe. First we need to make sure that the database operations performed in the migrations are safe. We use the django-migration-linter package for linting our database migrations. This package is very handy at detecting incorrect migrations, like adding a non-nullable column, removing columns or changing column or table names. It does not always catch everything, so a core part of it is also instructing our AI review bot to catch unsafe migrations, and the final layer is always human code review, which checks the migration is actually safe.

The second layer is making sure that the migration ran in production before we allow any code that uses it to be deployed to production. Imagine if v2026.06.17.0 was deployed in production, and you deployed the API service which depends on v2026.06.18.0 - all the work we went through to make the migration safe would go up in flames, as the API would try to access something that does not exist in the database yet. We handle this by recording the DB version after running each migration, and having an internal-only API which says what the latest version that ran migrations is. In our deployment process, for each service that uses the DB schema, we figure out what DB schema it needs (by looking at uv.lock) and compare that to the migrated version - and only allow the deployment to continue if the requirement is an older or equal version.

The version check is performed by our internal release management tool called Skytrain - when a deploy is triggered to a service, it verifies the version compatibility and will make the deployment fail if versions are not compatible. This alerts the release managers who can investigate the deployment.

Propagating changes to services

As the DB schema is published as a Python package, we can utilise an existing package management tool for automatically propagating the DB schema to each of our services. Our primary tool for dependency updates on the backend is GitHub’s Dependabot, which we have configured to update the DB schema package every day.

We have an extensive suite of tests in each of our services and GitHub’s merge queue setup as well, so for these DB schema updates (which always contain safe migrations and are already approved) we automatically merge these updates if the tests pass.

Making sure that each service uses as recent a schema as possible is important for some more complicated migrations, like removing columns and tables. For removing anything from the database, a two-stage migration is always needed - first make our Django ORM stop reading the column or table; and only when all services have stopped reading the column or table it can be hard-dropped from the database.

Local development & CI

If a developer wants to test out their new DB schema while working on a service, this is quite easy to do with editable package installs. We use uv for package management, so one just needs to run uv pip install -e <path-to-schema-package> . All changes made in the DB schema are immediately available, so the developer can run the service locally, run tests and anything else without extra effort. The python manage.py migrate_schemas command works as normal for local deployments as well, without any extra setup.

This approach obviously does not work if you want to open a pull request (or a draft) with some yet unreleased changes, and you want the CI to check all the tests with this new database change. For this, we publish pre-release versions for every open PR in the DB schema, which contains the DB schema as of the latest push. When developers want to run CI with some unreleased changes, they simply install that pre-release with uv and push.

Migration reversals

To reverse migrations we can’t easily un-apply migrations already deployed, so instead we reverse migrations by simply adding another migration that reverses it. Django migrations also provide the “replacing functionality”, which means we can also make sure that the incorrect migration does not run on new schemas (as we apply migrations per schema using django-tenants) and in tests.

Imagine you add a migration and try to push it to production in 0010. If that fails, or even worse if it fails only on some schemas, you can reverse it by adding a migration 0011 that fixes the migration and it will only be applied in schemas where 0010 passed. For schemas where the migration failed in the first place, you will add a migration that replaces 0010 and 0011 with a new one using Django’s squashmigrations command. This also makes sure that in tests or in local, only the correct migrations are run.

Results, lessons learned and trade-offs

The implementation of deploying migrations and managing the schema has worked very well. The main two issues we were having with database migration safety and schema drift have been entirely solved. We’ve stopped having migration-related issues when deploying our services, increasing stability and improving the customer experience, and we’ve been able to stop employing so many workarounds to attempt migrations to run safely. The number of services which we deploy independently of the main API service has doubled which allowed us to improve our CI time and increase flexibility when deploying big projects without blocking everything.

Any issues with migrations are also caught much earlier and can be dealt with quicker, and not creating a risk of services running with incorrect DB state, as now their deployments get blocked until migrations pass completely.

The benefits do come at a cost of course, there are trade-offs. The primary one is increased friction making any changes in the database - if as a dev you want to add a column and start using it, you must do it in two PRs, one in schema and the second in the service, when before it only took one. We’ve concluded that it’s worth it for the benefits. Before this implementation, only about 15% of raised pull requests added migrations, meaning the friction is not on every single pull request. On top of that it always forces developers to think about the database schema first, and considering the migration safety. We did some two-stage migrations before the DB schema split anyway (to mitigate the migration issues), so the friction increased just marginally.

Another consideration is custom model querysets and managers, and model properties and methods. With our setup, the DB schema package only contains the models themselves, not any business logic. This can be a little bit annoying, but in a way it actually just accelerated our move towards a more service-layer approach to writing code, where any code is written in functions or classes away from the model layer.

For existing model querysets and managers we’ve managed to develop a dynamic manager methodology, where each service can define its own manager. For existing model properties and methods we monkey-patched the models, which is quite a hacky solution but made the migration much less painful.