Video coming soon

The recording for this lesson has not been published yet.

What we cover

  • What makemigrations actually writes
  • Defaults, nullable columns and three-step changes
  • Data migrations with RunPython
  • Inspecting SQL with sqlmigrate
  • Squashing and fixing broken migration history

Lesson notes

1. Two truths that must agree

In the tutorial makemigrations and migrate were two magic words you typed after editing models.py. What they really do is keep three things in sync: your model code, the migration history, and the actual database schema.

models.py what you want makemigrations migration files 0001_initial.py 0002_category_tag.py = the "state" migrate database real tables, real columns, real rows applied names recorded in django_migrations edit by hand

Keep the words apart, because every confusing migration error comes from mixing them up:

  • State is what Django believes the schema is, computed by replaying all migration files in order.
  • Database is what is actually there.
  • django_migrations is a real table listing which migrations have already run. That is the only memory Django has.

2. What makemigrations actually writes

Add the Category model from lesson 2.1 and a foreign key on Post, then run:

python manage.py makemigrations blog
Migrations for 'blog':
  blog/migrations/0002_category_post_category.py
    + Create model Category
    + Add field category to post
# blog/migrations/0002_category_post_category.py
import django.db.models.deletion
from django.db import migrations, models


class Migration(migrations.Migration):
    dependencies = [
        ("blog", "0001_initial"),
    ]

    operations = [
        migrations.CreateModel(
            name="Category",
            fields=[
                ("id", models.BigAutoField(auto_created=True, primary_key=True,
                                           serialize=False, verbose_name="ID")),
                ("name", models.CharField(max_length=60, unique=True)),
                ("slug", models.SlugField(max_length=60, unique=True)),
            ],
            options={"verbose_name_plural": "categories", "ordering": ["name"]},
        ),
        migrations.AddField(
            model_name="post",
            name="category",
            field=models.ForeignKey(
                blank=True,
                null=True,
                on_delete=django.db.models.deletion.PROTECT,
                related_name="posts",
                to="blog.category",
            ),
        ),
    ]

Three things are worth noticing. It is plain Python you can edit. It declares a dependencies list, which is how Django orders migrations across apps - numbers alone are not enough. And the field is written out in full, so the file keeps working even after you change models.py again.

3. Read the SQL before you run it

python manage.py sqlmigrate blog 0002
BEGIN;
--
-- Create model Category
--
CREATE TABLE "blog_category" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "name" varchar(60) NOT NULL UNIQUE,
    "slug" varchar(60) NOT NULL UNIQUE
);
--
-- Add field category to post
--
ALTER TABLE "blog_post" ADD COLUMN "category_id" bigint NULL
    REFERENCES "blog_category" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "blog_post_category_id_1f2a3b4c" ON "blog_post" ("category_id");
COMMIT;

Two lessons hide in that output. A foreign key gets an index automatically - you do not add one. And on PostgreSQL some operations take an ACCESS EXCLUSIVE lock, which on a large table means downtime; sqlmigrate is how you find out before your users do.

python manage.py migrate --plan      # what would run, in order
python manage.py showmigrations blog # [X] applied, [ ] pending
python manage.py migrate blog 0001   # roll back to 0001, if the ops are reversible

4. Adding a non-null field to a table with data

Your blog already has posts. Now you want every post to have a slug that is NOT NULL and unique. Do it in one step and the database refuses: what value should the existing rows get? Django asks you interactively, and if you accept a one-off default, the unique constraint breaks on the second row.

The safe pattern is three migrations - and it is worth memorising:

StepMigrationWhy
1 AddField with null=True (no unique yet) a nullable column can always be added instantly
2 RunPython that fills every row you decide the value per row, not one default for all
3 AlterField to null=False, unique=True the constraint can now be satisfied

Step 1 - edit the model, generate, migrate:

class Post(models.Model):
    slug = models.SlugField(max_length=200, null=True, blank=True)

5. Data migrations with RunPython

Create the empty file first, then fill it in. This keeps the data change out of the auto-generated schema migration:

python manage.py makemigrations blog --empty --name backfill_post_slugs
# blog/migrations/0004_backfill_post_slugs.py
from django.db import migrations
from django.utils.text import slugify


def fill_slugs(apps, schema_editor):
    Post = apps.get_model("blog", "Post")
    for post in Post.objects.filter(slug__isnull=True).iterator():
        base = slugify(post.title)[:190] or "post"
        slug = base
        suffix = 2
        while Post.objects.filter(slug=slug).exclude(pk=post.pk).exists():
            slug = f"{base}-{suffix}"
            suffix += 1
        post.slug = slug
        post.save(update_fields=["slug"])


def clear_slugs(apps, schema_editor):
    Post = apps.get_model("blog", "Post")
    Post.objects.update(slug=None)


class Migration(migrations.Migration):
    dependencies = [
        ("blog", "0003_post_slug"),
    ]

    operations = [
        migrations.RunPython(fill_slugs, clear_slugs),
    ]

Always pass a reverse function - migrations.RunPython.noop if there is truly nothing to undo - so migrate blog 0003 still works. Historical models have no custom methods and no custom save(), so any logic you need must be written inside the migration.

Step 3 - tighten the column:

class Post(models.Model):
    slug = models.SlugField(max_length=200, unique=True)
python manage.py makemigrations blog   # 0005_alter_post_slug.py
python manage.py sqlmigrate blog 0005
python manage.py migrate

6. Squashing and repairing history

After a year you have 47 migrations and a fresh test database takes half a minute to build. Squash them into one:

python manage.py squashmigrations blog 0001 0047

Django writes 0001_squashed_0047_....py that replaces the old files. Keep both sets in the repository until every environment has applied past 0047, then delete the originals and remove the replaces attribute. Squashing usually cannot preserve RunPython steps - review the result by hand.

The three failures you will actually meet:

  • Conflicting migrations (two 0007 files). Two branches merged. Run python manage.py makemigrations --merge, which writes a small migration depending on both.
  • "Table already exists". The database is ahead of the history. Mark the migration as done without running it: python manage.py migrate blog 0002 --fake.
  • The whole app is out of sync. Delete nothing in production; instead reset the state with migrate blog zero --fake, then migrate blog --fake-initial so Django adopts the existing tables.

7. Exercise

  1. Add the three-step slug change to Post on a database that already has at least three posts, and confirm no data was lost.
  2. Run python manage.py sqlmigrate blog 0003 and 0005 and write down, in one sentence each, what SQL statement they emit.
  3. Roll the whole app back with python manage.py migrate blog 0002, then forward again. If it fails, find the operation that is not reversible.
  4. Write a data migration that creates an "Uncategorised" category and assigns it to every post whose category is NULL, with a working reverse function.
  5. Add a migrate --check call to your workflow and explain why it belongs in continuous integration.

Further reading

After this lesson you will

  • Read and understand a generated migration file
  • Add non-null fields to a table that has data
  • Move data with a migration instead of by hand