Django 2026-04-04

Local testing of Celery and Redis

Testing Celery and Redis locally is a crucial step in ensuring your background tasks and scheduled jobs are working correctly before deploying to production. In this post, we'll walk through how to set up a local testing environment using four separate terminals to monitor each component of the system.

What is Celery and Redis?

Before we dive into the testing procedure, let's briefly touch upon what these technologies are and why they are often used together.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. It allows you to run time-consuming tasks (like sending emails, image processing, or data scraping) in the background, keeping your web application responsive.

Redis

Redis (Remote Dictionary Server) is an open-source, in-memory data structure store, used as a database, cache, and message broker. In our Celery setup, Redis acts as the broker, acting as the intermediary between the Django application (which sends tasks) and the Celery workers (which execute tasks).

Python Dependency Management: uv vs pip

In this project, we use uv to manage our Python environment and dependencies. If you've been using pip, you might wonder what the difference is.

uv is an extremely fast Python package installer and resolver, written in Rust. It's designed as a drop-in replacement for pip, pip-tools, and virtualenv.

Feature	pip	uv
Speed	Standard	Up to 10-100x faster than pip
Language	Python	Rust
Environment Mgmt	Separate (virtualenv)	Built-in
Resolution	Standard	Advanced dependency resolution

The Architecture of Our Local Setup

To have a fully functional Celery environment, we need four main components working together:

Component	Role	Requirement
Redis	The Message Broker	Stores the tasks until a worker is ready to process them.
Celery Worker	The Consumer	Actually executes the tasks.
Celery Beat	The Scheduler	Sends tasks to the broker at scheduled intervals.
Django Server	The Producer	Where you trigger tasks or monitor them through the UI.

Step-by-Step Guide: Using Four Terminals

Open four terminal windows or tabs to run the following commands. This allows you to see the logs from each service in real-time.

1 Terminal 1: Redis Broker

Start the Redis server using Docker.

sudo docker run -it --rm --name redis -p 6379:6379 redis:8.6.2-alpine

2 Terminal 2: Celery Worker

Start the worker to consume and execute tasks.

celery -A core worker -l info

3 Terminal 3: Celery Beat

Start the scheduler for periodic tasks.

celery -A core beat -l info

4 Terminal 4: Django App

Run your Django development server.

uv run manage.py runserver --settings=core.dev-settings

Practical Example: task_send_email_about_ebook

In our tasks.py file, we have a task that fetches the title of a free ebook from PacktPub and sends an email. This is a perfect candidate for testing locally.


@app.task
def task_send_email_about_ebook():
    page = requests.get(FREE_BOOK_PACKT_URL)
    soup = BeautifulSoup(page.content, "html.parser")
    try:
        book_title = (
            soup.find("h3", attrs={"class": "product-info__title"})
            .text.split("-", 1)[1]
            .strip()
        )
    except AttributeError:
        book_title = "Something is wrong with the book's page!!!!"

    subject = "Your free e-book from PacktPub is available!"
    message = f"Your new e-book is '{book_title}'. \n"
    message += (
        f"Book is available at <a href='{FREE_BOOK_PACKT_URL}'>Free Learning</a>."
    )
    email = EmailMessage(
        subject, message, "info@scientificdev.net", ["artur.zacniewski@proton.me"]
    )
    email.send(fail_silently=False)

Workflow Diagram

Verification

When you run these four terminals, you should see:

In Terminal 2 (Worker): Logs showing the task frontend.tasks.task_send_email_about_ebook being received and executed.
In Terminal 3 (Beat): Logs showing that the task is being triggered at the scheduled time.
In Terminal 1 (Redis): You might see activity logs depending on the Redis configuration, but it's mainly there to facilitate the message transfer.