This is the first installment in a series on leveraging pydantic for Django-based projects. Part 2 builds an app on this concept with Docker and conda to show how to align development and production environments; part 3 explores how to deploy a Django app to Heroku.
Django projects used to frustrate me because I lacked a robust and scalable way to add new environments. By bringing together pydantic and Python type hints, I built the powerful foundation I needed.
As described in PEP 484, type hints support static analysis, but these same annotations are also available at runtime. Third-party packages like pydantic offer runtime type checking that uses this additional metadata. Pydantic uses Python type hints to help manage settings metadata and perform runtime data validation.
This pydantic tutorial will show the far-reaching, positive effects of using pydantic settings management with Django.
Our configuration adheres to the best practices described on the Twelve-Factor App website:
- Define nonconstant and secret configurations as environment variables.
- In development environments, define environment variables in a
.env
file and add the.env
to.gitignore
. - Use the cloud provider’s mechanisms to define (secret) environment variables for the QA, staging, and production environments.
- Use a single
settings.py
file that configures itself from the environment variables. - Use pydantic to read, check, validate, and typecast environment variables onto Python variables that define the Django configurations.
Alternatively, some developers create multiple settings files, like settings_dev.py
and settings_prod.py
. Unfortunately, this approach does not scale well. It leads to code duplication, confusion, hard-to-find bugs, and higher maintenance efforts.
Using the aforementioned best practices, adding any number of environments is easy, well-defined, and error-proof. Although we could explore a more complicated environment configuration, we will focus on two for clarity: development and production.
What does this look like in practice?
Pydantic Settings Management and Environment Variables
We now focus on an example in both development and production. We show how each environment configures its settings differently and how pydantic supports each.
Our example application requires a Django-supported database, so we need to store the database connection string. We move the database connection configuration information into an environment variable, DATABASE_URL
, using the Python package dj-database-url. Please note that this variable is of type str
and is formatted as follows:
postgres://{user}:{password}@{hostname}:{port}/{database-name}
mysql://{user}:{password}@{hostname}:{port}/{database-name}
oracle://{user}:{password}@{hostname}:{port}/{database-name}
sqlite:///PATH
In our development environment, we can use a Docker-contained PostgreSQL instance for ease of use, while in our production environment, we will point to a provisioned database service.
Another variable we want to define is a boolean, DEBUG
. The DEBUG flag in Django must never be turned on in a production deployment. It is intended to get additional feedback during development. For example, in debug mode, Django will display detailed error pages when an exception occurs.
Different values for development and production could be defined as follows:
Variable Name | Development | Production |
---|---|---|
DATABASE_URL |
postgres://postgres:mypw@localhost:5432/mydb |
postgres://foo1:foo2@foo3:5432/foo4 |
DEBUG |
True |
False |
We use the pydantic settings management module to manage these different sets of environment variable values depending on the environment.
Preparatory Steps
To put this into practice, we start configuring our development environment by creating our single .env
file with this content:
DATABASE_URL=postgres://postgres:mypw@localhost:5432/mydb
DEBUG=True
Next, we add the .env
file to the project’s .gitignore
file. The .gitignore
file avoids saving potentially sensitive information in source control.
Whereas this approach works well in our development environment, our production environment specification uses a different mechanism. Our best practices dictate that production environment variables use environment secrets. For example, on Heroku, these secrets are called Config Vars and are configured through the Heroku Dashboard. They are made available to the deployed application as environment variables:
After that, we need to adjust the application’s configuration to read these values from either environment automatically.
Configuring Django’s settings.py
Let’s start with a new Django project to provide the essential structure for our example. We scaffold a new Django project with the following terminal command:
$ django-admin startproject mysite
We now have a basic project structure for the mysite
project. The project’s file structure is as follows:
mysite/
manage.py
mysite/
__init__.py
settings.py
urls.py
asgi.py
wsgi.py
The settings.py
file contains boilerplate code that allows us to manage the application configuration. It has many predefined default settings that we must adjust to the relevant environment.
To manage these application settings using environment variables and pydantic, add this code to the top of the settings.py
file:
import os
from pathlib import Path
from pydantic import (
BaseSettings,
PostgresDsn,
EmailStr,
HttpUrl,
)
import dj_database_url
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
class SettingsFromEnvironment(BaseSettings):
"""Defines environment variables with their types and optional defaults"""
DATABASE_URL: PostgresDsn
DEBUG: bool = False
class Config:
"""Defines configuration for pydantic environment loading"""
env_file = str(BASE_DIR / ".env")
case_sensitive = True
config = SettingsFromEnvironment()
os.environ["DATABASE_URL"] = config.DATABASE_URL
DATABASES = {
"default": dj_database_url.config(conn_max_age=600, ssl_require=True)
}
DEBUG = config.DEBUG
This code does the following:
- Defines a class
SettingsFromEnvironment
, inheriting from pydantic’sBaseSettings
class. - Defines
DATABASE_URL
andDEBUG
, setting their type and optional default using Python type hints. - Defines a class
Config
telling pydantic to look for the variables in a.env
file if not present in the system’s environment variables. - Instantiates the
Config
class into the objectconfig
; the desired variables become available asconfig.DATABASE_URL
andconfig.DEBUG
. - Defines the regular Django variables
DATABASES
andDEBUG
from theseconfig
members.
The same code runs in all environments, and pydantic takes care of the following:
- It looks for the environment variables
DATABASE_URL
andDEBUG
.- If defined as environment variables, like in production, it will use those.
- Otherwise, it pulls those values from the
.env
file. - If it doesn’t find a value, it will do the following:
- For
DATABASE_URL
, it throws an error. - For
DEBUG
, it assigns a default value ofFalse
.
- For
- If it finds an environment variable, it will check field types and give an error if either of them are wrong:
- For
DATABASE_URL
, it verifies that its field type is aPostgresDsn
-style URL. - For
DEBUG
, it verifies that its field type is a valid, nonstrict pydantic boolean.
- For
Please note the explicit setting of the operating system’s environment variable from the configuration value for DATABASE_URL
. It may seem redundant to set os.environ["DATABASE_URL"] = config.DATABASE_URL
because DATABASE_URL
is already defined as an external environment variable. However, this allows pydantic to parse, check, and validate this variable. If the environment variable DATABASE_URL
is missing or formatted incorrectly, pydantic will give a clear error message. These error checks are invaluable as the application moves from development to subsequent environments.
If a variable is not defined, either a default will be assigned or an error will prompt for it to be defined. Any generated prompts also detail the desired variable type. A side benefit of these checks is that new team members and DevOps engineers more easily discover which variables need to be defined. This avoids the hard-to-find issues that result when the application runs without all the variables defined.
That’s it. The application now has a maintainable, settings-management implementation using a single version of settings.py
. The beauty of this approach is that it allows us to specify the correct environment variables in a .env
file or any other desired means available through our hosting environment.
The Scalable Path
I have been using Django settings management with pydantic runtime typing in all of my Django projects. I have found it leads to smaller, more maintainable codebases. It also provides a well-structured, self-documenting, and scalable approach to adding new environments.
The next article in this series is a step-by-step tutorial on building a Django application from scratch, with pydantic settings management; we then cover deploying it to Heroku.
The Toptal Engineering Blog extends its gratitude to Stephen Davidson for reviewing the code samples presented in this article.
An earlier version of this article emphasized a specific Python release. Since Python has supported type hints for several years, we have generalized the introductory text by removing this version reference.
Further Reading on the Toptal Engineering Blog:
Understanding the basics
Pydantic is a Python package that enforces type hints at runtime.
Pydantic provides data validation and settings management.
Yes, you should use Python type hints because your code becomes self-documenting. You can enforce static type safety with mypy and runtime type safety with pydantic.
Type hints do not make Python faster or slower, but they will speed up your development process by avoiding bugs and hard-to-find runtime errors.
Comments