Add Poetry to Dockerized Django project with Cookiecutter

In previous post, I just started a django project with cookiecutter. In this post, I will add poetry to the project.

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution.

poetry introduction

Prerequisite

• Python installed. (I am using pyenv to use different versions for other projects. For windows, you can use pyenv-win.)
• Docker and Docker-compose installed. (If you prefer GUI, you can install Docker Desktop, Docker Desktop includes docker compose.)
• Build Django Project with Cookiecutter

Install Poetry

First, you need to install the poetry globally. There are several ways to install poetry. You can see the options on their documentation. I used pipx to install.

pipx install poetry

To check the version of poetry or update, use the below commands.

poetry --version
pipx upgrade poetry

There are different commands to start a new project, or add to the existing project. Later case for us.

Add poetry to the existing project

cd pre-existing-project
poetry init

If you run the command above, you will be asked for configuration of pyproject.toml file like below screenshot.

You can add packages in this setup process, or later add manually.

Hit Enter at last, then it will automatically create pyproject.toml file in your project root directory.

You maybe need virtual environment for your project. you can either create your own, or let poetry to create one.

By default, Poetry creates a virtual environment in {cache-dir}/virtualenvs. You can change the cache-dir value by editing the Poetry configuration. Additionally, you can use the virtualenvs.in-project configuration variable to create virtual environments within your project directory.

In my case, I just run poetry shell and it automatically find the python version and create .venv like screenshot below.

After that, run poetry install to install the dependencies for your project. It will automatically create the poetry.lock file.

Possible Confusion

You are probably used to use pip. Then, you maybe confuse install command for poetry. With poetry, install command reads the pyproject.toml file from the current project, and installs them.

To install the package like pip install poetry uses add command. Like, poetry add <package name>

To learn more commands for poetry, click here.

Update and install all packages via poetry

After I run poetry install I still have errors like the screenshot below. Because I haven’t added any packages on setup process..

First, update the pyproject.toml file. add all the packages you want to install. Here is mine for example.

[tool.poetry]
name = "staff-info"
version = "0.1.0"
description = "gathering staff_info"
authors = ["Jisoo Oh"]
license = "MIT"
readme = "README.md"
packages = [{include = "staff_info"}]

# ==== from base.txt ====
[tool.poetry.dependencies]
python = "^3.11"
python-slugify = "^8.0.1" # https://github.com/un33k/python-slugify
Pillow = "^10.0.0" # https://github.com/python-pillow/Pillow
argon2-cffi = "^23.1.0" # https://github.com/hynek/argon2_cffi
whitenoise = "^6.5.0" # https://github.com/evansd/whitenoise
redis = "^5.0.0" # https://github.com/redis/redis-py
hiredis = "^2.2.3" # https://github.com/redis/hiredis-py
uvicorn = { extras = [
  "standard",
], version = "^0.23.2" } # https://github.com/encode/uvicorn
django = ">=4.2,<4.3" # pyup: < 5.0  # https://www.djangoproject.com/
django-environ = "^0.11.2" # https://github.com/joke2k/django-environ
django-model-utils = "^4.3.1" # https://github.com/jazzband/django-model-utils
django-allauth = "^0.56.1" # https://github.com/pennersr/django-allauth
django-crispy-forms = "^2.0" # https://github.com/django-crispy-forms/django-crispy-forms
crispy-bootstrap5 = "^0.7" # https://github.com/django-crispy-forms/crispy-bootstrap5
django-redis = "^5.3.0" # https://github.com/jazzband/django-redis
djangorestframework = "^3.14.0" # https://github.com/encode/django-rest-framework
django-cors-headers = "^4.2.0" # https://github.com/adamchainz/django-cors-headers
drf-spectacular = "^0.26.4" # https://github.com/tfranzel/drf-spectacular
django-webpack-loader = "^2.0.1" # https://github.com/django-webpack/django-webpack-loader

# ==== from local.txt ====
[tool.poetry.group.dev.dependencies]
Werkzeug = { extras = [
  "watchdog",
], version = "^2.3.7" } # https://github.com/samuelcolvin/watchgod
ipdb = "^0.13.13" # https://github.com/gotcha/ipdb
psycopg = { version = "^3.1.10", extras = ["binary"] } #https://github.com/psycopg/psycopg
watchfiles = "^0.20.0"  # https://github.com/samuelcolvin/watchfiles
mypy = "^1.4.1"  # https://github.com/python/mypy
django-stubs = { version = "^4.2.3", extras = ["compatible-mypy"] } # https://github.com/typeddjango/django-stubs
pytest = "^7.4.2" # https://github.com/pytest-dev/pytest
pytest-sugar = "^0.9.7" # https://github.com/Frozenball/pytest-sugar
djangorestframework-stubs = { version = "^3.14.2", extras = ["compatible-mypy"] } # https://github.com/typeddjango/djangorestframework-stubs
sphinx = "^7.2.5" # https://github.com/sphinx-doc/sphinx
sphinx-autobuild = "^2021.3.14" # https://github.com/GaretJax/sphinx-autobuild
flake8 = "^6.1.0"  # https://github.com/PyCQA/flake8
flake8-isort = "^6.0.0"  # https://github.com/gforcada/flake8-isort
coverage = "^7.3.1" # https://github.com/nedbat/coveragepy
black = "^23.9.1" # https://github.com/psf/black
djlint = "^1.32.1"  # https://github.com/Riverside-Healthcare/djLint
pylint-django = "^2.5.3"  # https://github.com/PyCQA/pylint-django
pre-commit = "^3.4.0"  # https://github.com/pre-commit/pre-commit
factory-boy = "^3.3.0"  # https://github.com/FactoryBoy/factory_boy
django-debug-toolbar = "^4.2.0" # https://github.com/jazzband/django-debug-toolbar
django-extensions = "^3.2.3" # https://github.com/django-extensions/django-extensions
django-coverage-plugin = "^3.1.0" # https://github.com/nedbat/django_coverage_plugin
pytest-django = "^4.5.2" # https://github.com/pytest-dev/pytest-django

# ==== from production.txt ====
[tool.poetry.group.prod.dependencies]
gunicorn = "^21.2.0"  # https://github.com/benoitc/gunicorn
psycopg = { version = "^3.1.10", extras = ["binary"] } #https://github.com/psycopg/psycopg
django-anymail = { extras = ["sendgrid"], version = "^10.1" } # https://github.com/anymail/django-anymail

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# ==== djLint ====
[tool.djlint]
blank_line_after_tag = "load,extends"
close_void_tags = true
format_css = true
format_js = true
# TODO: remove T002 when fixed https://github.com/Riverside-Healthcare/djLint/issues/687
ignore = "H006,H030,H031,T002"
include = "H017,H035"
indent = 2
max_line_length = 119
profile = "django"

[tool.djlint.css]
indent_size = 2

[tool.djlint.js]
indent_size = 2

I am not explaining how it works and why it is structured like that in details. You can see poetry documentation of managing dependencies.

If you run poetry install right away, You will get errors like below.

There are two options to solve this.

1. As you can see from the msg, you can run poetry lock --no-update to overwrite the lock file. Then, run poetry install again.
2. Or you can just simply delete the lock file and run poetry install again.

Then, it will install all the packages.

I still had same pylance error. In my case, I needed to restart the vscode. now it is gone.

Probably, some of you still have pylance error, reportMissingImports. In that case, please check the env list by this command.

poetry env list

You might have the random name of the virtualenv. Try run the below command and restart the vscode.

poetry config virtualenvs.in-project true

Export to requirements files for docker to catch all the packages

This needs to be done whenever you add package with poetry, I guess…… Not sure though…

Probably, when the docker is building, docker is trying to install the packages from the requirements files. base.txt, local.txt, and production.txt for production server. So, we need to export the pyproject.toml to those files. Use the below commands one by one to export them.

poetry export --without-hashes --with dev -f requirements.txt -o requirements/local.txt
poetry export --without-hashes -f requirements.txt -o requirements/base.txt
poetry export --without-hashes --with prod -f requirements.txt -o requirements/production.txt

You will be able to see the requirements files are modified.

Now build docker again.

docker compose -f local.yml build