Building a Python Project | Tim Watson's Blog

Open Table of contents

Aim
Virtual Environments
Setuptools
- VSCode extensions
- One file to rule them all
File Layout
- Entry points
Other tools in the toml file
pre-commit
Wrapping Up

Aim

Creating Python projects that are clean and import correctly should be relatively easy. But the tools and other dependencies around this have changed somewhat. So let’s step through creating a really simple project.

Virtual Environments

First things first, whenever interacting with Python it is pretty much a given that a virtual environment should be used. So let’s create one in our working directory.

python -m venv venv

Setuptools

In previous versions of Python there have been tools such as distutils and files like setup.py that define how projects are installed or packaged together. In fact working out the best way of creating a solution sometimes isn’t always entirely obvious as the landscape has changed quite a lot. With Python 3.12, distutils was removed so using one of the newer methods is now required (explained in the YouTube guide linked above). The preferred method will likely use a pyproject.toml file and the advantage of this (amongst others) is most of the setup and project related tools can all be setup and configured from this one centralised file.

[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

To start with, the file itself is effectively just a collection of definitions and any tools you are going to use can read it to find out what options you have specified. For build and project related settings we will define those using the appropriate definitions inside square brackets, see above. The above snippet defines that our build system will use setuptools; in older versions of Python this would have required other pip installs/dependencies to be met.

[project]
name = "example_project"
description = "creating an example python project template"
version = "0.0.1"
requires-python = ">=3.12.0"
dynamic = ["dependencies"]
readme = "README.md"

We can then set up a “project”, give it a name, describe it and give it other keywords. See the link above for other useful keywords.

VSCode extensions

When writing toml files, especially in vscode, it might be worth installing parsers such as Even Better TOML that will give hints and syntax highlighting.

One file to rule them all

Notice above where the line has dynamic = ["dependencies"], this is for a subsequent line:

[tool.setuptools.dynamic]
dependencies = { file = ["requirements.txt"] }

As you might expect, this works as in previous setup.py or projects where it specifies a requirements file to install from. So by defining the project as dynamic you can dynamically specify the requirements file.

However, you can “just” specify the requirements inline:

[project]
name = "example_project"
description = "creating an example python project template"
version = "0.0.1"
requires-python = ">=3.12.0"
#dynamic = ["dependencies"]
readme = "README.md"
dependencies = [
    "pandas",
]

See the writing pyproject toml link above, this gives another example with version pinning.

File Layout

Having gone over the quick basics of the pyproject.toml file, let’s create a project.

├───example_python_project
│   └───src
|        └───example_project
|             └───other_folder
|                    └───example_from_other.py
|             └───utilities
|                    └───example.py
|             ├───__main__.py
├───venv
├───pyproject.toml
├───README.md

In this example, like in our examples of a pyproject.toml above, our project is called “example_project”. This means that when we install this project the build system will look for a project with that name. In the above videos, you can either define your source folder as your project name or you can put your project folder inside a src folder, depending on how you want to layout your code. Following the above layout you will need to define a further option in your toml file:

[tool.setuptools.packages.find]
where = ["src"]

Now setuptools will look for our project (“example_project”) inside the src directory.

Entry points

Notice the file __main__.py, this can be used in different ways to invoke your Python project:

python -m example_project # invoke the module via PATH
python example_project # invoke it from current working directory

However, if you don’t have this file, you will get an error such as:

No module named example_project.__main__; 'example_project' is a package and cannot be directly executed

Depending on how you want to setup your project or what it is doing, you may not run from a __main__.py file. To get around this, you can define an entry point in an __init__.py file:

# Inside __init__.py
from example_project import main
main.main()

But this gets us back to adding more files and making the project more complicated. It also means that you will likely not be able to invoke the code via the python -m... syntax above. Instead, I believe the best way of formatting projects is to stick with the __main__ convention or only define specific entry points:

[project.scripts]
example = "example_project.__main__:main"

The above allows for the code to be ran on the command line via: “example”.

Other tools in the toml file

As aluded to earlier, the pyproject.toml can be used by a range of tools to define project specific rules or options. An example would be:

[tool.black]
line-length = 120

[tool.pylint]
max-line-length = 120

[tool.tox]
legacy_tox_ini = """
[tox]
envlist =
    py312

[testenv]
deps = 
    black
commands = 
    black . --check
"""

The above defines line lengths for two tools (black and pylint) and then for a testing tool tox it defines a Python environment to create and then what commands to run. In this case just black. As is the pattern, the tox configuration can either be defined in a separate file or inline/in the toml file by using the keyword: legacy_tox_ini. According to the docs this will be updated in the future.

pre-commit

Finally, one of the most if not the most important tool in a Python project (and any project) is most likely pre-commit. Code sanity or quality checks that should be ran before any code is pushed remotely, are done automatically by pre-commit without needing to be invoked manually.

Setting up pre-commit

The first step is that your Python virtual environment needs to have pre-commit. This should be a simple update for our toml file dependencies section:

dependencies = [
    "pre-commit"
]

Next, we need a .pre-commit-config.yaml file to define what checks to run:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-toml
      - id: check-yaml
      - id: end-of-file-fixer
      - id: trailing-whitespace

  - repo: https://github.com/psf/black-pre-commit-mirror
    rev: 24.8.0
    hooks:
      - id: black
        language_version: python3.12

The example in the above guide gives some pretty good and generic project checkers as well as this link explains how to add black to pre-commit. Above we have added these checks to a pre-commit-config.yaml file.

The only thing we need to run now is pre-commit install and this will create and install the relevant folders inside the .git folder.

After doing this, running a git commit command should invoke pre-commit and anything that goes wrong should get flagged…

check toml...............................................................Passed
check yaml...............................................................Passed
fix end of files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing .pre-commit-config.yaml
Fixing pyproject.toml

trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing .pre-commit-config.yaml
Fixing pyproject.toml

black................................................(no files to check)Skipped

Depending on the hooks used, some will attempt to fix the issues themselves but once the issues are fixed and another commit is ran it should pass:

(venv) PS ...\example_python_project> git commit -m "add pre-commit"
check toml...............................................................Passed
check yaml...............................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
black................................................(no files to check)Skipped
[main 7539de2] add pre-commit
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100644 .pre-commit-config.yaml

Wrapping Up

With the newer versions of Python, using IDE tools and focusing on utilising the pyproject.toml file to centralise configuration, Python projects can be relatively easily set up and distributed reliably. Other tools can then be added which will aid in code quality such as pre-commit. Further to what was shown here, it should be relatively easy to add a testing folder and extend the project using tox (for example).

The example project code is here.