Why (and how) I use poetry for python package management

I always remember to write about this when I hear someone in a meeting complaining about some broken package in python.

Here I’ll explain some about my experience.

What is poetry?

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

Okay. I copied the definition from their documentation page: https://python-poetry.org/docs/ . There you can understand better about this library and run the usual quick start

Why poetry?

For me, the main purpose of using poetry is related to “stop breaking my build/deploy”. With the pure requirements.txt + pip install we can pin a library version, so it will never update again, but there are problems related to dependencies. If a dependency not mentioned in “requirements.txt” has a newer version, then it’s updated without giving me a heads up.

Oh, but I could export the version of my local virtualenv and pin every single dependency, right? Yes, but we still have a problem there related to “never updating again”…

Why do we hate new versions?

When a program is designed, there’s a tendency of “hoping” that nothing changes, so our job, our deploy and our services are up and running forever without errors…

But we should be happy with newer versions. Bugs and security breaches being fixed for us all the time. And newer features, of course.

So the problem is not newer versions. The problem is newer bugs. Unexpected deploy breaking.

Compatibility between versions

Another good thing about using poetry is the compatibility check between versions. Suppose you’re using pandas 1.0 with numpy 1.19 and it works fine. If numpy updates to a newer version that is not compatible with pandas 1.0 we could have a problem there. But when you run poetry update , all the compatibility tables are checked before changing to newer version. This really helps preventing errors.

Virtual environment + dev libraries

Poetry creates a new environment automatically. You can even set the python version you want to avoid running with some old python versions.

• poetry shell: Create or access(shell) a new environment;
• poetry install: poetry shell + install what’s missing;
• poetry update: Update libraries;
• poetry add: Add new libraries;
• poetry add -dev: Add libraries only used locally or for testing.

When you add new libraries to your poetry configuration, the dev properties allows you to keep some test/dev libraries.

I usually add: pytest, autopep8, etc. Libraries not needed when running my code in production.

Development flow with poetry

Here’s how I (try to) work:

• Install new libraries with poetry;
• I avoid pinning most libraries. Only the core ones that could end up in API and methods changing like “pandas”;
• Poetry install them for me and keeps in a lock file all the versions (including dependencies);
• Use pyproject.toml and poetry.lock in my Dockerfile build. The deploys will never change until I want them to;
• Call “poetry update” frequently in development virtual environment. This way I’m trying to notice newer bugs while I’m working on my code. Not during a random deploy.

Conclusion

• Poetry keep your dependencies version locked — avoid updating by mistake;
• Keep your libraries compatibles — check before update;
• You can update frequently with the benefit of detecting problems while in development environment.
• Easy to install new virtual environments and dev libraries.

I have some examples to share, if you need some help.

Anyway, comment if you remember any other poetry feature that I didn’t mention.

:)