For much of its history, Python had no single agreed way to build, install, and distribute software, and the resulting tangle of overlapping tools became a recurring source of frustration. Over the years developers had to reckon with distutils, setuptools, easy_install, pip, the egg and wheel distribution formats, virtualenv for isolation, conda as a separate cross-language world, and later contenders like Poetry and Pipenv. Each solved a real problem, but together they meant newcomers faced a confusing pile of tools that did similar things in incompatible ways.
A core technical knot was that building a package required running its setup.py file, which was ordinary Python code. PEP 518 described the catch-22 directly: “You can’t execute a setup.py file without knowing its dependencies, but currently there is no standard way to know what those dependencies are in an automated fashion without executing the setup.py file where that information is stored” (https://peps.python.org/pep-0518/). There was no reliable way to know what a build needed before doing the build.
A related problem was that setuptools had become an unavoidable gatekeeper. PEP 517 noted that even though distutils and setuptools had limitations, “it’s very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like pip”, and set the goal of getting “distutils-sig out of the business of being a gatekeeper for Python build systems” (https://peps.python.org/pep-0517/).
The fix came as a series of standards centered on a single declarative file, pyproject.toml. PEP 518 added a [build-system] table so build dependencies could be declared without executing code, PEP 517 defined a build-backend interface so any tool could build packages, and PEP 621 standardized project metadata in a tool-agnostic [project] table to make “transitions between build systems easier” (https://peps.python.org/pep-0621/). The packaging story is not finished, but the long-criticized fragmentation has been converging, file by file, on a shared standard.