optional dependencies listed in setup.py as required

Bug #1775654 reported by Eli Schwartz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Rapid Photo Downloader
Fix Released
High
Damon Lynch

Bug Description

colorlog and pyprind are listed in the README.rst as "optional, but highly recommended", and come with try/except blocks in the codebase to work even if they're not installed.

However, it's entirely impossible to run it in the first place, if either of those two are not installed. This is because they're listed in setup.py as install_requires, which are mandatory, not extras_require, which are used for declaring optional recommendations which enhance the software. See https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

Since the program is started using setuptools entry_points generated scripts, this setuptools metadata is enforced at runtime no matter what.

...

Aside: you're currently misusing extras_require for dependencies that are only needed for given python versions... but you really should use the current best practice which is environment markers, as defined in https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies

Related branches

Eli Schwartz (eschwartz)
description: updated
Damon Lynch (dlynch3)
Changed in rapid:
status: New → Triaged
milestone: none → 0.9.10
importance: Undecided → High
assignee: nobody → Damon Lynch (dlynch3)
Revision history for this message
Eli Schwartz (eschwartz) wrote :

Also PyGObject isn't listed as a dependency in the setup.py at all (though it is in the README and it's obvious that to use the typelibs listed as dependencies you'll need PyGObject too).

Revision history for this message
Damon Lynch (dlynch3) wrote :

I'll have to look more into PyGObject. There was a reason why it wasn't in the dependencies a few years ago, although now I don't recall what it was. Perhaps due to the then lack of availability on pip? I really don't remember the details. But I do remember it was a bit tricky.

Damon Lynch (dlynch3)
Changed in rapid:
status: Triaged → Fix Committed
Damon Lynch (dlynch3)
Changed in rapid:
status: Fix Committed → Fix Released
Revision history for this message
Eli Schwartz (eschwartz) wrote :

https://bazaar.launchpad.net/~dlynch3/rapid/zeromq_pyqt/revision/1077

You moved colour to extras_require instead of colorlog, but colour is actually very much needed, and colorlog is optional. Oops?

Revision history for this message
Damon Lynch (dlynch3) wrote :

Yes looks like it. What is the actual practical outcome of this problem? What precisely enforces the metadata? Very recent versions of python or setuptools?

I ask because I'd like to avoid making a release with only this fix.

Revision history for this message
Eli Schwartz (eschwartz) wrote :

The dependencies are generated by setuptools, and encoded in the *.egg-info/ metadata directory in the requires.txt file.

The practical outcome is that

1) when you use pkg_resources to check whether the dependencies are met, if you installed colour because you know the codebase uses it, but did not install colorlog (because the readme says it is optional) then it throws an error at you. The generated entry_points scripts in /usr/bin/ use pkg_resources.load_entry_point(), and refuse to run with the error:

pkg_resources.DistributionNotFound: The 'colorlog' distribution was not found and is required by rapid-photo-downloader

2) if you use pip rather than your distribution package manager to install rapid-photo-downloader, it will automatically download and install dependencies from install_requires, but not from extras_require unless you specify e.g.

pip install /path/to/rapid-photo-downloader.tar.gz[progress_bar,color_ouput]

Since it doesn't install extras by default, that means trying to `import colour` will fail with a ModuleNotFoundError unless the user has already installed it (colour) for some other reason.

Revision history for this message
Damon Lynch (dlynch3) wrote :

Thank you for your explanation.

You write "when you use pkg_resources to check whether the dependencies are met". Who uses that? Anyone other than distro packagers?

Please note, for anyone who wants to use pip on the command line, I provide the install.py that takes care of installing all the dependencies (python and non-python) for supported distros. The package color is also found in requirements.txt

In any case I have corrected the mistake in the source. I don't know when 0.9.11 will be released.

Revision history for this message
Eli Schwartz (eschwartz) wrote :

Well, as I said, this pkg_resources functionality is the low-level method by which setup.py creates the executable launcher. it's just a matter of the dependency failing to properly be optional.

#2 is the really troubling issue, since it means that depending on how people install it, they might not get the unstated dependency, and if they manually uninstalled it because e.g. `pip show colour` reported that it was not "Required-by" any other modules, then they will get an import error. (Not only is a pkg_resources.DistributionNotFound error more clear, it also avoids the issue in the first place because it would report "Required-by: rapid-photo-downloader".)

But granted that users who simply installed with install.py and don't ever look for unused modules to delete, will never run into this issue.

Damon Lynch (dlynch3)
Changed in rapid:
status: Fix Released → In Progress
milestone: 0.9.10 → 0.9.11
status: In Progress → Fix Committed
Damon Lynch (dlynch3)
Changed in rapid:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.