Ubuntu
python-virtualenv package

virtualenv on xenial: easy_install syntax error

Bug #1912248 reported by Peter Sabaini on 2021-01-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	python-virtualenv (Ubuntu)	Fix Released	Undecided	Unassigned
	Xenial	Fix Released	Medium	Stefano Rivera

Bug Description

[Impact]

* Since Xenial has released, setuptools and pip have dropped support for
   Python 3.5 (and 2.7), upstream.
* Virtualenv installs the latest upstream pip and setuptools during
   virtualenv creation, from the Internet.
* Python packages gained a mechanism to declare python version support,
   but this support arrived since Xenial, so virtualenv can't know which
   versions of pip and setuptools are safe to install.

[Test Case]

# apt install python python3 virtualenv
$ virtualenv -p python3 ve3
$ ve3/bin/python -m pip install 'world < 4'

This should execute without error.

$ virtualenv -p python2 ve2
$ ve2/bin/python -m pip install 'world < 4'

And this should execute without error.

[Where problems could occur]

* Anyone who has discovered this broken behaviour could be suprised by the change. But they'd probably worked-around it by using venv or --no-download.

[Original Bug Report]

As of commit[0] setuptools appear to be incompatible with Python3.5 as shipped by xenial.

This makes a virtualenv for Python3 fail with a syntax error:

# virtualenv -ppython3 venv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /root/venv/bin/python3
Also creating executable in /root/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

# venv/bin/pip install pyyaml
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove supp
ort for this functionality.
Collecting pyyaml
  Using cached PyYAML-5.3.1.tar.gz (269 kB)
Building wheels for collected packages: pyyaml
  Building wheel for pyyaml (setup.py) ... error
  ...

    File "/usr/lib/python3.5/distutils/command/install.py", line 595, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/root/venv/lib/python3.5/site-packages/setuptools/command/install_scripts.py", line 17, in run
      import setuptools.command.easy_install as ei
    File "/root/venv/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 719
      ):
      ^
  SyntaxError: invalid syntax
  ----------------------------------------
  ERROR: Failed building wheel for pyyaml

# lsb_release -r
Release: 16.04

# venv/bin/python3 --version
Python 3.5.2

As Python3.5 is EOL we might need to pin setuptools for virtualenv to a lower version

[0] https://github.com/pypa/setuptools/commit/fc891f5cf6d93ad533e2afb5e15a2952408ab358

See original description

Tags:

Revision history for this message

Philip Roche (philroche) wrote on 2021-01-20:

This is being tracked in setuptools upstream too

https://github.com/pypa/setuptools/issues/2541

The cause is that python-virtualenv is pulling in setuptools 51.3.3

`virtualenv -p /usr/bin/python3 venv/py3 --verbose`

```
....snip...
Installing setuptools, pkg_resources, pip, wheel...
  Collecting setuptools
    Using cached https://files.pythonhosted.org/packages/b2/81/509db0082c0d2ca2af307c6652ea422865de1f83c14b1e1f3549e415cfac/setuptools-51.3.3-py3-none-any.whl
  Collecting pkg_resources
  Collecting pip
    Using cached https://files.pythonhosted.org/packages/54/eb/4a3642e971f404d69d4f6fa3885559d67562801b99d7592487f1ecc4e017/pip-20.3.3-py2.py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
  Installing collected packages: setuptools, pkg-resources, pip, wheel
  Successfully installed pip-20.3.3 pkg-resources-0.0.0 setuptools-51.3.3 wheel-0.36.2
...snip...
```

But 51.3.0 dropped support for python 3.5 which is what ships with Xenial.

The fix would be to pin setuptools to `<51.3.0`.

Revision history for this message

Philip Roche (philroche) wrote on 2021-01-20:

I have found the following workaround on Ubuntu 16.04 Xenial

Do not install setuptools on virtualenv creation:

`virtualenv -p /usr/bin/python3 venv/py3 --verbose --no-setuptools`

Install a supported setuptools version after virtualenv creation and activation:

`python3 -m pip install --upgrade 'setuptools; python_version >= "3.6"' 'setuptools<51.3.0; python_version < "3.6" and python_version >= "3.0"'`

> Ignoring setuptools: markers 'python_version >= "3.6"' don't match your environment
> Ignoring setuptools: markers 'python_version < "3.0"' don't match your environment
> Collecting setuptools<51.3.0
> Using cached setuptools-50.3.2-py3-none-any.whl (785 kB)
> Installing collected packages: setuptools
> Successfully installed setuptools-50.3.2

or by adding the following to your requirements.txt

```
setuptools; python_version >= "3.6"
setuptools<51.3.0; python_version < "3.6" and python_version >= "3.0"
```

This results in `python3 -m pip list`:

> pip 20.3.3
> setuptools 50.3.2
> wheel 0.36.2

Stefano Rivera (stefanor) on 2021-01-21

Changed in python-virtualenv (Ubuntu):
status:	New → Fix Released

Revision history for this message

Stefano Rivera (stefanor) wrote on 2021-01-21:

The issue is that virtualenv bundles pip 8.1.1, which doesn't know about python_version. It then tries to upgrade to the latest pip and setuptools, but that version isn't Python 3.5 compatible.

If you build the virtualenv with --no-download, or use venv instead of virtualenv, you'll get just pip 8.1.1, and setuptools 20.7.0. You can then explicitly upgrade them to setuptools < 51.3, as you describe.

We could add a hack to virtualenv to specify the setuptools < 51.3 restriction.
But given Ubuntu 18.04 is almost EOL, I'm tempted to just leave this one be. It has a workaround...

Stefano Rivera (stefanor) on 2021-01-21

Changed in python-virtualenv (Ubuntu Xenial):
status:	New → Confirmed
importance:	Undecided → Medium

Revision history for this message

Philip Roche (philroche) wrote on 2021-01-21:

@stefanor

Thanks for the explanation.

I have a one liner workaround now:

```
virtualenv -p /usr/bin/python3 venv/py3 --verbose --no-download && . venv/py3/bin/activate && python3 -m pip install --upgrade 'setuptools; python_version >= "3.6"' 'setuptools<51.3.0; python_version < "3.6" and python_version >= "3.0"' pip wheel && python3 -m pip list
```

Which uses the bundled pip, wheel and setuptools initially but then upgrades pip, wheel and setuptools using environment markers to ensure the supported version is installed.

I do question the EOL argument. Xenial 16.04 is nearly at the end of standard support yes, but there will be many users continuing to use Xenial 16.04 with ESM for another five years.

Revision history for this message

Stefano Rivera (stefanor) wrote on 2021-01-27:

python-virtualenv_15.0.1+ds-3ubuntu1.1.debdiff Edit (2.5 KiB, text/plain)

description:	updated
Changed in python-virtualenv (Ubuntu Xenial):
status:	Confirmed → In Progress
assignee:	nobody → Stefano Rivera (stefanor)

Revision history for this message

Brian Murray (brian-murray) wrote on 2021-02-16: Please test proposed package

Hello Peter, or anyone else affected,

Accepted python-virtualenv into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-virtualenv/15.0.1+ds-3ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-virtualenv (Ubuntu Xenial):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-xenial

Revision history for this message

Philip Roche (philroche) wrote on 2021-02-24:

I have verified fix in amd64 xenial VM.

Test case:

* `sudo apt install python3-virtualenv virtualenv python-virtualenv`
* Confirm that any `pip` operations fail as per bug description
  * `virtualenv -p /usr/bin/python3 venv/py3 --verbose`
  * `source ./venv/py3/bin/activate`
  * `pip list`
* Enable proposed as per https://wiki.ubuntu.com/Testing/EnableProposed
* Upgrade packages using `sudo apt install python3-virtualenv virtualenv python-virtualenv`
* Confirm that any `pip` operations succeed
  * `virtualenv -p /usr/bin/python3 venv/py3-verification --verbose`
    * I also confirm here that the pip and setup tools were being pinned here
      > Installing setuptools < 51.3, pkg_resources, pip < 21, wheel...
  * `source ./venv/py3-verification/bin/activate`
  * `pip list`

Tags added verification-done-xenial verification-done

tags:

added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial

Revision history for this message

Philip Roche (philroche) wrote on 2021-02-24:

I also verified the fix with a python2 virtualenv.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2021-03-02:

This bug was fixed in the package python-virtualenv - 15.0.1+ds-3ubuntu1.1

---------------
python-virtualenv (15.0.1+ds-3ubuntu1.1) xenial; urgency=medium

* Pin pip and setuptools to versions compatible with Python 2.7 and 3.5.
(LP: #1912248)

-- Stefano Rivera <email address hidden> Wed, 27 Jan 2021 11:29:14 -0700

Changed in python-virtualenv (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Revision history for this message

Brian Murray (brian-murray) wrote on 2021-03-02: Update Released

#10

The verification of the Stable Release Update for python-virtualenv has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.