git-ubuntu

Hirsute: FTBFS on master branch: /usr/lib/python3.9/tempfile.py:607: in getattr: KeyError: 'file'

Bug #1924983 reported by You-Sheng Yang on 2021-04-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	git-ubuntu	Fix Released	Low	Robie Basak

Bug Description

$ git log -1 --oneline
7e9ed50 (HEAD -> master, upstream/master, upstream/HEAD) add libfabric to whitelist
$ debuild -us -uc -b
...
I: pybuild base:232: cd /home/vicamo/.canonical/tools/usd-importer/.pybuild/cpython3_3.9/build; python3.9 -m pytest
============================= test session starts ==============================
platform linux -- Python 3.9.4, pytest-6.0.2, py-1.10.0, pluggy-0.13.0
rootdir: /home/vicamo/.canonical/tools/usd-importer, configfile: pytest.ini
collected 324 items / 1 error / 323 selected

==================================== ERRORS ====================================
___ ERROR collecting .pybuild/cpython3_3.9/build/gitubuntu/apt_repo_test.py ____
/usr/lib/python3/dist-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/lib/python3/dist-packages/pluggy/manager.py:92: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python3/dist-packages/pluggy/manager.py:83: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
/usr/lib/python3/dist-packages/_pytest/python.py:246: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
/usr/lib/python3/dist-packages/_pytest/python.py:454: in _genfunctions
    self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
/usr/lib/python3/dist-packages/pluggy/hooks.py:324: in call_extra
    return self(**kwargs)
/usr/lib/python3/dist-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/lib/python3/dist-packages/pluggy/manager.py:92: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python3/dist-packages/pluggy/manager.py:83: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
/usr/lib/python3/dist-packages/_pytest/python.py:137: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker) # type: ignore[misc]
/usr/lib/python3/dist-packages/_pytest/python.py:1007: in parametrize
    ids = self._resolve_arg_ids(
/usr/lib/python3/dist-packages/_pytest/python.py:1069: in _resolve_arg_ids
    return idmaker(argnames, parameters, idfn, ids_, self.config, nodeid=nodeid)
/usr/lib/python3/dist-packages/_pytest/python.py:1299: in idmaker
    resolved_ids = [
/usr/lib/python3/dist-packages/_pytest/python.py:1300: in <listcomp>
    _idvalset(
/usr/lib/python3/dist-packages/_pytest/python.py:1282: in _idvalset
    this_id = [
/usr/lib/python3/dist-packages/_pytest/python.py:1283: in <listcomp>
    _idval(val, argname, idx, idfn, nodeid=nodeid, config=config)
/usr/lib/python3/dist-packages/_pytest/python.py:1262: in _idval
    elif isinstance(getattr(val, "__name__", None), str):
/usr/lib/python3.9/tempfile.py:607: in __getattr__
    file = self.__dict__['file']
E KeyError: 'file'
=========================== short test summary info ============================
ERROR gitubuntu/apt_repo_test.py - KeyError: 'file'
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.82s ===============================
E: pybuild pybuild:353: test: plugin distutils failed with: exit code=2: cd /home/vicamo/.canonical/tools/usd-importer/.pybuild/cpython3_3.9/build; python3.9 -m pytest
dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p 3.9 returned exit code 13
make: *** [debian/rules:4: build] Error 25
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui -b failed
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu65
Architecture: amd64
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/controlC1: vicamo 2225 F.... pulseaudio
/dev/snd/controlC0: vicamo 2225 F.... pulseaudio
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 21.04
InstallationDate: Installed on 2019-09-28 (568 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Alpha amd64 (20190923)
MachineType: Apple Inc. MacBookPro11,1
NonfreeKernelModules: wl
Package: linux (not installed)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-5.12.0-7-generic root=UUID=38c16714-8883-4c88-baae-df71ffa89972 ro rootflags=subvol=@ quiet splash crashkernel=512M-:192M vt.handoff=7
ProcVersionSignature: Ubuntu 5.12.0-7.7-generic 5.12.0-rc7
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-5.12.0-7-generic N/A
linux-backports-modules-5.12.0-7-generic N/A
linux-firmware 1.197
Tags: hirsute
Uname: Linux 5.12.0-7-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 06/13/2019
dmi.bios.release: 0.1
dmi.bios.vendor: Apple Inc.
dmi.bios.version: 156.0.0.0.0
dmi.board.asset.tag: Base Board Asset Tag#
dmi.board.name: Mac-189A3D4F975D5FFC
dmi.board.vendor: Apple Inc.
dmi.board.version: MacBookPro11,1
dmi.chassis.type: 10
dmi.chassis.vendor: Apple Inc.
dmi.chassis.version: Mac-189A3D4F975D5FFC
dmi.modalias: dmi:bvnAppleInc.:bvr156.0.0.0.0:bd06/13/2019:br0.1:svnAppleInc.:pnMacBookPro11,1:pvr1.0:rvnAppleInc.:rnMac-189A3D4F975D5FFC:rvrMacBookPro11,1:cvnAppleInc.:ct10:cvrMac-189A3D4F975D5FFC:
dmi.product.family: Mac
dmi.product.name: MacBookPro11,1
dmi.product.sku: System SKU#
dmi.product.version: 1.0
dmi.sys.vendor: Apple Inc.

See original description

Tags:

Related branches

~racb/git-ubuntu:lp1924983

Merged into git-ubuntu:master at revision 44133960f62896da5193392e4b47a8ae2f7b531b

Server Team CI bot: Approve (continuous-integration) on 2021-04-19

You-Sheng Yang (community): Approve on 2021-04-19

git-ubuntu developers: Pending requested 2021-04-19

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19:

buildlog_ubuntu-hirsute-amd64.git-ubuntu_0-0_202104161321_ubuntu21.04.1_BUILDING.txt.gz Edit (164.7 KiB, text/plain)

Build recipt: https://code.launchpad.net/~vicamo/+recipe/usd-importer-daily
Failed build: https://code.launchpad.net/~vicamo/+archive/ubuntu/ppa/+build/21427854
Build log attached.

tags:	added: apport-collected hirsute
description:	updated

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: AlsaInfo.txt

AlsaInfo.txt Edit (53.4 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: CRDA.txt

CRDA.txt Edit (454 bytes, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: CurrentDmesg.txt

CurrentDmesg.txt Edit (155.8 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: IwConfig.txt

IwConfig.txt Edit (2.5 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: Lspci.txt

Lspci.txt Edit (63.9 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: Lspci-vt.txt

Lspci-vt.txt Edit (1.1 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: Lsusb.txt

Lsusb.txt Edit (439 bytes, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: Lsusb-t.txt

Lsusb-t.txt Edit (815 bytes, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: Lsusb-v.txt

#10

Lsusb-v.txt Edit (25.6 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: PaInfo.txt

#11

PaInfo.txt Edit (72.5 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: ProcCpuinfo.txt

#12

ProcCpuinfo.txt Edit (5.2 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: ProcCpuinfoMinimal.txt

#13

ProcCpuinfoMinimal.txt Edit (1.3 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: ProcEnviron.txt

#14

ProcEnviron.txt Edit (249 bytes, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: ProcInterrupts.txt

#15

ProcInterrupts.txt Edit (3.0 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: ProcModules.txt

#16

ProcModules.txt Edit (8.3 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: RebootRequiredPkgs.txt

#17

RebootRequiredPkgs.txt Edit (250 bytes, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: RfKill.txt

#18

RfKill.txt Edit (174 bytes, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: UdevDb.txt

#19

UdevDb.txt Edit (225.7 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: WifiSyslog.txt

#20

WifiSyslog.txt Edit (256.5 KiB, text/plain)

apport information

Revision history for this message

You-Sheng Yang (vicamo) wrote on 2021-04-19: acpidump.txt

#21

acpidump.txt Edit (249.9 KiB, text/plain)

apport information

Revision history for this message

Robie Basak (racb) wrote on 2021-04-19:

#22

Thank you for reporting this.

Although we hope to support deb builds in the future and welcome fixes towards this, for now the only supported route is the snap. Since the snap isn't affected by this, I'm setting Importance to Low. But I did spend some time looking into this and although the root cause seems quite complicated there is a trivial workaround available. I think just setting fp=io.BytesIO() is a quick workaround for the problem. Analysis below.

apt_repo_test.py is instantiating some urllib.error.HTTPError exception instances as parameters to the test_maybe_fetch_release_url_not_found and test_maybe_fetch_release_url_other_error tests. These have fp=None.

When these are instantiated, urllib.error.HTTPError is deliberately not calling superclass constructors:

        # The addinfourl classes depend on fp being a valid file
        # object. In some cases, the HTTPError may not have a valid
        # file object. If this happens, the simplest workaround is to
        # not initialize the base classes.
        if fp is not None:
            self.__super_init(fp, hdrs, url, code)

One of the superclasses is tempfile._TemporaryFileWrapper which sets self.file. This superclass also overrides __getattr__.

During pytest's collection phase, it attempts to look up the __name__ attribute on the test parameter, which is in this case the exception object. When it does so the tempfile._TemporaryFileWrapper __getattr__ override gets called. This method assumes that self.__dict__['file'] exists, as it was set in the constructor. But in our case the constructor was never called, so a KeyError is raised.

Since pytest only expects an AttributeError or success, the KeyError is passed through, and the test collection phase crashes.

This could be seen as:

1) A bug in how we're instantiating the exception. Is it permitted for a urllib _user_ to instantiate an exception? If not, how do we test our code's behaviour that depends on urllib? If so, how should we instantiate the exception such that the object will correctly fail with AttributeError when looking up an unknown attribute?

2) A bug in pytest; should it catch KeyError when looking up an attribute that might not exist? I don't think this would be a good idea.

3) A bug in Python. I shoudn't be able to instantiate an HTTPError that raises a KeyError when I look up an unknown attribute. Looking at the code it's possible that urllib.request will raise such an exception itself, and if so I think it would be a bug in Python. It would take further investigation to see if this is possible.

It doesn't look like there were any recent changes in Python that triggered this. pytest commit c22ce1a started looking up __name__ with hasattr in November 2019 that was released in pytest 5.3.0. Groovy to Hirsute goes from 4.6.11-1 to 6.0.2-2ubuntu1. So I think it's likely that this change in pytest triggered this regression for us.

Thank you for reporting this.

When these are instantiated, urllib.error.HTTPError is deliberately not calling superclass constructors:

# The addinfourl classes depend on fp being a valid file
        # object.  In some cases, the HTTPError may not have a valid
        # file object.  If this happens, the simplest workaround is to
        # not initialize the base classes.
        if fp is not None:
            self.__super_init(fp, hdrs, url, code)

One of the superclasses is tempfile._TemporaryFileWrapper which sets self.file. This superclass also overrides __getattr__.

Since pytest only expects an AttributeError or success, the KeyError is passed through, and the test collection phase crashes.

This could be seen as:

2) A bug in pytest; should it catch KeyError when looking up an attribute that might not exist? I don't think this would be a good idea.

Changed in usd-importer:
importance:	Undecided → Low
status:	New → In Progress
assignee:	nobody → Robie Basak (racb)

Robie Basak (racb) on 2021-04-23

Changed in usd-importer:
status:	In Progress → Fix Committed

Revision history for this message

Robie Basak (racb) wrote on 2023-08-31: Fix released in git-ubuntu

#23

Fix released in git-ubuntu version 1.1

Changed in git-ubuntu:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

git-ubuntu

Hirsute: FTBFS on master branch: /usr/lib/python3.9/tempfile.py:607: in __getattr__: KeyError: 'file'

Bug Description

Related branches

Other bug subscribers

Bug attachments

Remote bug watches

Hirsute: FTBFS on master branch: /usr/lib/python3.9/tempfile.py:607: in getattr: KeyError: 'file'