Merge ubuntu-debuginfod:fix-requests-get into ubuntu-debuginfod:master

[Sergio Durigan Junior]

This MP fixes the code that downloads things from LP.

I've found that requests will happily deflate gzip files because it always sends "Accept-Encoding: gzip deflate". Because some source packages have .diff.gz files, this automatically decompression ends up messing with "dpkg-source -x". I've been experiencing these errors locally.

Upstream says that one should use requests.raw when reading data from the object in this case:

https://github.com/psf/requests/issues/1586
https://requests.readthedocs.io/en/latest/user/quickstart/?highlight=deflate#raw-response-content

so this is what I'm proposing here.

[Bryce Harrington (bryce)]

The rtd article says to use stream=True in the get() call, which you're already doing. The other link explains convincingly that r.raw is the way to go.

I wonder if there might be plain text files you're downloading that do need the deflation, but presumably you've verified that not to be the case.

You might leave yourself these two links in your commit message (or as comments) for future reference.

I also checked the api docs for copyfileobj to see if there's any error conditions / side effects / exceptions / parameters / return codes to note but it has nothing special.

So, LGTM, +1

review: Approve

[Sergio Durigan Junior (sergiodj)]

On Thursday, December 15 2022, Bryce Harrington wrote:

> Review: Approve
>
> The rtd article says to use stream=True in the get() call, which you're already doing. The other link explains convincingly that r.raw is the way to go.
>
> I wonder if there might be plain text files you're downloading that do need the deflation, but presumably you've verified that not to be the case.
>
> You might leave yourself these two links in your commit message (or as comments) for future reference.
>
> I also checked the api docs for copyfileobj to see if there's any error conditions / side effects / exceptions / parameters / return codes to note but it has nothing special.
>
> So, LGTM, +1

Thanks, Bryce. I'll go ahead and merge the fix.

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

diff --git a/debuggetter.py b/debuggetter.py
index bafa914..e6b1de0 100644
--- a/debuggetter.py
+++ b/debuggetter.py
@@ -113,9 +113,7 @@ class DebugGetter:
                 with s.get(url, allow_redirects=True, timeout=10, stream=True) as r:
                     r.raise_for_status()
                     with tempfile.NamedTemporaryFile(mode="wb") as tmpfile:
-                        # 10 MB for chunk_size should be enough...
-                        for chunk in r.iter_content(chunk_size=10 * 1024 * 1024):
-                            tmpfile.write(chunk)
+                        shutil.copyfileobj(r.raw, tmpfile)
                         tmpfile.flush()
                         self._try_to_move_atomically(tmpfile.name, filepath)
         except (