Merge ~racb/ubuntu-dev-tools:diff-gz-identity into ubuntu-dev-tools:main

[Robie Basak]

[Robie Basak (racb)]

I force pushed to add some comments.

[Sebastien Bacher (seb128)]

Thanks, the explanations on the bug are convincing to me and the code changes match what was described (+bonus points for documenting the reason)

review: Approve

diff --git a/debian/changelog b/debian/changelog
index 64c7284..7b1618f 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,11 +1,16 @@
-ubuntu-dev-tools (0.193ubuntu5) UNRELEASED; urgency=medium
+ubuntu-dev-tools (0.193ubuntu5) mantic; urgency=medium
 
+  [ Steve Langasek ]
   * Remove references to deprecated
     http://people.canonical.com/~ubuntu-archive.
   * Remove references to architectures not supported in any active
     Ubuntu release.
 
- -- Steve Langasek <[e-mail]>  Tue, 30 May 2023 19:35:57 -0700
+  [ Robie Basak ]
+  * ubuntutools/misc: swap iter_content for raw stream with "Accept-Encoding:
+    identity" to fix .diff.gz downloads (LP: #2025748).
+
+ -- Robie Basak <[e-mail]>  Wed, 05 Jul 2023 15:32:43 +0100
 
 ubuntu-dev-tools (0.193ubuntu4) mantic; urgency=medium
 
diff --git a/ubuntutools/misc.py b/ubuntutools/misc.py
index 3f8f70e..0b31e42 100644
--- a/ubuntutools/misc.py
+++ b/ubuntutools/misc.py
@@ -348,7 +348,11 @@ def download(src, dst, size=0, *, blocksize=DOWNLOAD_BLOCKSIZE_DEFAULT):
     with tempfile.TemporaryDirectory() as tmpdir:
         tmpdst = Path(tmpdir) / "dst"
         try:
-            with requests.get(src, stream=True, timeout=60, auth=auth) as fsrc:
+            # We must use "Accept-Encoding: identity" so that Launchpad doesn't
+            # compress changes files. See LP: #2025748.
+            with requests.get(
+                src, stream=True, timeout=60, auth=auth, headers={"accept-encoding": "identity"}
+            ) as fsrc:
                 with tmpdst.open("wb") as fdst:
                     fsrc.raise_for_status()
                     _download(fsrc, fdst, size, blocksize=blocksize)
@@ -433,7 +437,16 @@ def _download(fsrc, fdst, size, *, blocksize):
 
     downloaded = 0
     try:
-        for block in fsrc.iter_content(blocksize):
+        while True:
+            # We use fsrc.raw so that compressed files stay compressed as we
+            # write them to disk. For example, if this is a .diff.gz, then it
+            # needs to remain compressed and unmodified to remain valid as part
+            # of a source package later, even though Launchpad sends
+            # "Content-Encoding: gzip" and the requests library therefore would
+            # want to decompress it. See LP: #2025748.
+            block = fsrc.raw.read(blocksize)
+            if not block:
+                break
             fdst.write(block)
             downloaded += len(block)
             progress_bar.update(downloaded, size)