"bizarre error - file size is not what the server reported" with private ppa

A second attempt to fetch it succeeded, so perhaps this was an intermittent server or client side error. Perhaps the http stream was aborted on the server side, in which case there might be a server side log?

I'm not sure how to rate this: it's a user visible error so it seems to count as critical but maybe it's also a one-off.

Some possibilities:
 - we don't write the file atomically
   unlikely, you can't reference a deb till its in the index, and it
gets written entirely beforehand
 - ssl disconnect partway through - if using http/1.0 that counts as
'EOF' unless content-length was emitted
 - but in apt where content-length + early disconnect gives this error.

I suspect an apt client side issue FWIW - possibly interacting with ssl on LP in some way, so we should keep an LP task.

On Friday 08 July 2011 02:21:56 you wrote:
> Some possibilities:
> - we don't write the file atomically
> unlikely, you can't reference a deb till its in the index, and it
> gets written entirely beforehand

In previous instances of this problem it's always been a transparent proxy
getting in the way.

On Friday 08 July 2011 02:29:54 you wrote:
> I suspect an apt client side issue FWIW - possibly interacting with ssl
> on LP in some way, so we should keep an LP task.

It's all served statically from Apache, how would LP be involved?

@bigjools
re: proxy - this is https://, so no interception can be occuring at the http layer.
re: LP task - LP is involved at the project level, and until we track down the cause, or permit bugs on launchpad-project directly, this is as good a place as any other to start gathering data about the problem.

When the PPA publisher runs it does not do anything atomically in dists/. This might be the cause of the bug as there could be a race we've not thought about.

Well, each file is written then moved into place near-atomically. But, critically, pool/ is written strictly before dists/, and there can be no file conflicts, so any file referenced in Packages or Sources must already be on disk.

I can reproduce this message by adding many https repositories to sources.list. For example, with these sources (which are public, so you can try this yourself):

deb https://debathena.mit.edu/apt precise debathena
deb-src https://debathena.mit.edu/apt precise debathena
deb https://debathena.mit.edu/apt quantal debathena
deb-src https://debathena.mit.edu/apt quantal debathena
deb https://debathena.mit.edu/apt raring debathena
deb-src https://debathena.mit.edu/apt raring debathena
deb https://debathena.mit.edu/apt saucy debathena
deb-src https://debathena.mit.edu/apt saucy debathena
deb https://debathena.mit.edu/apt trusty debathena
deb-src https://debathena.mit.edu/apt trusty debathena
deb https://debathena.mit.edu/apt utopic debathena
deb-src https://debathena.mit.edu/apt utopic debathena
deb https://debathena.mit.edu/apt vivid debathena
deb-src https://debathena.mit.edu/apt vivid debathena

I get:

$ sudo rm /var/lib/apt/lists/debathena*
$ sudo apt-get update
…
W: Size of file /var/lib/apt/lists/partial/debathena.mit.edu_apt_dists_quantal_debathena_source_Sources.bz2 is not what the server reported 31194 16209
W: Size of file /var/lib/apt/lists/partial/debathena.mit.edu_apt_dists_saucy_debathena_source_Sources.bz2 is not what the server reported 31029 24393
W: Size of file /var/lib/apt/lists/partial/debathena.mit.edu_apt_dists_trusty_debathena_source_Sources.bz2 is not what the server reported 26334 16209
W: Size of file /var/lib/apt/lists/partial/debathena.mit.edu_apt_dists_utopic_debathena_source_Sources.bz2 is not what the server reported 26237 16209
W: Size of file /var/lib/apt/lists/partial/debathena.mit.edu_apt_dists_vivid_debathena_source_Sources.bz2 is not what the server reported 14 251

This does not happen with the same repositories over plain http.

I submitted a patch to https://bugs.debian.org/781509.

This bug was fixed in the package apt - 1.0.9.9ubuntu1

---------------
apt (1.0.9.9ubuntu1) wily; urgency=low

  * merged from debian/sid

apt (1.0.9.9) unstable; urgency=medium

  [ David Kalnischkies ]
  * parse specific-arch dependencies correctly on single-arch systems
    (Closes: 777760)
  * remove "first package seen is native package" assumption.
    Thanks to Axel Beckert for testing (Closes: 782777)

apt (1.0.9.8) unstable; urgency=medium

  [ David Kalnischkies ]
  * fix another d(e)select-upgrade typo (LP: #1399037)
  * properly handle expected filesize in https.
    Thanks to Robert Edmonds and Anders Kaseorg for initial patchs
    (Closes: 777565, 781509) (LP: #807303)
  * avoid depends on std::string implementation for pkgAcquire::Item::Mode
    (Closes: 781858)
  * demote VectorizeString gcc attribute from const to pure
  * keyids in "apt-key del" should be case-insensitive (Closes: 781696)
  * parse specific-arch dependencies correctly on single-arch systems
    (Closes: 777760)

  [ Michael Vogt ]
  * fix crash in order writing in pkgDPkgPM::WriteApportReport() (LP: #1436626)

 -- Michael Vogt <email address hidden> Thu, 07 May 2015 10:40:54 +0200