missing main/debian-installer in repo causes debmirror to fail

How is this a bug? if the directory doesnt exist, debmirror cant mirror from it.

In my opinion, as main/debian-installer is not a section which is present all the time on all the dists, debmirror should have a condictional that check if exist.

The other option is to make madatory for the archive to have dists/*/main/debian-installer even when there is any packages for it.

I don't know if you don 't see the problem here, but the thing is, that with the actual debmirror you have to separate in diference directory each dists, in order to be able to mirror main/debian-installer:

Means that you have this:
ubuntu
  dists
    dapper
      main
        debian-installer
      restricted
      universe
      multiverse
ubuntu-security
  dists
    dapper-security
      main
      restricted
      ...
      ...
...

Instead of:
ubuntu
  dists
    dapper
      main
        debian-installer
      restricted
      universe
      multiverse
    dapper-security
     main
      restricted
      ...
      ...
...

BTW, I write a mail to the author and this was his response:

"You have to edit the script so it knows where to skip non-existing
sections. Debmirror was never written to handle different sections per
suite.

In line 631 add for example

   next if ($section =~ /debian-installer/ && $dist =~ /.*-security/);

MfG
       Goswin"

Carlos Javier

I'll try to describe this to be more easy.

I have a netboot setup in my LAN so that I can boot and install ubuntu from the a local server. To netboot and install, the main/debian-installer section needs to be available in my local mirror. I use preseeding to configure mirror information and packages so that souces.list points to my local server instead of archive.ubuntu.com (and installation requires almost no manual intervention). To that extent, my debmirror configuration automatically fetches the following:

DIST=feisty
SECTIONS=main,restricted,universe,multiverse,main/debian-installer

I run the debmirror script every night in a cronjob so that it stays updated with http://gb.archive.ubuntu.com. That means that updates that come to "feisty" repo will automatically hit my local mirror. This configuration allows me to install feisty with only the "feisty" repo available (updates, backports and security updates not available in local mirror and not available to clients).

But this means that the client has to use the internet and connect to archive.ubuntu.com to download other repos (fesity-updates, feisty-backports, feisty-security). I trust my LAN server and have no fear of it getting compromised, so I would like to host all the repos (including feisty-security) locally.

If I change the debmirror configuration to:

DIST=feisty,feisty-updates,feisty-backports,feisty-security
SECTIONS=main,restricted,universe,multiverse,main/debian-installer

It will try to download all updates but will fail when trying to get main/debian-installer from feisty-security (or backports, updates). Removing main/debian-installer will eliminate the problem, and all the repos will be downloaded. But I won't be able to netboot and install. (Note: debmirror automatically cleans the mirror, so anything that is not in the DIST and SECTIONS will be removed. This means that I cannot have 2 scripts; one to download main/debian-installer without all the other repos and another script to download the other repos without main/debian-installer)

FYI, in dapper, I am able to download all repos and the main/debian-installer section since dapper-security has a main/debian-installer section. But the section was added later, until which I had the same problem as I described above.

Summarizing the usecase problem is that I if feisty-security does not have a main/debian-installer section, local mirrors cannot serve feisty-security to their clients. If they do add feisty-security, then they will not be able to netboot and do automated installs.

Two possible solutions come to mind (and also suggested by Carlos Javier):
1. Modify debmirror to ignore repos that do not have main/debian-installer
2. Create main/debian-installer sections automatically for all repos

The second option sounds more simple and easy. (When feisty was released, there was feisty-security available, but no packages in it. This allowed the feisty ISO to enable the security repo without clients getting any 404 errors. Later, when packages were added to feisty-security, it trickled down to clients (through update-manager) without any additional sources.list modification.

Hope that helped.
--NetVenture

FYI,...

More information added.

Couldn't you just use 2 debmirror statements in your cronjob? One for DIST=feisty, and the other for DIST=feisty-updates,f-b,f-s ?

D'oh -- no you cannot just use 2 debmirror statements, unless you list two unique locat target locations. Using debmirror twice with the same target causes the first to be deleted by the second as unnecessary packages.

This bug's status is still set to incomplete. I'm not sure why, but maybe I can help out with an example of how to reproduce the bug yourself:
  cd /srv/mirror/ubuntu
  mkdir hardy
  debmirror --nosource --host de.archive.ubuntu.com --root ubuntu --method=http --dist hardy,hardy-updates,hardy-security --section main,restricted,universe,multiverse,main/debian-installer,restricted/debian-installer hardy/

That will give you an error:
Won't mirror without dists/hardy-security/main/debian-installer/binary-i386/Pack
ages.gz signature in Release at /usr/bin/debmirror line 1187
( this error message is from the debian sarge version, but the problem is the same)

Here is a workaround using 3 debmirror statements:
  mkdir hardy hardy-security hardy-updates
  debmirror --nosource --host de.archive.ubuntu.com --root ubuntu --method=http --dist hardy --section main,restricted,universe,multiverse,main/debian-installer,restricted/debian-installer hardy
  debmirror --nosource --host de.archive.ubuntu.com --root ubuntu --method=http --dist hardy-security --section main,restricted,universe,multiverse hardy-security
  debmirror --nosource --host de.archive.ubuntu.com --root ubuntu --method=http --dist hardy-updates --section main,restricted,universe,multiverse hardy-updates

Is there a fix/workaround for this?

Ingo: Using 3 debmirror statements is ok as long as you don't cleanup the mirror. But if you use a --cleanup (or post cleanup) then the last debmirror statement will deem the files downloaded by the previous statements as unneeded and hence, delete them.

Brak:

I've switched to using apt-mirror. There are some inconveniences
1: using a different mirror server for the same repo (ex: hardy) will result in a complete download of the repo
2: a slightly complex directory structure

If you want to use apt-mirror, here is what you can do:
$ sudo apt-get install apt-mirror
$ delete all the lines in /etc/apt/mirror.list and replace with the following:
==========Cut Here=========
############# config ##################
#
set base_path /home/Repository-hardy
#
# if you change the base path you must create the directories below with write privlages
#
set mirror_path $base_path/mirror
set skel_path $base_path/skel
set var_path $base_path/var
set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
set nthreads 20
set tilde 0
#
############# end config ##############

deb http://archive.ubuntu.com/ubuntu hardy main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu hardy-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu hardy-backports main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu hardy-security main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu hardy main/debian-installer restricted/debian-installer universe/debian-installer multiverse/debian-installer
#deb http://archive.ubuntu.com/ubuntu hardy-updates main/debian-installer restricted/debian-installer universe/debian-installer
#deb http://archive.ubuntu.com/ubuntu hardy-backports main/debian-installer
#deb http://archive.ubuntu.com/ubuntu hardy-security main/debian-installer universe/debian-installer
==========Cut Here=========

As you can see, the debian-installer for hardy-updates, backports and security have been commented out. This is because they are not available on the server yet (at the time of this writing). You need to check manually on the server occasionally and then enable them in the mirror.list file.

This debmirror issue has not been resolved even after nearly 2 years. I hope that someone gets to fix it and make it a better alternative to apt-mirror.

--NV

Hi,

I have the same issue. I was trying to download a lenny repository and i found the main/debian-installer source problem.

Well, i've done a little workaround in debmirror. I just check if the section contains "debian-installer", if so, i don't list the directories and source files to download.

It ___seems___ to work but is not fully tested. Use carefully.

Hello, i've just finished the download of lenny repository with debmirror + patch. Works for me pretty well :-)

This patch should be a valid debdiff to apply to the source package in intrepid (as of today).
It incorperates Carlos's patch.

Subscribed ubuntu-universe-sponsors.
Hopefully the debdiff is valid and this bug can get closed soon.

Hi Carlos,

The first hunk of your patch has

  + if ($section=~"debian-installer") {

doesn't that mean that it will only mirror "Sources" etc. if it is the debian-installer
section, which is the opposite of what we want?

Apologies if I have missed something, I am not familiar with the package.

Thanks,

James

I've made some testing with stock debmirror in jaunty, it seems to me it works correctly. Keep in mind main/debian-installer is available everywhere but $developmentrelease-{proposed,updates,security,backports} just because these sections haven't been initialized yet.

James is correct; this patch did introduce a regression. Fortunately, it only affects the estimated download size calculation. Here is a fix.

The fixed package is also available in my PPA: <https://launchpad.net/~anders-kaseorg/+archive/ppa>

Thanks for the patch! This should be uploaded shortly.

This bug was fixed in the package debmirror - 20070123ubuntu3

---------------
debmirror (20070123ubuntu3) jaunty; urgency=low

  * Fix a regression in the estimated download size calculation introduced
    by skip-installer.patch. (LP: #34376)

 -- Anders Kaseorg <email address hidden> Sun, 15 Feb 2009 16:42:27 -0500

There is a real regression between the debmirror for intrepid release (20070123ubuntu1) and jaunty release (20070123ubuntu3).
I'm using a script to mirror hardy, intrepid and now jaunty for i386 and amd64, is working quite well on intrepid doing is mirroring job ... but since i use since yesterday another machine with jaunty the script stop on debmirrors errors ..
Errors:
 dists/jaunty-backports/main/debian-installer/binary-amd64/Packages.bz2 missing
Etc ...

For testing i tried chanhing protocol http to rsync and using 3 different sources but no way ..

Tow different behavior for the same script there is really a modification between the tow versions ... i guess when the final jaunty repo is up it'll work .. but for the next release (KK) if i want to prepare my repo it won't be possible with debmirror if i want the nework install part for all my releases.
Here my script ...

Juste to be sure :
- i reboot my intrepid machine and copy the exact script (removing --run-dry flag) and launch it => update ok see attachment => debmirror-intrepid.log
- on my jaunty machine running the script (removing --run-dry flag) => update non ok see attachment => debmirror-jaunty.log

For info my resources for repositories are on another server (centos) via NFS.

I can confirm that this patch is no longer effective for some reason:

$ debmirror --method=rsync --host=ubuntu.retrosnub.co.uk --root=:ubuntu --pdiff=mirror '--ignore=^(releases/|cdimage$|\.htaccess$|\.cdimage-lock$)' '--rsync-options=-aIL --partial --no-motd' --getcontents --dist=hardy,hardy-updates,hardy-security,hardy-backports,intrepid,intrepid-updates,intrepid-security,intrepid-backports,jaunty,jaunty-updates,jaunty-security,jaunty-backports,karmic,karmic-updates,karmic-security,karmic-backports --section=main,restricted,universe,multiverse,main/debian-installer --arch=i386,amd64 -v -p /srv/mirror/ubuntu/
(...)
Errors:
 dists/karmic-backports/main/debian-installer/binary-amd64/Packages.bz2 missing
 dists/karmic-backports/main/debian-installer/binary-i386/Packages.bz2 missing
 dists/karmic-security/main/debian-installer/binary-amd64/Packages.bz2 missing
 dists/karmic-security/main/debian-installer/binary-i386/Packages.bz2 missing
Failed to download some Package, Sources or Release files!
WARNING: releasing 1 pending lock...

Hi,

As there is nothing to sponsor at the moment, I'll unsubscribe Ubuntu Sponsors for Universe. Please feel free to subscribe it again when something needs sponsoring.

Thanks,
Fabrice

I've just added a watch on Debian bug #542826, which I think is thesame as this.
It was fixed in August, so I think the latest Ubuntu will have this fix? Can someone running Ubuntu test it?
kk

KarlGoetz if you want us to test something describe it step by step and i will do it ;)

For me the bug is still present, i still can't debmirror karmic-backports and karmic-security as the main/debian-installer isn't initialized ....

I don't get it really ... why the structures of repositories aren't initialized correctly as soon the alpha1 is launched ?
I mean "correctly" with all necessary (structures, gz etc...) for mirroring even if there is no deb in it.

Hope a change in future release, it's really a pain to maintain a partial mirror ...

I can also confirm the bug is still present. I even downloaded the latest release http://ftp.de.debian.org/debian/pool/main/d/debmirror/debmirror_2.3.1.tar.gz and it did not help.

I, like Malcolm, also have this issue building my debmirror for karmic. Debmirror box is running jaunty, debmirror version 20070123ubuntu3.

Still getting this error on karmic server 64-bit. This seems a killer bug as it prevents netinstalling from a local mirror (except using apt-mirror).

Seems like debmirror has an undocumented --skip-installer option vis:

      # skip debian-installer for each dist in --skip-installer=foo,...
      if ($section =~ /debian-installer/) {
        my $skipped = 0;
        foreach my $skipped_release (@skip_installer) {
          if ($dist eq $skipped_release) {
            $skipped = 1;
            last;
          }
        }
        next if $skipped;
      }

I currently configure manually, however next time I have to change the debmirror script, I'm going to rsync probe the debian-installers existance to dynamically build the $skip_installer variable which contains the --skip-installer argument for by current debmirror script:

$debug debmirror --nosource --passive --verbose \
        $md5sum \
        $pdiff \
        $max \
        $dry \
        $nocleanup \
        $small \
        $rbatch \
        $skip_installer \
        $ignore_installer_arch \
        --ignore /dist-upgrader-all/ \
        --section=main,restricted,universe,multiverse,main/debian-installer \
        --method=$method \
        --host=$rhost \
        --dist=$dist \
        --arch=$arch \
        --root=$root \
        $spoolA/ubuntu
)

So maybe the real bug now is a documentation bug for debmirror(1) as at least debmirror --help shows --skip-installer as an option.

The issue this bug was originally about debmirror failing when using debian-installer was resolved in debmirror version 1:1.0.1 which has been available since Lucid. Furthermore, I tested this using the following command with debmirror from Natty:

(natty-i386)root@flash:/tmp/debmirror-2.4.6ubuntu1# debmirror --host=ubuntu.osuosl.org --method=http --dist=karmic --section=multiverse,main/debian-installer --ignore-release-gpg ../crap/
gpgv: keyblock resource `/root/.gnupg/trustedkeys.gpg': file open error
gpgv: Signature made Wed Oct 28 07:23:20 2009 PDT using DSA key ID 437D05B5
[GNUPG:] ERRSIG 40976EAF437D05B5 17 2 00 1256739800 9
[GNUPG:] NO_PUBKEY 40976EAF437D05B5
gpgv: Can't check signature: public key not found

As you'll note I received no errors during the mirroring process. Subsequently, I am setting the bug report to Fix Released.

The problem with the test Brian is that karmic has a main/debian-installer section and the problem occurs when main/debian-installer doesn't exist.

The problem usually manifests itself in new dists, in the -updates, -proposed, and -backports trees, and maybe even security.

If you want to test it you need to test it against a dist that doesn't have a debian-installer section at the moment like natty-updates.

Scott thanks for the additional info I tried mirroring natty and didn't run into any problems there either:

debmirror --host=ubuntu.osuosl.org --method=http --dist=natty,natty-updates --section=multiverse,main/debian-installer --ignore-release-gpg mirror/
gpgv: keyblock resource `/root/.gnupg/trustedkeys.gpg': file open error
gpgv: Signature made Mon Feb 7 06:16:46 2011 PST using DSA key ID 437D05B5
[GNUPG:] ERRSIG 40976EAF437D05B5 17 2 00 1297088206 9
[GNUPG:] NO_PUBKEY 40976EAF437D05B5
gpgv: Can't check signature: public key not found

Is that sufficient?

You have to look at the created hierarchy and ensure there isn't a main/debian-installer directory. If there is, then such a section was added to natty-updates and the test was invalid. You can use the multiverse directory structure as a guide.

This bug is marked 'fix released' (I'm not sure thats really the case yet...), but I thought I'd include some more links. These are all debian bugs *related* to this issue, directly or indirectly. If there is some agreement, I'll add them as watchers to this bug.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=267721
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=232093
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=619146
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=542826 (already linked)
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=576576
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=619361

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=581013

Finally, A link to a git branch with a couple of (perhaps dodgy) patches I made to fix 581013: https://gitorious.org/branches-of-others/debmirror

Joeyh didn't seem all that interested in them (He didn't agree they were solving a real problem), but they've made my mirror scripts much nicer :) I don't mirror the cd images, so the code around that might (will probably) result in the files being cleaned up if the md5sums fail - not sure what to do about that, perhaps someone has an idea.