ubuntutools/archive.*SourcePackage::pull() fails to fallback correctly to Launchpad

It's working again now, so I'm not absolutely confident that this problem actually exists.

Although tracker.debian.org is reachable again.

afaik we have no dependencies on anything over the network except Launchpad.

We think it is likely that the library call to "pull-debian-source" that the importer uses is causing this, and we'd prefer for the importer to depend and download from Launchpad only. So this needs further investigation.

OK, I think I figured it out (lots of indirection here):

ubuntutools.archive.SourcePackage::__init__:

        if self.distribution:
            self.masters = [UDTConfig.defaults['%s_MIRROR'
                                               % self.distribution.upper()]]

ubuntutools.archive.SourcePackage::_source_urls:

        "Generator of sources for name"
        if self._dsc_source:
            yield os.path.join(os.path.dirname(self._dsc_source), name)
        for mirror in self.mirrors:
            yield self._mirror_url(mirror, name)
        for mirror in self.masters:
            if mirror not in self.mirrors:
                yield self._mirror_url(mirror, name)
        yield self._lp_url(name)

What would give us only LP URLs is that last line.

ubuntutools.config::UDTConfig.defaults:

    defaults = {
        'BUILDER': 'pbuilder',
        'DEBIAN_MIRROR': 'http://deb.debian.org/debian',
        'DEBSEC_MIRROR': 'http://security.debian.org',
        'LPINSTANCE': 'production',
        'MIRROR_FALLBACK': True,
        'UBUNTU_MIRROR': 'http://archive.ubuntu.com/ubuntu',
        'UBUNTU_PORTS_MIRROR': 'http://ports.ubuntu.com',
        'UPDATE_BUILDER': False,
        'WORKDIR': None,
        'KEYID': None,
    }

So, by default, Ubuntu goes to archive.ubuntu.com and Debian goes to deb.debian.org.

So I think we could, trivially, subclass DebianSourcePackage and UbuntuSourcePackage for our purposese to LaunchpadDebianSourcePackage and LaunchpadUbuntuSourcePackage where by those subclasses (of DebianSourcePackage and UbuntuSourcePackage respectively), call super(), but immediately after unset the masters and mirrors lists?

I'm not sure the consequence of doing that, but will ask cjwatson and wgrant.

Note there is a comment in archive.py that specifically states not all source packages are mirrored and the mirroring can lag.

The code in python3-ubuntutools seems to do the right thing when we pull a package:

            for url in self._source_urls(name):
                try:
                    if self._download_file(url, name):
                        break

_source_urls() will iterate over mirrors first, and then masters, and finally LP, as Nish mentioned. That seems robust. If one is down, move to the next one, until a good one is found.

What may not be working well enough is this "switch to the next one". Maybe detecting a broken mirror doesn't work. Maybe it takes too much time to move to the next one. Maybe it lacks a way to mark a mirror as broken, so that it is not tried again the next time.

Replacing this mechanism with having it all centered on LP may not be the best fix. For example, there is that one comment in the source that not all debian packages get imported into LP. Perhaps we should instead enhance how these source_urls are tried.

Do we have more information on exactly what was failing? Maybe we should try to reproduce this problem by blacklisting (firewall or proxy) the debian resources, or just some of them, and retry an import and see what happens when that blacklisted url is tried. Does the code move to the next one quickly enough?

Nish, Robie, I've rewritten pull-*-* and much of ubuntutools/archive.py, if this is still a problem for you please test with my build for bug 1453330 and let me know.

Is this still reproducable for anyone?

[Expired for ubuntu-dev-tools (Ubuntu) because there has been no activity for 60 days.]