On Thu, Mar 25, 2010 at 5:48 PM, Julian Edwards <
<email address hidden>> wrote:

> >
> > === modified file 'lib/lp/archivepublisher/tests/test_publisher.py'
> > --- lib/lp/archivepublisher/tests/test_publisher.py   2010-02-10 00:25:55
> +0000
> > +++ lib/lp/archivepublisher/tests/test_publisher.py   2010-03-25 20:23:14
> +0000
> > @@ -26,7 +26,7 @@
> >  from canonical.database.constants import UTC_NOW
> >  from canonical.launchpad.ftests.keys_for_tests import gpgkeysdir
> >  from lp.soyuz.interfaces.archive import (
> > -    ArchivePurpose, IArchiveSet)
> > +    ArchivePurpose, ArchiveStatus, IArchiveSet)
> >  from lp.soyuz.interfaces.binarypackagerelease import (
> >      BinaryPackageFormat)
> >  from lp.registry.interfaces.distribution import IDistributionSet
> > @@ -95,6 +95,31 @@
> >          foo_path = "%s/main/f/foo/foo_666.dsc" % self.pool_dir
> >          self.assertEqual(open(foo_path).read().strip(), 'Hello world')
> >
> > +    def testDeletingPPA(self):
> > +        """Test deleting a PPA"""
> > +        ubuntu_team = getUtility(IPersonSet).getByName('ubuntu-team')
> > +        test_archive = getUtility(IArchiveSet).new(
> > +            distribution=self.ubuntutest, owner=ubuntu_team,
> > +            purpose=ArchivePurpose.PPA)
> > +
> > +        publisher = getPublisher(test_archive, None, self.logger)
> > +
> > +        pub_source = self.getPubSource(
> > +            sourcename="foo", filename="foo_1.dsc",
> > +            filecontent='I am a file.',
> > +            status=PackagePublishingStatus.PENDING,
> archive=test_archive)
> > +
> > +        publisher.A_publish(False)
> > +        self.layer.txn.commit()
> > +        publisher.C_writeIndexes(False)
> > +        publisher.D_writeReleaseFiles(False)
> > +        pub_source.sync()
>
> You don't need to do these last 5 lines, nor the getPubSource.
>
> Doing more than necessary makes for a fragile test if other parts of the
> system change.  A better test would be to throw some random files into the
> root directory of the archive and then assert that they get deleted when
> you call deleteArchive, along with the status change.
>

You bring up a good point. However, I think what we need to do to
successfully test our ability to delete an archive and not just our ability
to delete a directory with some files in it is to attempt deleting a simple
but typical archive. If there is higher level way of achieving the same
result instead of calling those publisher methods manually then I think that
would be a good compromise. What do you think?


> > +
> > +        self.assertTrue(os.path.exists(publisher._config.archiveroot))
> > +        publisher.deleteArchive()
> > +        self.assertTrue(not
> os.path.exists(publisher._config.archiveroot))
>
> How about assertFalse?
>

ACK.

<snip>

> === modified file 'lib/lp/soyuz/scripts/publishdistro.py'
> > --- lib/lp/soyuz/scripts/publishdistro.py     2010-02-24 11:53:22 +0000
> > +++ lib/lp/soyuz/scripts/publishdistro.py     2010-03-25 20:23:14 +0000
> > @@ -16,7 +16,7 @@
> >  from canonical.database.sqlbase import (
> >      clear_current_connection_cache, flush_database_updates)
> >  from lp.soyuz.interfaces.archive import (
> > -    ArchivePurpose, IArchiveSet, MAIN_ARCHIVE_PURPOSES)
> > +    ArchivePurpose, ArchiveStatus, IArchiveSet, MAIN_ARCHIVE_PURPOSES)
> >  from lp.registry.interfaces.distribution import IDistributionSet
> >  from canonical.launchpad.scripts import logger, logger_options
> >  from lp.services.scripts.base import LaunchpadScriptFailure
> > @@ -191,24 +191,34 @@
> >          else:
> >              log.info("Processing %s" % archive.archive_url)
> >              publisher = getPublisher(archive, allowed_suites, log)
> > -
> > -        try_and_commit("publishing", publisher.A_publish,
> > -                       options.careful or options.careful_publishing)
> > -        # Flag dirty pockets for any outstanding deletions.
> > -        publisher.A2_markPocketsWithDeletionsDirty()
> > -        try_and_commit("dominating", publisher.B_dominate,
> > -                       options.careful or options.careful_domination)
> > -
> > -        # The primary and copy archives use apt-ftparchive to generate
> the
> > -        # indexes, everything else uses the newer internal LP code.
> > -        if archive.purpose in (ArchivePurpose.PRIMARY,
> ArchivePurpose.COPY):
> > -            try_and_commit("doing apt-ftparchive",
> publisher.C_doFTPArchive,
> > -                           options.careful or options.careful_apt)
> > +
> > +        # Do we need to delete the archive or publish it?
> > +        if archive.status == ArchiveStatus.DELETING:
> > +            if archive.purpose == ArchivePurpose.PPA:
> > +                # Delete the archive.
> > +                try_and_commit("publishing", publisher.deleteArchive,
> > +                               options.careful or
> options.careful_publishing)
> > +            else:
> > +                # Other types of archives do not currently support
> deletion.
> > +                pass
> >          else:
> > -            try_and_commit("building indexes", publisher.C_writeIndexes,
> > +            try_and_commit("publishing", publisher.A_publish,
> > +                           options.careful or
> options.careful_publishing)
> > +            # Flag dirty pockets for any outstanding deletions.
> > +            publisher.A2_markPocketsWithDeletionsDirty()
> > +            try_and_commit("dominating", publisher.B_dominate,
> > +                           options.careful or
> options.careful_domination)
> > +
> > +            # The primary and copy archives use apt-ftparchive to
> generate
> the
> > +            # indexes, everything else uses the newer internal LP code.
> > +            if archive.purpose in (ArchivePurpose.PRIMARY,
> ArchivePurpose.COPY):
> > +                try_and_commit("doing apt-ftparchive",
> publisher.C_doFTPArchive,
> > +                               options.careful or options.careful_apt)
> > +            else:
> > +                try_and_commit("building indexes",
> publisher.C_writeIndexes,
> > +                               options.careful or options.careful_apt)
> > +
> > +            try_and_commit("doing release files",
> publisher.D_writeReleaseFiles,
> >                             options.careful or options.careful_apt)
> >
> > -        try_and_commit("doing release files",
> publisher.D_writeReleaseFiles,
> > -                       options.careful or options.careful_apt)
> > -
> >      log.debug("Ciao")
>
> This code has got a subtle bug - it will only process archives marked with
> publishing enabled because of the filter just above the context of the
> diff.
>

This was actually an intentional decision. Deleting an archive is a function
of the publisher. If the publisher is disabled for an archive, I don't think
we should process requests to delete the archive just like we wouldn't
process pending publications or requests to copy a package into the archive.

You're welcome to disagree on me this. I suspect that you might reply with
that the flag's actual intention is to 'disable publishing for the archive',
not 'disable the publisher for that archive'. However, I think even if this
is the case then we should consider re-purposing it to include this slightly
broader scope of functionality. If you're interested in my rationale or some
potential use cases for this, I'd be happy to discuss it further with you on
IRC.

However, looking at your UI branch, it seems that you disable the PPA as
well as set the status to ArchiveStatus.DELETING which of course puts a
'wrench' in this plan. See below for further thoughts related to this.

I also think in retrospect it's slightly the wrong approach to take here.
> What do you think to adding IArchiveSet.getByStatus()?  Then at the end of
> the publisher script we can grab a list of archives that are in the
> DELETING state and simply call publisher.deleteArchive().  The new method
> on IArchiveSet would require a simple new test.
>

Could you elaborate on what exactly you're referring to? I think there is a
lot that could be improved in this script (its a bit of a hack!).

However, you've made me notice a glaring bug. Once an archive's status is
set to ArchiveStatus.DELETED it'll start getting published again. However,
because PPAs are filtered by those with pending publications it means that
deleted PPAs won't get published unless it somehow gets a pending
publication (which we don't necessarily do anything to actively prevent and
I wouldn't trust that such a scenario wouldn't somehow occur).

I think it may make sense to take the time and refractor how we get the
archives we want the publisher to consider (this might be a good opportunity
to move some of the soyuz methods out of the registry Distribution model,
normalize them, and move them into IArchiveSet). The end result that I think
this could provide us with is easier reuse and testing of the logic used to
determine which archives needed to be considered by the publisher. For
example, how do you test that the publisher won't publish archives that have
publishing disabled or that are deleted?

Lets look at all the different ways we get the archive(s) we want to publish
depending on the mode of the script:

1. [distribution.getArchiveByComponent('partner')]
2. distribution.getAllPPAs()
3. distribution.getPendingPublicationPPAs()
4. getUtility(IArchiveSet).getByDistroPurpose(
            distribution, ArchivePurpose.DEBUG)
5. getUtility(IArchiveSet).getArchivesForDistribution(
            distribution, purposes=[ArchivePurpose.COPY])
6. [distribution.main_archive]

Do we really want to add to this list? Besides, do we really want to slap on
yet another method to IArchiveSet - maybe yes, maybe no but I don't think so
for the former.

Also, should this script be responsible for determining what the publisher
should do with an archive? In my opinion, I think it might make sense for
that responsibility to belong to the publisher and for this script to be
responsible for configuring the publisher based on the settings passed to
the script and passing the set of archives that we want it to consider.

Otherwise, we're nearly there!  I have a UI branch that's very close to
> done.
>

Woot! :) We are indeed.

Cheers,

Cody