Merge lp:~jtv/launchpad/bug-527170 into lp:launchpad

Proposed by Jeroen T. Vermeulen
Status: Superseded
Proposed branch: lp:~jtv/launchpad/bug-527170
Merge into: lp:launchpad
Diff against target: 328 lines (+196/-12)
2 files modified
utilities/soyuz-sampledata-cleanup.py (+175/-12)
utilities/start-dev-soyuz.sh (+21/-0)
To merge this branch: bzr merge lp:~jtv/launchpad/bug-527170
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) code Needs Fixing
Review via email: mp+20433@code.launchpad.net

This proposal has been superseded by a proposal from 2010-03-02.

Commit message

Automate more manual Soyuz dev startup.

To post a comment you must log in.
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

= Bug 527170 =

This branch automates more of the manual work needed to get a working
local Soyuz setup. For the current process, see:

    https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally

I already implemented a script that does part of the work needed on the
dev sample data, based on wgrant's original, but some features are
added:

 * --amd64 option automates setup of amd64 support.

 * Creates a user ppa-user, with a fixed name.

 * Signs the Ubuntu Code of Conduct for ppa-user (with a fake key).

 * Gives you a fixed URL for adding your own GPG key.

 * --email option lets you specify your email address of choice.

 * Adds ppa-user to ubuntu-team.

The dev server won't actually be able to send email to your chosen email
address (although it will try, see the wiki page) but the choice makes
it easier for you to attach your GPG key. This is needed when you sign
uploads.

I tried automating the registration of a GPG key, or use of a dedicated
one, but no solution was really satisfactory. Well, working with gpgme
to get the real key based on your email address would have been, but it
seems that our scripting environment forces gpg to live in a temporary
directory rather than in ~/.gnupg.

This is a utilities script acting on the dev playground database, so
there's no real testing. But at least the script is exercised by the
test suite so we can detect obvious breakage:
{{{
./bin/test -vv -t sampledata-cleanup
}}}

No lint.

There's also a new script utilities/start-dev-soyuz.sh, also based on
wgrant's original. I regularlized it a bit and made sure the necessary
directories in /var/tmp are created. I have no idea how to test this
second script without (1) upsetting the test environment, or (2) porting
it to python and adding a lot of weight. For this script, used in
manual testing only and easy enough to edit, I don't think that's worth
the trouble.

Jeroen

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

There's still a problem with this that escaped my notice because zeca was retaining my GPG key from previous tests.

review: Needs Fixing (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'utilities/soyuz-sampledata-cleanup.py'
--- utilities/soyuz-sampledata-cleanup.py 2010-02-24 12:13:29 +0000
+++ utilities/soyuz-sampledata-cleanup.py 2010-03-02 08:52:26 +0000
@@ -13,6 +13,9 @@
1313
14DO NOT RUN ON PRODUCTION SYSTEMS. This script deletes lots of14DO NOT RUN ON PRODUCTION SYSTEMS. This script deletes lots of
15Ubuntu-related data.15Ubuntu-related data.
16
17This script creates a user "ppa-user" (email ppa-user@example.com,
18password test) who is able to create PPAs.
16"""19"""
1720
18__metaclass__ = type21__metaclass__ = type
@@ -20,9 +23,12 @@
20import _pythonpath23import _pythonpath
2124
22from optparse import OptionParser25from optparse import OptionParser
23from os import getenv
24import re26import re
27import os
28import subprocess
25import sys29import sys
30from textwrap import dedent
31import transaction
2632
27from zope.component import getUtility33from zope.component import getUtility
28from zope.event import notify34from zope.event import notify
@@ -35,20 +41,33 @@
3541
36from canonical.lp import initZopeless42from canonical.lp import initZopeless
3743
44from canonical.launchpad.ftests.keys_for_tests import gpgkeysdir
45from canonical.launchpad.interfaces.account import AccountStatus
46from canonical.launchpad.interfaces.emailaddress import EmailAddressStatus
47from canonical.launchpad.interfaces.gpghandler import IGPGHandler
38from canonical.launchpad.interfaces.launchpad import (48from canonical.launchpad.interfaces.launchpad import (
39 ILaunchpadCelebrities)49 ILaunchpadCelebrities)
40from canonical.launchpad.scripts import execute_zcml_for_scripts50from canonical.launchpad.scripts import execute_zcml_for_scripts
41from canonical.launchpad.scripts.logger import logger, logger_options51from canonical.launchpad.scripts.logger import logger, logger_options
42from canonical.launchpad.webapp.interfaces import (52from canonical.launchpad.webapp.interfaces import (
43 IStoreSelector, MAIN_STORE, SLAVE_FLAVOR)53 IStoreSelector, MAIN_STORE, MASTER_FLAVOR, SLAVE_FLAVOR)
4454
55from lp.registry.interfaces.codeofconduct import ISignedCodeOfConductSet
56from lp.registry.interfaces.gpg import GPGKeyAlgorithm, IGPGKeySet
45from lp.registry.interfaces.series import SeriesStatus57from lp.registry.interfaces.series import SeriesStatus
58from lp.registry.model.codeofconduct import SignedCodeOfConduct
46from lp.soyuz.interfaces.component import IComponentSet59from lp.soyuz.interfaces.component import IComponentSet
60from lp.soyuz.interfaces.processor import IProcessorFamilySet
47from lp.soyuz.interfaces.section import ISectionSet61from lp.soyuz.interfaces.section import ISectionSet
48from lp.soyuz.interfaces.sourcepackageformat import (62from lp.soyuz.interfaces.sourcepackageformat import (
49 ISourcePackageFormatSelectionSet, SourcePackageFormat)63 ISourcePackageFormatSelectionSet, SourcePackageFormat)
50from lp.soyuz.model.section import SectionSelection64from lp.soyuz.model.section import SectionSelection
51from lp.soyuz.model.component import ComponentSelection65from lp.soyuz.model.component import ComponentSelection
66from lp.testing.factory import LaunchpadObjectFactory
67
68
69user_name = 'ppa-user'
70default_email = '%s@example.com' % user_name
5271
5372
54class DoNotRunOnProduction(Exception):73class DoNotRunOnProduction(Exception):
@@ -82,7 +101,7 @@
82 # For some configs it's just absolutely clear this script shouldn't101 # For some configs it's just absolutely clear this script shouldn't
83 # run. Don't even accept --force there.102 # run. Don't even accept --force there.
84 forbidden_configs = re.compile('(edge|lpnet|production)')103 forbidden_configs = re.compile('(edge|lpnet|production)')
85 current_config = getenv('LPCONFIG', 'an unknown config')104 current_config = os.getenv('LPCONFIG', 'an unknown config')
86 if forbidden_configs.match(current_config):105 if forbidden_configs.match(current_config):
87 raise DoNotRunOnProduction(106 raise DoNotRunOnProduction(
88 "I won't delete Ubuntu data on %s and you can't --force me."107 "I won't delete Ubuntu data on %s and you can't --force me."
@@ -95,22 +114,31 @@
95 :return: (options, args, logger)114 :return: (options, args, logger)
96 """115 """
97 parser = OptionParser(116 parser = OptionParser(
98 description="Delete existing Ubuntu releases and set up new ones.")117 description="Set up fresh Ubuntu series and %s identity." % user_name)
99 parser.add_option('-f', '--force', action='store_true', dest='force',118 parser.add_option('-f', '--force', action='store_true', dest='force',
100 help="DANGEROUS: run even if the database looks production-like.")119 help="DANGEROUS: run even if the database looks production-like.")
101 parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',120 parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
102 help="Do not commit changes.")121 help="Do not commit changes.")
122 parser.add_option('-A', '--amd64', action='store_true', dest='amd64',
123 help="Support amd64 architecture.")
124 parser.add_option('-e', '--email', action='store', dest='email',
125 default=default_email,
126 help=(
127 "Email address to use for %s. Should match your GPG key."
128 % user_name))
129
103 logger_options(parser)130 logger_options(parser)
104131
105 options, args = parser.parse_args(arguments)132 options, args = parser.parse_args(arguments)
133
106 return options, args, logger(options)134 return options, args, logger(options)
107135
108136
109def get_person(name):137def get_person_set():
110 """Return `IPersonSet` utility."""138 """Return `IPersonSet` utility."""
111 # Avoid circular import.139 # Avoid circular import.
112 from lp.registry.interfaces.person import IPersonSet140 from lp.registry.interfaces.person import IPersonSet
113 return getUtility(IPersonSet).getByName(name)141 return getUtility(IPersonSet)
114142
115143
116def retire_series(distribution):144def retire_series(distribution):
@@ -150,6 +178,26 @@
150components = main restricted universe multiverse'''178components = main restricted universe multiverse'''
151179
152180
181def add_architecture(distroseries, architecture_name):
182 """Add a DistroArchSeries for the given architecture to `distroseries`."""
183 # Avoid circular import.
184 from lp.soyuz.model.distroarchseries import DistroArchSeries
185
186 store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
187 family = getUtility(IProcessorFamilySet).getByName(architecture_name)
188 archseries = DistroArchSeries(
189 distroseries=distroseries, processorfamily=family,
190 owner=distroseries.owner, official=True,
191 architecturetag=architecture_name)
192 store.add(archseries)
193
194
195def add_architectures(distroseries, options):
196 """Add support for additional architectures as specified by options."""
197 if options.amd64:
198 add_architecture(distroseries, 'amd64')
199
200
153def create_sections(distroseries):201def create_sections(distroseries):
154 """Set up some sections for `distroseries`."""202 """Set up some sections for `distroseries`."""
155 section_names = (203 section_names = (
@@ -253,7 +301,7 @@
253 # published binaries without corresponding sources.301 # published binaries without corresponding sources.
254302
255 log.info("Deleting all items in official archives...")303 log.info("Deleting all items in official archives...")
256 retire_distro_archives(distribution, get_person('name16'))304 retire_distro_archives(distribution, get_person_set().getByName('name16'))
257305
258 # Disable publishing of all PPAs, as they probably have broken306 # Disable publishing of all PPAs, as they probably have broken
259 # publishings too.307 # publishings too.
@@ -271,7 +319,7 @@
271 utility.add(distroseries, format)319 utility.add(distroseries, format)
272320
273321
274def populate(distribution, parent_series_name, uploader_name, log):322def populate(distribution, parent_series_name, uploader_name, options, log):
275 """Set up sample data on `distribution`."""323 """Set up sample data on `distribution`."""
276 parent_series = distribution.getSeries(parent_series_name)324 parent_series = distribution.getSeries(parent_series_name)
277325
@@ -281,15 +329,115 @@
281329
282 log.info("Configuring sections...")330 log.info("Configuring sections...")
283 create_sections(parent_series)331 create_sections(parent_series)
332 add_architectures(parent_series, options)
284333
285 log.info("Configuring components and permissions...")334 log.info("Configuring components and permissions...")
286 create_components(parent_series, get_person(uploader_name))335 uploader = get_person_set().getByName(uploader_name)
336 create_components(parent_series, uploader)
287337
288 set_source_package_format(parent_series)338 set_source_package_format(parent_series)
289339
290 create_sample_series(parent_series, log)340 create_sample_series(parent_series, log)
291341
292342
343def sign_code_of_conduct(person, log):
344 """Sign Ubuntu Code of Conduct for `person`, if necessary."""
345 if person.is_ubuntu_coc_signer:
346 # Already signed.
347 return
348
349 log.info("Signing Ubuntu code of conduct.")
350 signedcocset = getUtility(ISignedCodeOfConductSet)
351 person_id = person.id
352 if signedcocset.searchByUser(person_id).count() == 0:
353 fake_gpg_key = LaunchpadObjectFactory().makeGPGKey(person)
354 Store.of(person).add(SignedCodeOfConduct(
355 owner=person, signingkey=fake_gpg_key,
356 signedcode="Normally a signed CoC would go here.", active=True))
357
358
359def create_ppa_user(username, options, approver, log):
360 """Create new user, with password "test," and sign code of conduct."""
361 # Avoid circular import.
362 from lp.registry.interfaces.person import PersonCreationRationale
363
364 store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
365
366 log.info("Creating %s with email address '%s'." % (
367 user_name, options.email))
368
369 person = get_person_set().getByName(username)
370 if person is None:
371 person, email = get_person_set().createPersonAndEmail(
372 options.email, PersonCreationRationale.OWNER_CREATED_LAUNCHPAD,
373 name=username, displayname=username, password='test')
374 email.status = EmailAddressStatus.PREFERRED
375 email.account.status = AccountStatus.ACTIVE
376
377 if not options.dry_run:
378 transaction.commit()
379
380 sign_code_of_conduct(person, log)
381 get_person_set().getByName('ubuntu-team').addMember(person, approver)
382
383 return person
384
385
386def parse_fingerprints(gpg_output):
387 """Find key fingerprints in "gpg --fingerprint <email>" output."""
388 line_prefix = re.compile('\s*Key fingerprint\s*=\s*')
389 return [
390 ''.join(re.sub(line_prefix, '', line).split())
391 for line in gpg_output.splitlines()
392 if line_prefix.match(line)
393 ]
394
395
396def add_gpg_key(person, fingerprint, log):
397 """Add the GPG key with the given fingerprint to `person`."""
398 log.info("Adding GPG key %s" % fingerprint)
399 gpghandler = getUtility(IGPGHandler)
400 key = gpghandler.retrieveKey(fingerprint)
401
402 gpgkeyset = getUtility(IGPGKeySet)
403 if gpgkeyset.getByFingerprint(fingerprint) is not None:
404 # We already have this key.
405 return
406
407 algorithm = GPGKeyAlgorithm.items[key.algorithm]
408 can_encrypt = True
409 lpkey = gpgkeyset.new(
410 person.id, key.keyid, fingerprint, key.keysize, algorithm,
411 active=True, can_encrypt=can_encrypt)
412 Store.of(person).add(lpkey)
413 log.info("Created Launchpad key %s" % lpkey.displayname)
414
415
416def attach_gpg_keys(options, person, log):
417 """Attach the selected GPG key to `person`."""
418 log.info("Looking for GPG keys.")
419
420 # Need to override GNUPGHOME or we'll get a dummy GPG in a temp
421 # directory, which won't find any keys.
422 env = os.environ.copy()
423 if 'GNUPGHOME' in env:
424 del env['GNUPGHOME']
425 pipe = subprocess.Popen(
426 ['gpg', '--fingerprint', options.email], env=env,
427 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
428 stdout, stderr = pipe.communicate()
429 if stderr != '':
430 log.error(stderr)
431 if pipe.returncode != 0:
432 raise Exception("GPG error.")
433
434 fingerprints = parse_fingerprints(stdout)
435 if len(fingerprints) == 0:
436 log.warn("No GPG key fingerprints found!")
437 for fingerprint in fingerprints:
438 add_gpg_key(person, fingerprint, log)
439
440
293def main(argv):441def main(argv):
294 options, args, log = parse_args(argv[1:])442 options, args, log = parse_args(argv[1:])
295443
@@ -302,8 +450,24 @@
302 clean_up(ubuntu, log)450 clean_up(ubuntu, log)
303451
304 # Use Hoary as the root, as Breezy and Grumpy are broken.452 # Use Hoary as the root, as Breezy and Grumpy are broken.
305 populate(ubuntu, 'hoary', 'ubuntu-team', log)453 populate(ubuntu, 'hoary', 'ubuntu-team', options, log)
306454
455 admin = get_person_set().getByName('name16')
456 person = create_ppa_user(user_name, options, admin, log)
457
458 gave_email = (options.email != default_email)
459 print dedent("""
460 Now start your local Launchpad with "make run" and log into
461 https://launchpad.dev/ as "%(email)s" with "test" as the password.
462 Your user name will be %(user_name)s."""
463 % {
464 'email': options.email,
465 'user_name': user_name,
466 })
467
468 if gave_email:
469 attach_gpg_keys(options, person, log)
470
307 if options.dry_run:471 if options.dry_run:
308 txn.abort()472 txn.abort()
309 else:473 else:
@@ -311,6 +475,5 @@
311475
312 log.info("Done.")476 log.info("Done.")
313477
314
315if __name__ == "__main__":478if __name__ == "__main__":
316 main(sys.argv)479 main(sys.argv)
317480
=== added file 'utilities/start-dev-soyuz.sh'
--- utilities/start-dev-soyuz.sh 1970-01-01 00:00:00 +0000
+++ utilities/start-dev-soyuz.sh 2010-03-02 08:52:26 +0000
@@ -0,0 +1,21 @@
1#!/bin/sh -e
2# Start up Soyuz for local testing on a dev machine.
3
4start_twistd() {
5 # Start twistd for service $1.
6 mkdir -p "/var/tmp/$1"
7 echo "Starting $1."
8 bin/twistd \
9 --logfile "/var/tmp/development-$1.log" \
10 --pidfile "/var/tmp/development-$1.pid" \
11 -y "daemons/$1.tac"
12}
13
14start_twistd zeca
15start_twistd buildd-manager
16
17echo "Starting poppy."
18mkdir -p /var/tmp/poppy
19bin/py daemons/poppy-upload.py /var/tmp/poppy/incoming 2121 &
20
21echo "Done."