Offspring

Merge lp:~salgado/offspring/path-independence into lp:offspring

  1. path-independence
  2. Merge into trunk
Proposed by Guilherme Salgado
Status: Superseded
Proposed branch: lp:~salgado/offspring/path-independence
Merge into: lp:offspring
Prerequisite: lp:~salgado/offspring/all-python-config-plus
Diff against target: 287 lines (+57/-23)
12 files modified
.bzrignore (+2/-0)
Makefile (+1/-1)
config/offspring.cfg (+11/-3)
lib/offspring/__init__.py (+5/-0)
lib/offspring/build/bin/offspring-build (+7/-1)
lib/offspring/build/scripts/archive-builds (+7/-3)
lib/offspring/config.py (+8/-4)
lib/offspring/slave/slave.py (+6/-2)
lib/offspring/slave/taskmonitor.py (+8/-2)
lib/offspring/web/queuemanager/models.py (+2/-1)
lib/offspring/web/settings.py (+0/-3)
lib/offspring/web/settings_production.py (+0/-3)
To merge this branch: bzr merge lp:~salgado/offspring/path-independence
Reviewer Review Type Date Requested Status
Offspring Hackers Pending
Review via email: mp+76277@code.launchpad.net

This proposal has been superseded by a proposal from 2011-09-21.

Description of the change

This is a second attempt at trying to get these changes merged; they simplify things a lot for developers.

Here's what Michael said in the original merge proposal:

This finally is an I think ready branch to allow offspring to be easily run from places other than /srv.

The code to find offspring-builder-config is a touch horrible, but it won't be needed in the slave.

To post a comment you must log in.
lp:~salgado/offspring/path-independence updated
29. By Guilherme Salgado

merge lp:~mwhudson/offspring/path-independence

30. By Guilherme Salgado

Fix one final place which was using offspring-builder-script instead of offspring-builder-config

31. By Guilherme Salgado

merge all-python-config

32. By Guilherme Salgado

merge all-python-config

33. By Guilherme Salgado

Update one remaining place which was still running offspring-builder-config

34. By Guilherme Salgado

merge trunk

35. By Guilherme Salgado

Some minor tweaks

36. By Guilherme Salgado

revert a few unnecessary changes

37. By Guilherme Salgado

remove a .bzrignore change that's no longer needed

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file '.bzrignore'
--- .bzrignore 2010-12-03 04:46:30 +0000
+++ .bzrignore 2011-09-21 20:44:26 +0000
@@ -1,9 +1,11 @@
1./.download-cache/1./.download-cache/
2./.virtualenv/2./.virtualenv/
3./bin/3./bin/
4./cron/
4./logs/5./logs/
5./lib/offspring.egg-info/6./lib/offspring.egg-info/
6./lib/offspring/web/media/js/isomorphic/7./lib/offspring/web/media/js/isomorphic/
7./config/launchpad.oauth8./config/launchpad.oauth
8./builds/9./builds/
10./config/offspring.cfg.*
9./scripts/11./scripts/
1012
=== modified file 'Makefile'
--- Makefile 2011-09-01 14:42:20 +0000
+++ Makefile 2011-09-21 20:44:26 +0000
@@ -18,7 +18,7 @@
18 [ ! -d builds ] || chmod -r builds18 [ ! -d builds ] || chmod -r builds
19 ./.virtualenv/bin/python setup.py develop --script-dir=./bin/19 ./.virtualenv/bin/python setup.py develop --script-dir=./bin/
20 [ ! -d builds ] || chmod +r builds20 [ ! -d builds ] || chmod +r builds
21 [ -f bin/offspring-build ] || ln -s `pwd`/lib/offspring/build/bin/* bin/21 [ -f bin/offspring-build ] || ln -s `pwd`/lib/offspring/build/bin/offspring-build bin/
22 mkdir -p logs22 mkdir -p logs
23 mkdir -p scripts && ln -s --force `pwd`/lib/offspring/build/scripts/* scripts/23 mkdir -p scripts && ln -s --force `pwd`/lib/offspring/build/scripts/* scripts/
2424
2525
=== modified file 'config/offspring.cfg'
--- config/offspring.cfg 2011-09-21 20:44:26 +0000
+++ config/offspring.cfg 2011-09-21 20:44:26 +0000
@@ -11,7 +11,15 @@
11#===============================================================================11#===============================================================================
12# Offspring Global Defaults12# Offspring Global Defaults
13#13#
14# Some options in this section are given defaults by code so that they
15# can be referenced in subsitutions:
16#
14# base_dir: Base directory of offspring install17# base_dir: Base directory of offspring install
18# hostname: The hostname of the running machine
19# username: The name of the user running offspring
20#
21# The other options below are set normally:
22#
15# bin_dir: Directory to offspring binaries23# bin_dir: Directory to offspring binaries
16# config_dir: Directory to offspring configuration files24# config_dir: Directory to offspring configuration files
17# admin_email: E-mail admin notifications should be sent to by default25# admin_email: E-mail admin notifications should be sent to by default
@@ -25,7 +33,6 @@
25#===============================================================================33#===============================================================================
2634
27[DEFAULT]35[DEFAULT]
28base_dir: /srv
29bin_dir: %(base_dir)s/bin36bin_dir: %(base_dir)s/bin
30config_dir: %(base_dir)s/config37config_dir: %(base_dir)s/config
31log_dir: %(base_dir)s/logs38log_dir: %(base_dir)s/logs
@@ -178,6 +185,7 @@
178poll_frequency: 20185poll_frequency: 20
179logfile: %(log_dir)s/offspring-master.log186logfile: %(log_dir)s/offspring-master.log
180launchpad_oauth: %(config_dir)s/launchpad.oauth187launchpad_oauth: %(config_dir)s/launchpad.oauth
188result_directory: %(base_dir)s/builds
181189
182#===============================================================================190#===============================================================================
183# Offspring publisher191# Offspring publisher
@@ -197,8 +205,8 @@
197205
198[publisher]206[publisher]
199db: postgres://offspring:temp1234@localhost/offspring207db: postgres://offspring:temp1234@localhost/offspring
200development_pool: /srv/builds/208development_pool: %(base_dir)s/builds/
201production_pool: /srv/partners/209production_pool: %(base_dir)s/partners/
202signer_directory: 210signer_directory:
203signer_files: 211signer_files:
204signer_results: 212signer_results:
205213
=== modified file 'lib/offspring/__init__.py'
--- lib/offspring/__init__.py 2010-11-29 08:27:24 +0000
+++ lib/offspring/__init__.py 2011-09-21 20:44:26 +0000
@@ -1,4 +1,9 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4import os
5
4__version__ = '0.0.1'6__version__ = '0.0.1'
7
8offspring_root = os.path.abspath(
9 os.path.join(os.path.dirname(__file__), '..', '..'))
510
=== modified file 'lib/offspring/build/bin/offspring-build'
--- lib/offspring/build/bin/offspring-build 2011-09-21 20:44:26 +0000
+++ lib/offspring/build/bin/offspring-build 2011-09-21 20:44:26 +0000
@@ -7,7 +7,13 @@
77
8_Initialize() {8_Initialize() {
9 if [ -z "${OFFSPRING_CONFIGSCRIPT}" ]; then9 if [ -z "${OFFSPRING_CONFIGSCRIPT}" ]; then
10 OFFSPRING_CONFIGSCRIPT="/srv/bin/offspring-builder-config"10 SCRIPT_PATH="$(readlink -e "${0}" || which "${0}")"
11 OFFSPRING_ROOT=$SCRIPT_PATH
12 for i in 1 2 3 4 5;
13 do
14 OFFSPRING_ROOT="$(dirname "${OFFSPRING_ROOT}")"
15 done
16 OFFSPRING_CONFIGSCRIPT="${OFFSPRING_ROOT}/bin/offspring-builder-config"
11 fi17 fi
1218
13 if [ \! -x "${OFFSPRING_CONFIGSCRIPT}" ]; then19 if [ \! -x "${OFFSPRING_CONFIGSCRIPT}" ]; then
1420
=== modified file 'lib/offspring/build/scripts/archive-builds'
--- lib/offspring/build/scripts/archive-builds 2011-09-21 20:44:26 +0000
+++ lib/offspring/build/scripts/archive-builds 2011-09-21 20:44:26 +0000
@@ -9,7 +9,13 @@
9#9#
1010
11if [ -z "${OFFSPRING_CONFIGSCRIPT}" ]; then11if [ -z "${OFFSPRING_CONFIGSCRIPT}" ]; then
12 OFFSPRING_CONFIGSCRIPT="/srv/bin/offspring-builder-config"12 SCRIPT_PATH="$(readlink -e "${0}" || which "${0}")"
13 OFFSPRING_ROOT=$SCRIPT_PATH
14 for i in 1 2 3 4 5;
15 do
16 OFFSPRING_ROOT="$(dirname "${OFFSPRING_ROOT}")"
17 done
18 OFFSPRING_CONFIGSCRIPT="${OFFSPRING_ROOT}/bin/offspring-builder-config"
13fi19fi
1420
15if [ \! -x "${OFFSPRING_CONFIGSCRIPT}" ]; then21if [ \! -x "${OFFSPRING_CONFIGSCRIPT}" ]; then
@@ -24,8 +30,6 @@
24 exit 130 exit 1
25fi31fi
2632
27source "$CONFIGSCRIPT"
28
29# calculateBuildAge DATE33# calculateBuildAge DATE
30calculateBuildAge() {34calculateBuildAge() {
31 date2jd() {35 date2jd() {
3236
=== modified file 'lib/offspring/config.py'
--- lib/offspring/config.py 2011-09-21 20:44:26 +0000
+++ lib/offspring/config.py 2011-09-21 20:44:26 +0000
@@ -8,16 +8,23 @@
8import os8import os
9import sys9import sys
1010
11from offspring import offspring_root
12
11config = None13config = None
1214
13def _read_config():15def _read_config():
14 global config16 global config
15 config = SafeConfigParser()17 config = SafeConfigParser()
18 hostname = os.popen('hostname').read().strip()
19 config.set('DEFAULT', 'base_dir', offspring_root)
20 config.set('DEFAULT', 'hostname', hostname)
21 config.set('DEFAULT', 'username', os.environ['USERNAME'])
1622
17 if os.environ.has_key("OFFSPRING_CONFIGFILE"):23 if os.environ.has_key("OFFSPRING_CONFIGFILE"):
18 CONFIGFILE_PATH = os.environ["OFFSPRING_CONFIGFILE"]24 CONFIGFILE_PATH = os.environ["OFFSPRING_CONFIGFILE"]
19 else:25 else:
20 CONFIGFILE_PATH = "/srv/config/offspring.cfg"26 CONFIGFILE_PATH = os.path.join(
27 offspring_root, 'config', 'offspring.cfg')
21 configFile = None28 configFile = None
22 try:29 try:
23 configFile = open(CONFIGFILE_PATH)30 configFile = open(CONFIGFILE_PATH)
@@ -27,9 +34,6 @@
27 % (CONFIGFILE_PATH, e) )34 % (CONFIGFILE_PATH, e) )
28 sys.exit(1)35 sys.exit(1)
29 else:36 else:
30 hostname = os.popen('hostname').read().strip()
31 config.set('DEFAULT', 'hostname', hostname)
32 config.set('DEFAULT', 'username', os.environ['USERNAME'])
33 config.read([CONFIGFILE_PATH + '.' + hostname])37 config.read([CONFIGFILE_PATH + '.' + hostname])
34 finally:38 finally:
35 if configFile is not None:39 if configFile is not None:
3640
=== modified file 'lib/offspring/slave/slave.py'
--- lib/offspring/slave/slave.py 2011-05-30 22:17:10 +0000
+++ lib/offspring/slave/slave.py 2011-09-21 20:44:26 +0000
@@ -8,11 +8,12 @@
8)8)
99
10import logging10import logging
11import os
11import platform12import platform
12import time13import time
13import xmlrpclib14import xmlrpclib
1415
15from offspring import config16from offspring import config, offspring_root
16from offspring.daemon import (17from offspring.daemon import (
17 LexbuilderDaemon,18 LexbuilderDaemon,
18 LexbuilderXMLRPCServer19 LexbuilderXMLRPCServer
@@ -160,7 +161,10 @@
160161
161 def __init__(self, supervisor, projectName, configBranchURL):162 def __init__(self, supervisor, projectName, configBranchURL):
162 MonitoredProcess.__init__(163 MonitoredProcess.__init__(
163 self, supervisor, [projectName, configBranchURL])164 self, supervisor, [projectName, configBranchURL],
165 environment_updates={
166 'OFFSPRING_CONFIGSCRIPT': os.path.join(
167 offspring_root, 'bin', 'offspring-builder-config')})
164 self.projectName = projectName168 self.projectName = projectName
165169
166 def setup(self):170 def setup(self):
167171
=== modified file 'lib/offspring/slave/taskmonitor.py'
--- lib/offspring/slave/taskmonitor.py 2011-09-08 19:05:50 +0000
+++ lib/offspring/slave/taskmonitor.py 2011-09-21 20:44:26 +0000
@@ -2,6 +2,7 @@
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4import datetime4import datetime
5import os
5import threading6import threading
6import subprocess7import subprocess
78
@@ -54,19 +55,24 @@
5455
55class MonitoredProcess(MonitoredThread):56class MonitoredProcess(MonitoredThread):
56 def __init__(self, master=None, arguments=[], shell=False,57 def __init__(self, master=None, arguments=[], shell=False,
57 stdout=subprocess.PIPE, stderr=subprocess.PIPE):58 stdout=subprocess.PIPE, stderr=subprocess.PIPE,
59 environment_updates={}):
58 self.process = None60 self.process = None
59 self.shell = shell61 self.shell = shell
60 self.stdout = stdout62 self.stdout = stdout
61 self.stderr = stderr63 self.stderr = stderr
64 self.environment_updates = environment_updates
62 MonitoredThread.__init__(self, master, arguments)65 MonitoredThread.__init__(self, master, arguments)
6366
64 def execute(self):67 def execute(self):
68 child_environ = os.environ.copy()
69 child_environ.update(self.environment_updates)
65 self.process = subprocess.Popen(70 self.process = subprocess.Popen(
66 [self.getCommandPath()] + self.arguments,71 [self.getCommandPath()] + self.arguments,
67 shell=self.shell,72 shell=self.shell,
68 stdout=self.stdout,73 stdout=self.stdout,
69 stderr=self.stderr)74 stderr=self.stderr,
75 env=child_environ)
70 self.stdoutdata, self.stderrdata = self.process.communicate()76 self.stdoutdata, self.stderrdata = self.process.communicate()
71 return self.process.returncode77 return self.process.returncode
7278
7379
=== modified file 'lib/offspring/web/queuemanager/models.py'
--- lib/offspring/web/queuemanager/models.py 2011-02-22 09:46:13 +0000
+++ lib/offspring/web/queuemanager/models.py 2011-09-21 20:44:26 +0000
@@ -19,6 +19,7 @@
19 models19 models
20)20)
2121
22from offspring import config
22from offspring.enums import ProjectBuildStates23from offspring.enums import ProjectBuildStates
2324
2425
@@ -327,7 +328,7 @@
327 def result_directory(self):328 def result_directory(self):
328 if self.name:329 if self.name:
329 try:330 try:
330 base_dir = getattr(settings, 'BUILDRESULTS_DIRECTORY', '/srv/builds/')331 base_dir = config.master('result_directory')
331 buildDate, buildId = self.name.rsplit('-', 1)332 buildDate, buildId = self.name.rsplit('-', 1)
332 except:333 except:
333 return None334 return None
334335
=== modified file 'lib/offspring/web/settings.py'
--- lib/offspring/web/settings.py 2011-08-11 00:16:09 +0000
+++ lib/offspring/web/settings.py 2011-09-21 20:44:26 +0000
@@ -130,6 +130,3 @@
130CELERY_RESULT_BACKEND = 'djcelery.backends.database.DatabaseBackend'130CELERY_RESULT_BACKEND = 'djcelery.backends.database.DatabaseBackend'
131CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'131CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
132djcelery.setup_loader()132djcelery.setup_loader()
133
134# Path to build results
135BUILDRESULTS_DIRECTORY = '/srv/builds/'
136133
=== modified file 'lib/offspring/web/settings_production.py'
--- lib/offspring/web/settings_production.py 2010-11-29 08:27:24 +0000
+++ lib/offspring/web/settings_production.py 2011-09-21 20:44:26 +0000
@@ -16,6 +16,3 @@
16EMAIL_HOST = 'localhost'16EMAIL_HOST = 'localhost'
1717
18MEDIA_URL = 'https://offspring.com/assets/'18MEDIA_URL = 'https://offspring.com/assets/'
19
20# Path to build results
21BUILDRESULTS_DIRECTORY = '/srv/offspring.com/www/builds/'

Subscribers

People subscribed via source and target branches