cupstream2distro Configuration

Merge lp:~mrazik/cupstream2distro-config/mbs-support into lp:cupstream2distro-config

  1. mbs-support
  2. Merge into trunk
Proposed by Martin Mrazik
Status: Merged
Approved by: Francis Ginther
Approved revision: 84
Merged at revision: 77
Proposed branch: lp:~mrazik/cupstream2distro-config/mbs-support
Merge into: lp:cupstream2distro-config
Diff against target: 227 lines (+52/-28)
6 files modified
c2dconfigutils/cu2dUpdateCi.py (+3/-0)
ci/jenkins-templates/mbs-autolanding-config.xml.tmpl (+17/-14)
ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl (+8/-3)
stacks/experimental/100scopes.cfg (+3/-2)
stacks/head/oif.cfg (+1/-4)
tests/test_cu2dUpdateCi.py (+20/-5)
To merge this branch: bzr merge lp:~mrazik/cupstream2distro-config/mbs-support
Reviewer Review Type Date Requested Status
Francis Ginther Approve
PS Jenkins bot continuous-integration Approve
Review via email: mp+153531@code.launchpad.net

Commit message

Adding MBS support (local archive) to cupstream2distro-config. With this everything needed (except for the server side config where the local repo lives) is in cupstream2distro-config tools and tamplates.
Job-specific "local dput" jobs were completely removed and the logic moved to the top-level autolanding job as a build step.

Description of the change

Adding MBS support (local archive) to cupstream2distro-config. With this everything needed (except for the server side config where the local repo lives) is in cupstream2distro-config tools and tamplates.
Job-specific "local dput" jobs were completely removed and the logic moved to the top-level autolanding job as a build step.

To post a comment you must log in.
lp:~mrazik/cupstream2distro-config/mbs-support updated
78. By Martin Mrazik

removing dput_job from autolanding template

79. By Martin Mrazik

fixed tests

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:79
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/28/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-ci/28/console

Click here to trigger a rebuild:
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/28/rebuild

review: Approve (continuous-integration)
lp:~mrazik/cupstream2distro-config/mbs-support updated
80. By Martin Mrazik

renamed "project" variable to "project_name" and it is now available in all jobs

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:80
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/29/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-ci/29/console

Click here to trigger a rebuild:
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/29/rebuild

review: Approve (continuous-integration)
Revision history for this message
Martin Mrazik (mrazik) wrote :

btw. I just deployed this for the 100scopes stack for some real, production testing ;)

lp:~mrazik/cupstream2distro-config/mbs-support updated
81. By Martin Mrazik

using $JOB_NAME instead of $flavor for the tmp dir

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:81
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/30/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-ci/30/console

Click here to trigger a rebuild:
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/30/rebuild

review: Approve (continuous-integration)
lp:~mrazik/cupstream2distro-config/mbs-support updated
82. By Martin Mrazik

merging with trunk

83. By Martin Mrazik

oif stack migration to the new mbs jobs

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:83
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/31/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-ci/31/console

Click here to trigger a rebuild:
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/31/rebuild

review: Approve (continuous-integration)
Revision history for this message
Francis Ginther (fginther) wrote :

The changes look good, the template changes adequately replace the old dput-mbs-pt jobs. There are some pep8 errors so marking as needs-fixing:
$ pep8 .
./tests/test_cu2dUpdateCi.py:82:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:92:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:104:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:119:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:148:63: E231 missing whitespace after ','

This replaces the bulk of the mbs jenkins jobs. The missing piece is the automatic rebuild of downstream dependencies. For example, if a new compiz is built, the old mbs jobs will rebuild unity with the unity-build-mbs-pt job. There is also the *-check-mbs-pt jobs which are used to defer a rebuild if another upstream MP is pending, but these should be removed anyway. If we care about unintended API breakages, we still have some work to do (open a bug for this?).

review: Needs Fixing
lp:~mrazik/cupstream2distro-config/mbs-support updated
84. By Martin Mrazik

fixed pep8 issues

Revision history for this message
Martin Mrazik (mrazik) wrote :

> The changes look good, the template changes adequately replace the old dput-
> mbs-pt jobs. There are some pep8 errors so marking as needs-fixing:
> $ pep8 .
> ./tests/test_cu2dUpdateCi.py:82:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:92:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:104:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:119:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:148:63: E231 missing whitespace after ','

Oops. Fixed.

> This replaces the bulk of the mbs jenkins jobs. The missing piece is the
> automatic rebuild of downstream dependencies. For example, if a new compiz is
> built, the old mbs jobs will rebuild unity with the unity-build-mbs-pt job.

Is this a regression in the context of lp:cupstream2distro-config? I was actually wondering about this earlier today and had an impression the current oif deployment suffers from the same problem?

> There is also the *-check-mbs-pt jobs which are used to defer a rebuild if
> another upstream MP is pending, but these should be removed anyway. If we care
> about unintended API breakages, we still have some work to do (open a bug for
> this?).

If its not a regression (in lp:cupstream2distro-config) then yes, we should open a bug.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:84
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/32/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-ci/32/console

Click here to trigger a rebuild:
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/32/rebuild

review: Approve (continuous-integration)
Revision history for this message
Francis Ginther (fginther) wrote :

> Oops. Fixed.

Thanks

> > This replaces the bulk of the mbs jenkins jobs. The missing piece is the
> > automatic rebuild of downstream dependencies. For example, if a new compiz
> is
> > built, the old mbs jobs will rebuild unity with the unity-build-mbs-pt job.
>
> Is this a regression in the context of lp:cupstream2distro-config? I was
> actually wondering about this earlier today and had an impression the current
> oif deployment suffers from the same problem?

With this MP, the oif stack will be broken. The old dput jobs (i.e. frame-dput-mbs-pt) have additional bits to call the other mbs jobs. This has really only been useful for unity, which we can manually hack into the jenkins jobs if necessary.

> > There is also the *-check-mbs-pt jobs which are used to defer a rebuild if
> > another upstream MP is pending, but these should be removed anyway. If we
> care
> > about unintended API breakages, we still have some work to do (open a bug
> for
> > this?).
>
> If its not a regression (in lp:cupstream2distro-config) then yes, we should
> open a bug.

Revision history for this message
Martin Mrazik (mrazik) wrote :

I've created https://bugs.launchpad.net/cupstream2distro-config/+bug/1155676 to address the remaining issues.

Revision history for this message
Francis Ginther (fginther) wrote :

Approve

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'c2dconfigutils/cu2dUpdateCi.py'
2--- c2dconfigutils/cu2dUpdateCi.py 2013-03-14 08:11:12 +0000
3+++ c2dconfigutils/cu2dUpdateCi.py 2013-03-15 15:29:21 +0000
4@@ -174,6 +174,9 @@
5 for parameter in parameters:
6 ctx[parameter] = parameters[parameter]
7 self.add_parameter(ctx, parameter, parameters[parameter])
8+ # add project name as well
9+ ctx['project_name'] = project_name
10+ self.add_parameter(ctx, 'project_name', project_name)
11 return ctx
12
13 def generate_jobs(self, job_list, project_name, job_type, job_config,
14
15=== modified file 'ci/jenkins-templates/mbs-autolanding-config.xml.tmpl'
16--- ci/jenkins-templates/mbs-autolanding-config.xml.tmpl 2013-03-14 18:53:34 +0000
17+++ ci/jenkins-templates/mbs-autolanding-config.xml.tmpl 2013-03-15 15:29:21 +0000
18@@ -45,6 +45,9 @@
19 <hudson.plugins.parameterizedtrigger.BlockableBuildTriggerConfig>
20 <configs>
21 <hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
22+ <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
23+ <properties>UPSTREAM_BUILD_NUMBER=$BUILD_NUMBER</properties>
24+ </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
25 </configs>
26 <projects>{{ builder_list }}</projects>
27 <condition>ALWAYS</condition>
28@@ -70,6 +73,20 @@
29 </hudson.plugins.parameterizedtrigger.BlockableBuildTriggerConfig>
30 </configs>
31 </hudson.plugins.parameterizedtrigger.TriggerBuilder>
32+ <hudson.tasks.Shell>
33+ <command>#!/bin/bash
34+set -x
35+
36+temp_dir=${local_archive_tmp}/${project_name}-al-${local_archive_pocket}-${BUILD_NUMBER}
37+ppa_build_number=${BUILD_NUMBER}
38+
39+
40+for p in `ssh ${local_archive_login}@${local_archive_host} ls ${temp_dir}`; do
41+ ssh ${local_archive_login}@${local_archive_host} dput -ufU --config /home/ubuntu/.dput.cf ${local_archive} ${temp_dir}/${p}/*.changes
42+ # Execute mini-dinstall to update the local archive
43+ ssh ${local_archive_login}@${local_archive_host} mini-dinstall --no-db --batch --config=&quot;/home/ubuntu/.mini-dinstall-${local_archive}.conf&quot;
44+done</command>
45+ </hudson.tasks.Shell>
46 {% if aggregate_tests_script %}
47 <hudson.tasks.Shell>
48 <command>{{ aggregate_tests_script }}</command>
49@@ -149,20 +166,6 @@
50 <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
51 <properties>local_archive_host=$local_archive_host
52 local_archive_login=$local_archive_login
53-temp_dir=${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}
54-ppa_build_number=${BUILD_NUMBER}
55-distributions=</properties>
56- </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
57- </configs>
58- <projects>{{ dput_job }}</projects>
59- <condition>SUCCESS</condition>
60- <triggerWithNoParameters>false</triggerWithNoParameters>
61- </hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
62- <hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
63- <configs>
64- <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
65- <properties>local_archive_host=$local_archive_host
66-local_archive_login=$local_archive_login
67 temp_dir=${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}</properties>
68 </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
69 </configs>
70
71=== modified file 'ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl'
72--- ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl 2013-03-14 14:59:37 +0000
73+++ ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl 2013-03-15 15:29:21 +0000
74@@ -17,6 +17,11 @@
75 <description>Specifies that this is an autolanding/merge job. Used to properly set package version string.</description>
76 <defaultValue></defaultValue>
77 </hudson.model.StringParameterDefinition>
78+ <hudson.model.StringParameterDefinition>
79+ <name>UPSTREAM_BUILD_NUMBER</name>
80+ <description>Build number of the upstream job used in the tmp directory with the deb output</description>
81+ <defaultValue></defaultValue>
82+ </hudson.model.StringParameterDefinition>
83 {% for parameter in parameter_list %}
84 <hudson.model.StringParameterDefinition>
85 <name>{{ parameter.name }}</name>
86@@ -62,17 +67,17 @@
87 exit 0
88 fi
89
90-tmp_dir=&quot;${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}&quot;
91+tmp_dir=&quot;${local_archive_tmp}/${project_name}-al-${local_archive_pocket}-${UPSTREAM_BUILD_NUMBER}&quot;
92
93 # mkdir &quot;${tmp_dir}&quot; can fail due to multiple jobs attempting to create the directory
94 ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}&quot; || true
95-ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}/${flavor}&quot;
96+ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}/${JOB_NAME}&quot;
97
98 # Switch to the output directory containing the build results
99 cd work/output
100
101 # Copy the build results to the temporary holding area, if all jobs succeed, these will be dput into the ppa archive.
102-scp * ${local_archive_login}@${local_archive_host}:${tmp_dir}/${flavor}</command>
103+scp * ${local_archive_login}@${local_archive_host}:${tmp_dir}/${JOB_NAME}</command>
104 </hudson.tasks.Shell>
105 </builders>
106 <publishers>
107
108=== modified file 'stacks/experimental/100scopes.cfg'
109--- stacks/experimental/100scopes.cfg 2013-03-15 13:31:45 +0000
110+++ stacks/experimental/100scopes.cfg 2013-03-15 15:29:21 +0000
111@@ -6,7 +6,8 @@
112 dest: ubuntu-unity/experimental-certified
113 schedule: 0 7 * * 1-5
114 ci_default:
115- hooks: D08add_ppa-ubuntu-unity-experimental-prevalidation #TODO: anythin mbs specific?
116+ hooks: D08add_ppa-ubuntu-unity-experimental-prevalidation D00mbs_archive H05set_package_version H40disable_gensymbols_check
117+ hook_source: lp:~private-ps-quality-team/+junk/scopes-hooks
118 contact_email: "martin.mrazik@canonical.com"
119 autolanding_template: mbs-autolanding-config.xml.tmpl
120 configurations:
121@@ -16,9 +17,9 @@
122 local_archive_host: naartjie
123 local_archive_login: ubuntu
124 local_archive_tmp: /home/ubuntu/ppa_archive/100scopes-experimental
125+ local_archive: 100scopes-experimental
126 local_archive_source: http://naartjie/archive/100scopes-experimental
127 local_archive_pocket: raring
128- #TODO: what about dput_job
129 projects:
130 libunity:
131 target_branch: lp:~unity-team/libunity/libunity-7.0
132
133=== modified file 'stacks/head/oif.cfg'
134--- stacks/head/oif.cfg 2013-03-12 17:42:05 +0000
135+++ stacks/head/oif.cfg 2013-03-15 15:29:21 +0000
136@@ -28,6 +28,7 @@
137 local_archive_host: naartjie
138 local_archive_login: ubuntu
139 local_archive_tmp: /home/ubuntu/ppa_archive/oif-staging
140+ local_archive: oif-staging
141 local_archive_source: http://naartjie/archive/oif-staging
142 local_archive_pocket: raring
143 dependencies:
144@@ -43,10 +44,6 @@
145 raring-armhf:
146 template: pbuilder-config.xml.tmpl
147 frame:
148- dput_job: frame-dput-mbs-pt
149 geis:
150- dput_job: geis-dput-mbs-pt
151 grail:
152- dput_job: grail-dput-mbs-pt
153 libgrip:
154- dput_job: libgrip-dput-mbs-pt
155
156=== modified file 'tests/test_cu2dUpdateCi.py'
157--- tests/test_cu2dUpdateCi.py 2013-03-14 02:44:09 +0000
158+++ tests/test_cu2dUpdateCi.py 2013-03-15 15:29:21 +0000
159@@ -75,8 +75,11 @@
160 def test_project_config_implied_target_branch(self):
161 config = {}
162 expected = {'target_branch': 'lp:project',
163+ 'project_name': 'project',
164 'parameter_list': [JobParameter('target_branch',
165- 'lp:project')]}
166+ 'lp:project'),
167+ JobParameter('project_name',
168+ 'project')]}
169 actual = self.update_ci.process_project_config('project', config)
170 self.assertEqual(expected, actual)
171
172@@ -84,15 +87,21 @@
173 config = {'target_branch': 'lp:project/sub'}
174 expected = {'target_branch': 'lp:project/sub',
175 'parameter_list': [JobParameter('target_branch',
176- 'lp:project/sub')]}
177+ 'lp:project/sub'),
178+ JobParameter('project_name',
179+ 'project')],
180+ 'project_name': 'project'}
181 actual = self.update_ci.process_project_config('project', config)
182 self.assertEqual(expected, actual)
183
184 def test_project_config_ctx_only(self):
185 config = {'team': 'team_name'}
186 expected = {'target_branch': 'lp:project',
187+ 'project_name': 'project',
188 'parameter_list': [JobParameter('target_branch',
189- 'lp:project')],
190+ 'lp:project'),
191+ JobParameter('project_name',
192+ 'project')],
193 'team': 'team_name'}
194 actual = self.update_ci.process_project_config('project', config)
195 self.assertEqual(expected, actual)
196@@ -101,10 +110,13 @@
197 config = {'some_parameter': 'some_value'}
198 expected = {'target_branch': 'lp:project',
199 'some_parameter': 'some_value',
200+ 'project_name': 'project',
201 'parameter_list': [JobParameter('target_branch',
202 'lp:project'),
203 JobParameter('some_parameter',
204- 'some_value')]}
205+ 'some_value'),
206+ JobParameter('project_name',
207+ 'project')]}
208 actual = self.update_ci.process_project_config('project', config)
209 self.assertEqual(expected, actual)
210
211@@ -125,12 +137,15 @@
212 expected = {'hook_location': '/tmp/$JOB_NAME-hooks',
213 'target_branch': 'lp:project',
214 'hook_source': 'lp:hooks',
215+ 'project_name': 'project',
216 'acquire_hook_script': script.format(
217 DEFAULT_HOOK_LOCATION='/tmp/$JOB_NAME-hooks'),
218 'parameter_list': [JobParameter('target_branch',
219 'lp:project'),
220 JobParameter('hook_source',
221- 'lp:hooks')]}
222+ 'lp:hooks'),
223+ JobParameter('project_name',
224+ 'project')]}
225 with patch('c2dconfigutils.cu2dUpdateCi.open',
226 create=True) as mock_open:
227 mock_open.return_value = MagicMock(spec=file)

Subscribers

People subscribed via source and target branches

to all changes: