Merge lp:~mrazik/cupstream2distro-config/mbs-support into lp:cupstream2distro-config

Proposed by Martin Mrazik
Status: Merged
Approved by: Francis Ginther
Approved revision: 84
Merged at revision: 77
Proposed branch: lp:~mrazik/cupstream2distro-config/mbs-support
Merge into: lp:cupstream2distro-config
Diff against target: 227 lines (+52/-28)
6 files modified
c2dconfigutils/cu2dUpdateCi.py (+3/-0)
ci/jenkins-templates/mbs-autolanding-config.xml.tmpl (+17/-14)
ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl (+8/-3)
stacks/experimental/100scopes.cfg (+3/-2)
stacks/head/oif.cfg (+1/-4)
tests/test_cu2dUpdateCi.py (+20/-5)
To merge this branch: bzr merge lp:~mrazik/cupstream2distro-config/mbs-support
Reviewer Review Type Date Requested Status
Francis Ginther Approve
PS Jenkins bot continuous-integration Approve
Review via email: mp+153531@code.launchpad.net

Commit message

Adding MBS support (local archive) to cupstream2distro-config. With this everything needed (except for the server side config where the local repo lives) is in cupstream2distro-config tools and tamplates.
Job-specific "local dput" jobs were completely removed and the logic moved to the top-level autolanding job as a build step.

Description of the change

Adding MBS support (local archive) to cupstream2distro-config. With this everything needed (except for the server side config where the local repo lives) is in cupstream2distro-config tools and tamplates.
Job-specific "local dput" jobs were completely removed and the logic moved to the top-level autolanding job as a build step.

To post a comment you must log in.
78. By Martin Mrazik

removing dput_job from autolanding template

79. By Martin Mrazik

fixed tests

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
80. By Martin Mrazik

renamed "project" variable to "project_name" and it is now available in all jobs

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
Martin Mrazik (mrazik) wrote :

btw. I just deployed this for the 100scopes stack for some real, production testing ;)

81. By Martin Mrazik

using $JOB_NAME instead of $flavor for the tmp dir

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
82. By Martin Mrazik

merging with trunk

83. By Martin Mrazik

oif stack migration to the new mbs jobs

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
Francis Ginther (fginther) wrote :

The changes look good, the template changes adequately replace the old dput-mbs-pt jobs. There are some pep8 errors so marking as needs-fixing:
$ pep8 .
./tests/test_cu2dUpdateCi.py:82:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:92:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:104:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:119:63: E231 missing whitespace after ','
./tests/test_cu2dUpdateCi.py:148:63: E231 missing whitespace after ','

This replaces the bulk of the mbs jenkins jobs. The missing piece is the automatic rebuild of downstream dependencies. For example, if a new compiz is built, the old mbs jobs will rebuild unity with the unity-build-mbs-pt job. There is also the *-check-mbs-pt jobs which are used to defer a rebuild if another upstream MP is pending, but these should be removed anyway. If we care about unintended API breakages, we still have some work to do (open a bug for this?).

review: Needs Fixing
84. By Martin Mrazik

fixed pep8 issues

Revision history for this message
Martin Mrazik (mrazik) wrote :

> The changes look good, the template changes adequately replace the old dput-
> mbs-pt jobs. There are some pep8 errors so marking as needs-fixing:
> $ pep8 .
> ./tests/test_cu2dUpdateCi.py:82:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:92:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:104:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:119:63: E231 missing whitespace after ','
> ./tests/test_cu2dUpdateCi.py:148:63: E231 missing whitespace after ','

Oops. Fixed.

> This replaces the bulk of the mbs jenkins jobs. The missing piece is the
> automatic rebuild of downstream dependencies. For example, if a new compiz is
> built, the old mbs jobs will rebuild unity with the unity-build-mbs-pt job.

Is this a regression in the context of lp:cupstream2distro-config? I was actually wondering about this earlier today and had an impression the current oif deployment suffers from the same problem?

> There is also the *-check-mbs-pt jobs which are used to defer a rebuild if
> another upstream MP is pending, but these should be removed anyway. If we care
> about unintended API breakages, we still have some work to do (open a bug for
> this?).

If its not a regression (in lp:cupstream2distro-config) then yes, we should open a bug.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
Francis Ginther (fginther) wrote :

> Oops. Fixed.

Thanks

> > This replaces the bulk of the mbs jenkins jobs. The missing piece is the
> > automatic rebuild of downstream dependencies. For example, if a new compiz
> is
> > built, the old mbs jobs will rebuild unity with the unity-build-mbs-pt job.
>
> Is this a regression in the context of lp:cupstream2distro-config? I was
> actually wondering about this earlier today and had an impression the current
> oif deployment suffers from the same problem?

With this MP, the oif stack will be broken. The old dput jobs (i.e. frame-dput-mbs-pt) have additional bits to call the other mbs jobs. This has really only been useful for unity, which we can manually hack into the jenkins jobs if necessary.

> > There is also the *-check-mbs-pt jobs which are used to defer a rebuild if
> > another upstream MP is pending, but these should be removed anyway. If we
> care
> > about unintended API breakages, we still have some work to do (open a bug
> for
> > this?).
>
> If its not a regression (in lp:cupstream2distro-config) then yes, we should
> open a bug.

Revision history for this message
Martin Mrazik (mrazik) wrote :

I've created https://bugs.launchpad.net/cupstream2distro-config/+bug/1155676 to address the remaining issues.

Revision history for this message
Francis Ginther (fginther) wrote :

Approve

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'c2dconfigutils/cu2dUpdateCi.py'
--- c2dconfigutils/cu2dUpdateCi.py 2013-03-14 08:11:12 +0000
+++ c2dconfigutils/cu2dUpdateCi.py 2013-03-15 15:29:21 +0000
@@ -174,6 +174,9 @@
174 for parameter in parameters:174 for parameter in parameters:
175 ctx[parameter] = parameters[parameter]175 ctx[parameter] = parameters[parameter]
176 self.add_parameter(ctx, parameter, parameters[parameter])176 self.add_parameter(ctx, parameter, parameters[parameter])
177 # add project name as well
178 ctx['project_name'] = project_name
179 self.add_parameter(ctx, 'project_name', project_name)
177 return ctx180 return ctx
178181
179 def generate_jobs(self, job_list, project_name, job_type, job_config,182 def generate_jobs(self, job_list, project_name, job_type, job_config,
180183
=== modified file 'ci/jenkins-templates/mbs-autolanding-config.xml.tmpl'
--- ci/jenkins-templates/mbs-autolanding-config.xml.tmpl 2013-03-14 18:53:34 +0000
+++ ci/jenkins-templates/mbs-autolanding-config.xml.tmpl 2013-03-15 15:29:21 +0000
@@ -45,6 +45,9 @@
45 <hudson.plugins.parameterizedtrigger.BlockableBuildTriggerConfig>45 <hudson.plugins.parameterizedtrigger.BlockableBuildTriggerConfig>
46 <configs>46 <configs>
47 <hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>47 <hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
48 <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
49 <properties>UPSTREAM_BUILD_NUMBER=$BUILD_NUMBER</properties>
50 </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
48 </configs>51 </configs>
49 <projects>{{ builder_list }}</projects>52 <projects>{{ builder_list }}</projects>
50 <condition>ALWAYS</condition>53 <condition>ALWAYS</condition>
@@ -70,6 +73,20 @@
70 </hudson.plugins.parameterizedtrigger.BlockableBuildTriggerConfig>73 </hudson.plugins.parameterizedtrigger.BlockableBuildTriggerConfig>
71 </configs>74 </configs>
72 </hudson.plugins.parameterizedtrigger.TriggerBuilder>75 </hudson.plugins.parameterizedtrigger.TriggerBuilder>
76 <hudson.tasks.Shell>
77 <command>#!/bin/bash
78set -x
79
80temp_dir=${local_archive_tmp}/${project_name}-al-${local_archive_pocket}-${BUILD_NUMBER}
81ppa_build_number=${BUILD_NUMBER}
82
83
84for p in `ssh ${local_archive_login}@${local_archive_host} ls ${temp_dir}`; do
85 ssh ${local_archive_login}@${local_archive_host} dput -ufU --config /home/ubuntu/.dput.cf ${local_archive} ${temp_dir}/${p}/*.changes
86 # Execute mini-dinstall to update the local archive
87 ssh ${local_archive_login}@${local_archive_host} mini-dinstall --no-db --batch --config=&quot;/home/ubuntu/.mini-dinstall-${local_archive}.conf&quot;
88done</command>
89 </hudson.tasks.Shell>
73{% if aggregate_tests_script %}90{% if aggregate_tests_script %}
74 <hudson.tasks.Shell>91 <hudson.tasks.Shell>
75 <command>{{ aggregate_tests_script }}</command>92 <command>{{ aggregate_tests_script }}</command>
@@ -149,20 +166,6 @@
149 <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>166 <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
150 <properties>local_archive_host=$local_archive_host167 <properties>local_archive_host=$local_archive_host
151local_archive_login=$local_archive_login168local_archive_login=$local_archive_login
152temp_dir=${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}
153ppa_build_number=${BUILD_NUMBER}
154distributions=</properties>
155 </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
156 </configs>
157 <projects>{{ dput_job }}</projects>
158 <condition>SUCCESS</condition>
159 <triggerWithNoParameters>false</triggerWithNoParameters>
160 </hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
161 <hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
162 <configs>
163 <hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
164 <properties>local_archive_host=$local_archive_host
165local_archive_login=$local_archive_login
166temp_dir=${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}</properties>169temp_dir=${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}</properties>
167 </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>170 </hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
168 </configs>171 </configs>
169172
=== modified file 'ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl'
--- ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl 2013-03-14 14:59:37 +0000
+++ ci/jenkins-templates/mbs-pbuilder-config.xml.tmpl 2013-03-15 15:29:21 +0000
@@ -17,6 +17,11 @@
17 <description>Specifies that this is an autolanding/merge job. Used to properly set package version string.</description>17 <description>Specifies that this is an autolanding/merge job. Used to properly set package version string.</description>
18 <defaultValue></defaultValue>18 <defaultValue></defaultValue>
19 </hudson.model.StringParameterDefinition>19 </hudson.model.StringParameterDefinition>
20 <hudson.model.StringParameterDefinition>
21 <name>UPSTREAM_BUILD_NUMBER</name>
22 <description>Build number of the upstream job used in the tmp directory with the deb output</description>
23 <defaultValue></defaultValue>
24 </hudson.model.StringParameterDefinition>
20{% for parameter in parameter_list %}25{% for parameter in parameter_list %}
21 <hudson.model.StringParameterDefinition>26 <hudson.model.StringParameterDefinition>
22 <name>{{ parameter.name }}</name>27 <name>{{ parameter.name }}</name>
@@ -62,17 +67,17 @@
62 exit 067 exit 0
63fi68fi
6469
65tmp_dir=&quot;${local_archive_tmp}/${project}-al-${local_archive_pocket}-${BUILD_NUMBER}&quot;70tmp_dir=&quot;${local_archive_tmp}/${project_name}-al-${local_archive_pocket}-${UPSTREAM_BUILD_NUMBER}&quot;
6671
67# mkdir &quot;${tmp_dir}&quot; can fail due to multiple jobs attempting to create the directory72# mkdir &quot;${tmp_dir}&quot; can fail due to multiple jobs attempting to create the directory
68ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}&quot; || true73ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}&quot; || true
69ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}/${flavor}&quot;74ssh ${local_archive_login}@${local_archive_host} mkdir &quot;${tmp_dir}/${JOB_NAME}&quot;
7075
71# Switch to the output directory containing the build results76# Switch to the output directory containing the build results
72cd work/output77cd work/output
7378
74# Copy the build results to the temporary holding area, if all jobs succeed, these will be dput into the ppa archive.79# Copy the build results to the temporary holding area, if all jobs succeed, these will be dput into the ppa archive.
75scp * ${local_archive_login}@${local_archive_host}:${tmp_dir}/${flavor}</command>80scp * ${local_archive_login}@${local_archive_host}:${tmp_dir}/${JOB_NAME}</command>
76 </hudson.tasks.Shell>81 </hudson.tasks.Shell>
77 </builders>82 </builders>
78 <publishers>83 <publishers>
7984
=== modified file 'stacks/experimental/100scopes.cfg'
--- stacks/experimental/100scopes.cfg 2013-03-15 13:31:45 +0000
+++ stacks/experimental/100scopes.cfg 2013-03-15 15:29:21 +0000
@@ -6,7 +6,8 @@
6 dest: ubuntu-unity/experimental-certified6 dest: ubuntu-unity/experimental-certified
7 schedule: 0 7 * * 1-57 schedule: 0 7 * * 1-5
8 ci_default:8 ci_default:
9 hooks: D08add_ppa-ubuntu-unity-experimental-prevalidation #TODO: anythin mbs specific?9 hooks: D08add_ppa-ubuntu-unity-experimental-prevalidation D00mbs_archive H05set_package_version H40disable_gensymbols_check
10 hook_source: lp:~private-ps-quality-team/+junk/scopes-hooks
10 contact_email: "martin.mrazik@canonical.com"11 contact_email: "martin.mrazik@canonical.com"
11 autolanding_template: mbs-autolanding-config.xml.tmpl12 autolanding_template: mbs-autolanding-config.xml.tmpl
12 configurations:13 configurations:
@@ -16,9 +17,9 @@
16 local_archive_host: naartjie17 local_archive_host: naartjie
17 local_archive_login: ubuntu 18 local_archive_login: ubuntu
18 local_archive_tmp: /home/ubuntu/ppa_archive/100scopes-experimental19 local_archive_tmp: /home/ubuntu/ppa_archive/100scopes-experimental
20 local_archive: 100scopes-experimental
19 local_archive_source: http://naartjie/archive/100scopes-experimental21 local_archive_source: http://naartjie/archive/100scopes-experimental
20 local_archive_pocket: raring22 local_archive_pocket: raring
21 #TODO: what about dput_job
22 projects:23 projects:
23 libunity:24 libunity:
24 target_branch: lp:~unity-team/libunity/libunity-7.025 target_branch: lp:~unity-team/libunity/libunity-7.0
2526
=== modified file 'stacks/head/oif.cfg'
--- stacks/head/oif.cfg 2013-03-12 17:42:05 +0000
+++ stacks/head/oif.cfg 2013-03-15 15:29:21 +0000
@@ -28,6 +28,7 @@
28 local_archive_host: naartjie28 local_archive_host: naartjie
29 local_archive_login: ubuntu29 local_archive_login: ubuntu
30 local_archive_tmp: /home/ubuntu/ppa_archive/oif-staging30 local_archive_tmp: /home/ubuntu/ppa_archive/oif-staging
31 local_archive: oif-staging
31 local_archive_source: http://naartjie/archive/oif-staging32 local_archive_source: http://naartjie/archive/oif-staging
32 local_archive_pocket: raring33 local_archive_pocket: raring
33 dependencies:34 dependencies:
@@ -43,10 +44,6 @@
43 raring-armhf:44 raring-armhf:
44 template: pbuilder-config.xml.tmpl45 template: pbuilder-config.xml.tmpl
45 frame:46 frame:
46 dput_job: frame-dput-mbs-pt
47 geis:47 geis:
48 dput_job: geis-dput-mbs-pt
49 grail:48 grail:
50 dput_job: grail-dput-mbs-pt
51 libgrip:49 libgrip:
52 dput_job: libgrip-dput-mbs-pt
5350
=== modified file 'tests/test_cu2dUpdateCi.py'
--- tests/test_cu2dUpdateCi.py 2013-03-14 02:44:09 +0000
+++ tests/test_cu2dUpdateCi.py 2013-03-15 15:29:21 +0000
@@ -75,8 +75,11 @@
75 def test_project_config_implied_target_branch(self):75 def test_project_config_implied_target_branch(self):
76 config = {}76 config = {}
77 expected = {'target_branch': 'lp:project',77 expected = {'target_branch': 'lp:project',
78 'project_name': 'project',
78 'parameter_list': [JobParameter('target_branch',79 'parameter_list': [JobParameter('target_branch',
79 'lp:project')]}80 'lp:project'),
81 JobParameter('project_name',
82 'project')]}
80 actual = self.update_ci.process_project_config('project', config)83 actual = self.update_ci.process_project_config('project', config)
81 self.assertEqual(expected, actual)84 self.assertEqual(expected, actual)
8285
@@ -84,15 +87,21 @@
84 config = {'target_branch': 'lp:project/sub'}87 config = {'target_branch': 'lp:project/sub'}
85 expected = {'target_branch': 'lp:project/sub',88 expected = {'target_branch': 'lp:project/sub',
86 'parameter_list': [JobParameter('target_branch',89 'parameter_list': [JobParameter('target_branch',
87 'lp:project/sub')]}90 'lp:project/sub'),
91 JobParameter('project_name',
92 'project')],
93 'project_name': 'project'}
88 actual = self.update_ci.process_project_config('project', config)94 actual = self.update_ci.process_project_config('project', config)
89 self.assertEqual(expected, actual)95 self.assertEqual(expected, actual)
9096
91 def test_project_config_ctx_only(self):97 def test_project_config_ctx_only(self):
92 config = {'team': 'team_name'}98 config = {'team': 'team_name'}
93 expected = {'target_branch': 'lp:project',99 expected = {'target_branch': 'lp:project',
100 'project_name': 'project',
94 'parameter_list': [JobParameter('target_branch',101 'parameter_list': [JobParameter('target_branch',
95 'lp:project')],102 'lp:project'),
103 JobParameter('project_name',
104 'project')],
96 'team': 'team_name'}105 'team': 'team_name'}
97 actual = self.update_ci.process_project_config('project', config)106 actual = self.update_ci.process_project_config('project', config)
98 self.assertEqual(expected, actual)107 self.assertEqual(expected, actual)
@@ -101,10 +110,13 @@
101 config = {'some_parameter': 'some_value'}110 config = {'some_parameter': 'some_value'}
102 expected = {'target_branch': 'lp:project',111 expected = {'target_branch': 'lp:project',
103 'some_parameter': 'some_value',112 'some_parameter': 'some_value',
113 'project_name': 'project',
104 'parameter_list': [JobParameter('target_branch',114 'parameter_list': [JobParameter('target_branch',
105 'lp:project'),115 'lp:project'),
106 JobParameter('some_parameter',116 JobParameter('some_parameter',
107 'some_value')]}117 'some_value'),
118 JobParameter('project_name',
119 'project')]}
108 actual = self.update_ci.process_project_config('project', config)120 actual = self.update_ci.process_project_config('project', config)
109 self.assertEqual(expected, actual)121 self.assertEqual(expected, actual)
110122
@@ -125,12 +137,15 @@
125 expected = {'hook_location': '/tmp/$JOB_NAME-hooks',137 expected = {'hook_location': '/tmp/$JOB_NAME-hooks',
126 'target_branch': 'lp:project',138 'target_branch': 'lp:project',
127 'hook_source': 'lp:hooks',139 'hook_source': 'lp:hooks',
140 'project_name': 'project',
128 'acquire_hook_script': script.format(141 'acquire_hook_script': script.format(
129 DEFAULT_HOOK_LOCATION='/tmp/$JOB_NAME-hooks'),142 DEFAULT_HOOK_LOCATION='/tmp/$JOB_NAME-hooks'),
130 'parameter_list': [JobParameter('target_branch',143 'parameter_list': [JobParameter('target_branch',
131 'lp:project'),144 'lp:project'),
132 JobParameter('hook_source',145 JobParameter('hook_source',
133 'lp:hooks')]}146 'lp:hooks'),
147 JobParameter('project_name',
148 'project')]}
134 with patch('c2dconfigutils.cu2dUpdateCi.open',149 with patch('c2dconfigutils.cu2dUpdateCi.open',
135 create=True) as mock_open:150 create=True) as mock_open:
136 mock_open.return_value = MagicMock(spec=file)151 mock_open.return_value = MagicMock(spec=file)

Subscribers

People subscribed via source and target branches

to all changes: