Merge lp:~allanlesage/qa-coverage-dashboard/aggregate-by-product into lp:qa-coverage-dashboard

Proposed by Allan LeSage on 2014-05-22
Status: Needs review
Proposed branch: lp:~allanlesage/qa-coverage-dashboard/aggregate-by-product
Merge into: lp:qa-coverage-dashboard
Diff against target: 1397 lines (+701/-159)
24 files modified
gaps/__init__.py (+185/-0)
gaps/admin.py (+2/-2)
gaps/api.py (+13/-13)
gaps/management/commands/jenkins_pull_coverage.py (+4/-13)
gaps/migrations/0001_initial.py (+9/-9)
gaps/migrations/0002_auto__del_field_coverageproject_stack.py (+14/-14)
gaps/migrations/0003_auto__add_stackcoverageobservation.py (+13/-13)
gaps/models.py (+16/-16)
gaps/static/gaps/js/gaps_chart.js (+14/-6)
gaps/templates/gaps/integration_list.html (+3/-3)
gaps/templates/gaps/product_detail.html (+5/-5)
gaps/templates/gaps/product_list.html (+42/-13)
gaps/tests/__init__.py (+5/-0)
gaps/tests/test_add.py (+2/-2)
gaps/tests/test_cu2d.py (+133/-0)
gaps/tests/test_jenkins_pull_coverage.py (+5/-5)
gaps/urls.py (+2/-2)
gaps/urls_api.py (+6/-6)
gaps/util/__init__.py (+35/-0)
gaps/util/add.py (+14/-11)
gaps/util/cu2d/__init__.py (+128/-0)
gaps/util/jenkins_pull.py (+22/-9)
gaps/views.py (+17/-17)
qa_dashboard/settings.py (+12/-0)
To merge this branch: bzr merge lp:~allanlesage/qa-coverage-dashboard/aggregate-by-product
Reviewer Review Type Date Requested Status
Chris Gagnon (community) 2014-05-22 Needs Fixing on 2014-06-17
Brendan Donegan 2014-05-22 Pending
Review via email: mp+220585@code.launchpad.net

Description of the change

Make it possible to get the names of Jenkins jobs by LP-repository-name, so that we can aggregate by product instead of stack; meanwhile drag the cu2d utils from management/commands to util, improving tests. Also, start to explore the definition of product by making a primitive dict.

To post a comment you must log in.
Allan LeSage (allanlesage) wrote :

Work in progress for the moment! I'm going to attempt a pull, possibly during a test deployment. But please offer a review.

Brendan Donegan (brendan-donegan) wrote :

7 +from gaps.models import (CoverageData, CoverageProject, CoverageProduct, CoverageBuild)

Pre-existing problem I know but I think the:

from x.y import (
    A,
    B,
    C
)

pattern is better

84 +#from qa_dashboard.settings import CU2D_DEFAULT_CONFIG_DIR

commented code is evil

There might be more, but this is it for now

Allan LeSage (allanlesage) wrote :

Gentlemen this is a massive change however it's ready for review at your leisure.

Chris Gagnon (chris.gagnon) wrote :

can you move the inline js to static?

The todos should also be finished or have a bug number next to them

I'll probably have a few more things to fix before the review is complete, but we can start here. :D

review: Needs Fixing

Unmerged revisions

781. By Allan LeSage on 2014-05-27

Move product_list js back into html, broken in production.

780. By Allan LeSage on 2014-05-26

Merge chris.gagnon's axis indenting fix.

779. By Allan LeSage on 2014-05-23

Move templates around for product.

778. By Allan LeSage on 2014-05-23

Some pull fixes, fill in dict of products and projects with health check info.

777. By Allan LeSage on 2014-05-22

Merge trunk, resolving conflicts.

776. By Allan LeSage on 2014-05-22

Remove an obviated cu2d module.

775. By Allan LeSage on 2014-05-22

Skip a test until further notice.

774. By Allan LeSage on 2014-05-22

Add some logging for clarity.

773. By Allan LeSage on 2014-05-22

Fix url_artifact last_build_only, another fix for actual pull.

772. By Allan LeSage on 2014-05-22

Merge jenkinsapi_singleton, resolving conflicts.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'gaps/__init__.py'
2--- gaps/__init__.py 2014-02-20 01:46:07 +0000
3+++ gaps/__init__.py 2014-05-27 13:57:11 +0000
4@@ -0,0 +1,185 @@
5+
6+PRODUCTS_TO_PROJECTS = [
7+ {
8+ 'name': 'content-hub',
9+ 'lp_project_names': [
10+ 'content-hub',
11+ ]
12+ },
13+ {
14+ 'name': 'cordova',
15+ 'lp_project_names': [
16+ 'cordova-ubuntu',
17+ 'cordova-cli'
18+ ]
19+ },
20+ {
21+ 'name': 'system-apps',
22+ 'lp_project_names': [
23+ 'dialer-app',
24+ 'address-book5D-app',
25+ 'messaging-app',
26+ 'gallery-app',
27+ 'camera-app',
28+ 'notes-app',
29+ 'history-service',
30+ 'telepathy-ofono',
31+ 'ubuntu-keyboard',
32+ ]
33+ },
34+ {
35+ 'name': 'hardware',
36+ 'lp_project_names': [
37+ 'ofono',
38+ 'libhybris',
39+ 'platform-api',
40+ 'powerd',
41+ #'emulator'
42+ ]
43+ },
44+ {
45+ 'name': 'mir',
46+ 'lp_project_names': ['mir']
47+ },
48+ {
49+ 'name': 'multimedia',
50+ 'lp_project_names': [
51+ 'media-hub',
52+ ]
53+ },
54+ {
55+ 'name': 'online-accounts',
56+ 'lp_project_names': [
57+ 'account-plugins',
58+ 'gnome-control-center-signon',
59+ 'libaccounts-glib',
60+ 'libaccounts-qt',
61+ 'libsignon-glib',
62+ 'signon',
63+ 'signon-apparmor-extension',
64+ 'signon-keyring-extension',
65+ 'signon-plugin-oauth2',
66+ 'signon-ui',
67+ 'uoa-integration-tests',
68+ 'webaccounts-browser-extension',
69+ 'ubuntu-system-settings-online-accounts',
70+ ]
71+ },
72+ {
73+ 'name': 'oxide-web-browser',
74+ 'lp_project_names': [
75+ 'oxide',
76+ 'webbrowser-app'
77+ ]
78+ },
79+ {
80+ 'name': 'push',
81+ 'lp_project_names': [
82+ ]
83+ },
84+ {
85+ 'name': 'qt',
86+ 'lp_project_names': [
87+ 'qt3d-opensource-src',
88+ 'qtpim-opensource-src',
89+ 'qtsvg-opensource-src',
90+ 'qtbase-opensource-src',
91+ 'qtscript-opensource-src',
92+ 'qtsystems-opensource-src',
93+ 'qtsensors-opensource-src',
94+ 'qtfeedback-opensource-src',
95+ 'qtlocation-opensource-src',
96+ 'qtmultimedia-opensource-src',
97+ 'qtdeclarative-opensource-src',
98+ 'qtxmlpatterns-opensource-src',
99+ 'qtimageformats-opensource-src',
100+ 'qtquickcontrols-opensource-src',
101+ 'qtx11extras-opensource-src',
102+ 'qtconnectivity-opensource-src',
103+ 'qtserialport-opensource-src',
104+ ]
105+ },
106+ {
107+ 'name': 'qt-creator',
108+ 'lp_project_names': [
109+ 'qtcreator_plugin_ubuntu',
110+ 'qtcreator',
111+ ]
112+ },
113+ {
114+ 'name': 'scope-click',
115+ 'lp_project_names': [
116+ 'unity-scope-click'
117+ ]
118+ },
119+ {
120+ 'name': 'image-updates',
121+ 'lp_project_names': [
122+ 'ubuntu-system-image',
123+ 'ubuntu-download-manager'
124+ ]
125+ },
126+ {
127+ 'name': 'system-settings',
128+ 'lp_project_names': [
129+ 'ubuntu-system-settings',
130+ 'ubuntu-system-settings-online-accounts',
131+ 'gsettings-qt',
132+ ]
133+ },
134+ {
135+ 'name': 'ui-toolkit',
136+ 'lp_project_names': [
137+ 'ubuntu-ui-toolkit'
138+ ]
139+ },
140+ {
141+ 'name': 'unity8',
142+ 'lp_project_names': [
143+ 'unity8'
144+ ]
145+ },
146+ {
147+ 'name': 'unity-apis',
148+ 'lp_project_names': [
149+ 'connectivity-api',
150+ 'unity-action-api',
151+ 'url-dispatcher',
152+ 'unity-scope-click',
153+ 'hud',
154+ 'indicator-bluetooth',
155+ 'indicator-datetime',
156+ 'indicator-keyboard',
157+ 'indicator-location',
158+ 'indicator-messages',
159+ 'indicator-network',
160+ 'indicator-power',
161+ 'indicator-session',
162+ 'indicator-sound',
163+ 'indicator-sync',
164+ ]
165+ },
166+ {
167+ 'name': 'webapps',
168+ 'lp_project_names': [
169+ 'libunity-webapps',
170+ 'webapps-applications',
171+ 'unity-chromium-extension',
172+ 'unity-firefox-extension',
173+ 'webapps-greasemonkey',
174+ 'webapps-tests',
175+ 'webbrowser-app',
176+ 'webapps-core',
177+ 'unity-webapps-qml',
178+ 'unity-webapps-facebookmessenger',
179+ 'unity-webapps-gmail',
180+ 'unity-webapps-googlecalendar',
181+ 'unity-webapps-googledocs',
182+ 'unity-webapps-googleplus',
183+ 'unity-webapps-linkedin',
184+ 'unity-webapps-tumblr',
185+ 'unity-webapps-twitter',
186+ 'unity-webapps-youtube',
187+ ]
188+ },
189+]
190
191=== modified file 'gaps/admin.py'
192--- gaps/admin.py 2014-02-21 00:11:59 +0000
193+++ gaps/admin.py 2014-05-27 13:57:11 +0000
194@@ -1,8 +1,8 @@
195 from django.contrib import admin
196-from gaps.models import (CoverageData, CoverageProject, CoverageStack, CoverageBuild)
197+from gaps.models import (CoverageData, CoverageProject, CoverageProduct, CoverageBuild)
198
199 admin.site.register(CoverageData)
200 admin.site.register(CoverageProject)
201-admin.site.register(CoverageStack)
202+admin.site.register(CoverageProduct)
203 admin.site.register(CoverageBuild)
204
205
206=== modified file 'gaps/api.py'
207--- gaps/api.py 2014-03-09 01:05:53 +0000
208+++ gaps/api.py 2014-05-27 13:57:11 +0000
209@@ -23,9 +23,9 @@
210 from gaps.models import (CoverageBuild,
211 CoverageData,
212 CoverageProject,
213- CoverageStack,
214+ CoverageProduct,
215 JenkinsBuild,
216- StackCoverageObservation)
217+ ProductCoverageObservation)
218
219 logger = logging.getLogger('qa_dashboard')
220
221@@ -62,9 +62,9 @@
222 data = json.dumps(data)
223 return JSONResponse(data)
224
225-def stack(request, name):
226- stack = get_object_or_404(
227- CoverageStack,
228+def product(request, name):
229+ product = get_object_or_404(
230+ CoverageProduct,
231 name=name)
232 line_coverage_data = {
233 'key': 'Line coverage',
234@@ -74,8 +74,8 @@
235 'key': 'Branch coverage',
236 'color': '#772953'
237 }
238- observations = StackCoverageObservation.objects.filter(
239- stack=stack).order_by('-timestamp')
240+ observations = ProductCoverageObservation.objects.filter(
241+ product=product).order_by('-timestamp')
242 line_coverage_values = []
243 branch_coverage_values = []
244 # TODO: more Pythonic pls
245@@ -94,13 +94,13 @@
246 data = json.dumps(data)
247 return JSONResponse(data)
248
249-def stack_list(request):
250- stacks = CoverageStack.objects.all()
251+def product_list(request):
252+ products = CoverageProduct.objects.all()
253 data = []
254- for stack in stacks:
255- line_coverage_data = {'key': stack.name}
256- observations = StackCoverageObservation.objects.filter(
257- stack=stack).order_by('-timestamp')
258+ for product in products:
259+ line_coverage_data = {'key': product.name}
260+ observations = ProductCoverageObservation.objects.filter(
261+ product=product).order_by('-timestamp')
262 line_coverage_values = []
263 for observation in observations:
264 line_coverage_values.append(
265
266=== modified file 'gaps/management/commands/jenkins_pull_coverage.py'
267--- gaps/management/commands/jenkins_pull_coverage.py 2014-05-19 12:06:12 +0000
268+++ gaps/management/commands/jenkins_pull_coverage.py 2014-05-27 13:57:11 +0000
269@@ -5,6 +5,9 @@
270 from django.core.management.base import BaseCommand
271 from optparse import make_option
272
273+from gaps.util import get_coverage
274+#from qa_dashboard.settings import CU2D_DEFAULT_CONFIG_DIR
275+
276 logger = logging.getLogger('qa_dashboard')
277
278
279@@ -21,16 +24,4 @@
280 )
281
282 def handle(self, *args, **options):
283- BASEDIR = os.path.dirname(__file__)
284-
285- # add local c2configutils
286- if os.path.isdir(os.path.join(BASEDIR, '../c2dconfigutils')):
287- sys.path.insert(0, os.path.join(BASEDIR, '..'))
288-
289- from c2dconfigutils.cu2dOutputCi import OutputCi
290-
291- get_coverage = OutputCi()
292- #if options['just_last_build']:
293- # get_coverage(BASEDIR, True)
294- #else:
295- get_coverage(BASEDIR, options['just_last_build'])
296+ get_coverage(just_last_build=options['just_last_build'])
297
298=== modified file 'gaps/migrations/0001_initial.py'
299--- gaps/migrations/0001_initial.py 2014-02-25 00:15:12 +0000
300+++ gaps/migrations/0001_initial.py 2014-05-27 13:57:11 +0000
301@@ -12,18 +12,18 @@
302 )
303
304 def forwards(self, orm):
305- # Adding model 'CoverageStack'
306- db.create_table(u'gaps_coveragestack', (
307+ # Adding model 'CoverageProduct'
308+ db.create_table(u'gaps_coverageproduct', (
309 (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
310 ('name', self.gf('django.db.models.fields.CharField')(max_length=64)),
311 ))
312- db.send_create_signal(u'gaps', ['CoverageStack'])
313+ db.send_create_signal(u'gaps', ['CoverageProduct'])
314
315 # Adding model 'CoverageProject'
316 db.create_table(u'gaps_coverageproject', (
317 (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
318 ('name', self.gf('django.db.models.fields.CharField')(unique=True, max_length=256)),
319- ('stack', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['gaps.CoverageStack'])),
320+ ('product', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['gaps.CoverageProduct'])),
321 ))
322 db.send_create_signal(u'gaps', ['CoverageProject'])
323
324@@ -52,8 +52,8 @@
325
326
327 def backwards(self, orm):
328- # Deleting model 'CoverageStack'
329- db.delete_table(u'gaps_coveragestack')
330+ # Deleting model 'CoverageProduct'
331+ db.delete_table(u'gaps_coverageproduct')
332
333 # Deleting model 'CoverageProject'
334 db.delete_table(u'gaps_coverageproject')
335@@ -127,10 +127,10 @@
336 'Meta': {'object_name': 'CoverageProject'},
337 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
338 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'}),
339- 'stack': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['gaps.CoverageStack']"})
340+ 'product': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['gaps.CoverageProduct']"})
341 },
342- u'gaps.coveragestack': {
343- 'Meta': {'object_name': 'CoverageStack'},
344+ u'gaps.coverageproduct': {
345+ 'Meta': {'object_name': 'CoverageProduct'},
346 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
347 'name': ('django.db.models.fields.CharField', [], {'max_length': '64'})
348 }
349
350=== modified file 'gaps/migrations/0002_auto__del_field_coverageproject_stack.py'
351--- gaps/migrations/0002_auto__del_field_coverageproject_stack.py 2014-03-06 17:01:11 +0000
352+++ gaps/migrations/0002_auto__del_field_coverageproject_stack.py 2014-05-27 13:57:11 +0000
353@@ -8,24 +8,24 @@
354 class Migration(SchemaMigration):
355
356 def forwards(self, orm):
357- # Deleting field 'CoverageProject.stack'
358- db.delete_column(u'gaps_coverageproject', 'stack_id')
359+ # Deleting field 'CoverageProject.product'
360+ db.delete_column(u'gaps_coverageproject', 'product_id')
361
362- # Adding M2M table for field stack on 'CoverageProject'
363- db.create_table(u'gaps_coverageproject_stack', (
364+ # Adding M2M table for field product on 'CoverageProject'
365+ db.create_table(u'gaps_coverageproject_product', (
366 ('id', models.AutoField(verbose_name='ID', primary_key=True, auto_created=True)),
367 ('coverageproject', models.ForeignKey(orm[u'gaps.coverageproject'], null=False)),
368- ('coveragestack', models.ForeignKey(orm[u'gaps.coveragestack'], null=False))
369+ ('coverageproduct', models.ForeignKey(orm[u'gaps.coverageproduct'], null=False))
370 ))
371- db.create_unique(u'gaps_coverageproject_stack', ['coverageproject_id', 'coveragestack_id'])
372+ db.create_unique(u'gaps_coverageproject_product', ['coverageproject_id', 'coverageproduct_id'])
373
374
375 def backwards(self, orm):
376
377- # User chose to not deal with backwards NULL issues for 'CoverageProject.stack'
378- raise RuntimeError("Cannot reverse this migration. 'CoverageProject.stack' and its values cannot be restored.")
379- # Removing M2M table for field stack on 'CoverageProject'
380- db.delete_table('gaps_coverageproject_stack')
381+ # User chose to not deal with backwards NULL issues for 'CoverageProject.product'
382+ raise RuntimeError("Cannot reverse this migration. 'CoverageProject.product' and its values cannot be restored.")
383+ # Removing M2M table for field product on 'CoverageProject'
384+ db.delete_table('gaps_coverageproject_product')
385
386
387 models = {
388@@ -90,13 +90,13 @@
389 'Meta': {'object_name': 'CoverageProject'},
390 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
391 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'}),
392- 'stack': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['gaps.CoverageStack']", 'symmetrical': 'False'})
393+ 'product': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['gaps.CoverageProduct']", 'symmetrical': 'False'})
394 },
395- u'gaps.coveragestack': {
396- 'Meta': {'object_name': 'CoverageStack'},
397+ u'gaps.coverageproduct': {
398+ 'Meta': {'object_name': 'CoverageProduct'},
399 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
400 'name': ('django.db.models.fields.CharField', [], {'max_length': '64'})
401 }
402 }
403
404- complete_apps = ['gaps']
405\ No newline at end of file
406+ complete_apps = ['gaps']
407
408=== modified file 'gaps/migrations/0003_auto__add_stackcoverageobservation.py'
409--- gaps/migrations/0003_auto__add_stackcoverageobservation.py 2014-03-06 22:13:30 +0000
410+++ gaps/migrations/0003_auto__add_stackcoverageobservation.py 2014-05-27 13:57:11 +0000
411@@ -8,20 +8,20 @@
412 class Migration(SchemaMigration):
413
414 def forwards(self, orm):
415- # Adding model 'StackCoverageObservation'
416- db.create_table(u'gaps_stackcoverageobservation', (
417+ # Adding model 'ProductCoverageObservation'
418+ db.create_table(u'gaps_productcoverageobservation', (
419 (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
420 ('timestamp', self.gf('django.db.models.fields.DateTimeField')()),
421- ('stack', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['gaps.CoverageStack'])),
422+ ('product', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['gaps.CoverageProduct'])),
423 ('line_coverage', self.gf('django.db.models.fields.DecimalField')(null=True, max_digits=6, decimal_places=3, blank=True)),
424 ('branch_coverage', self.gf('django.db.models.fields.DecimalField')(null=True, max_digits=6, decimal_places=3, blank=True)),
425 ))
426- db.send_create_signal(u'gaps', ['StackCoverageObservation'])
427+ db.send_create_signal(u'gaps', ['ProductCoverageObservation'])
428
429
430 def backwards(self, orm):
431- # Deleting model 'StackCoverageObservation'
432- db.delete_table(u'gaps_stackcoverageobservation')
433+ # Deleting model 'ProductCoverageObservation'
434+ db.delete_table(u'gaps_productcoverageobservation')
435
436
437 models = {
438@@ -86,21 +86,21 @@
439 'Meta': {'object_name': 'CoverageProject'},
440 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
441 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'}),
442- 'stack': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['gaps.CoverageStack']", 'symmetrical': 'False'})
443+ 'product': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['gaps.CoverageProduct']", 'symmetrical': 'False'})
444 },
445- u'gaps.coveragestack': {
446- 'Meta': {'object_name': 'CoverageStack'},
447+ u'gaps.coverageproduct': {
448+ 'Meta': {'object_name': 'CoverageProduct'},
449 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
450 'name': ('django.db.models.fields.CharField', [], {'max_length': '64'})
451 },
452- u'gaps.stackcoverageobservation': {
453- 'Meta': {'object_name': 'StackCoverageObservation'},
454+ u'gaps.productcoverageobservation': {
455+ 'Meta': {'object_name': 'ProductCoverageObservation'},
456 'branch_coverage': ('django.db.models.fields.DecimalField', [], {'null': 'True', 'max_digits': '6', 'decimal_places': '3', 'blank': 'True'}),
457 u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
458 'line_coverage': ('django.db.models.fields.DecimalField', [], {'null': 'True', 'max_digits': '6', 'decimal_places': '3', 'blank': 'True'}),
459- 'stack': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['gaps.CoverageStack']"}),
460+ 'product': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['gaps.CoverageProduct']"}),
461 'timestamp': ('django.db.models.fields.DateTimeField', [], {})
462 }
463 }
464
465- complete_apps = ['gaps']
466\ No newline at end of file
467+ complete_apps = ['gaps']
468
469=== modified file 'gaps/models.py'
470--- gaps/models.py 2014-03-09 23:29:14 +0000
471+++ gaps/models.py 2014-05-27 13:57:11 +0000
472@@ -7,36 +7,36 @@
473 logger = logging.getLogger("qa_dashboard")
474
475
476-class CoverageStack(models.Model):
477- """Canonical upstream to distro stack.
478+class CoverageProduct(models.Model):
479+ """Ubuntu QA weekly health check 'product'.
480
481- An agglomeration of projects often released together.
482+ Examples include 'mir', 'content-hub', etc.
483 """
484 name = models.CharField(max_length=64)
485
486 @property
487 def last_coverage_observation(self):
488 try:
489- return StackCoverageObservation.objects.filter(
490- stack=self).order_by('-timestamp')[0]
491+ return ProductCoverageObservation.objects.filter(
492+ product=self).order_by('-timestamp')[0]
493 except IndexError:
494 return None
495
496 def refresh_coverage_observations(self):
497- StackCoverageObservation.objects.filter(stack=self).delete()
498- # for each coveragedata relevant to this stack, make a new observation (and compute coverages)
499- coverage_projects = CoverageProject.objects.filter(stack=self)
500+ ProductCoverageObservation.objects.filter(product=self).delete()
501+ # for each coveragedata relevant to this product, make a new observation (and compute coverages)
502+ coverage_projects = CoverageProject.objects.filter(product=self)
503 coverage_datas = CoverageData.objects.filter(
504 coverage_build__project__in=coverage_projects)
505 for coverage_data in coverage_datas:
506- observation = StackCoverageObservation(
507- stack=self,
508+ observation = ProductCoverageObservation(
509+ product=self,
510 timestamp=coverage_data.coverage_build.ran_at)
511 observation.compute_coverage()
512
513 @property
514 def percent_reporting(self):
515- coverage_projects = CoverageProject.objects.filter(stack=self)
516+ coverage_projects = CoverageProject.objects.filter(product=self)
517 coverage_data_exists_count = 0
518 for coverage_project in coverage_projects:
519 if coverage_project.last_coverage_data is not None:
520@@ -58,7 +58,7 @@
521 class CoverageProject(models.Model):
522 """Our first-class object which links Jenkins to Launchpad."""
523 name = models.CharField(max_length=256, unique=True)
524- stack = models.ManyToManyField(CoverageStack)
525+ product = models.ManyToManyField(CoverageProduct)
526
527 @property
528 def last_build(self):
529@@ -118,13 +118,13 @@
530 # -- build_description - charfield
531 # -- project
532 # ---- name
533- # ---- stack
534+ # ---- product
535 # ------ name
536
537
538-class StackCoverageObservation(models.Model):
539+class ProductCoverageObservation(models.Model):
540 timestamp = models.DateTimeField()
541- stack = models.ForeignKey(CoverageStack)
542+ product = models.ForeignKey(CoverageProduct)
543 line_coverage = models.DecimalField(max_digits=6,
544 decimal_places=3,
545 null=True,
546@@ -137,7 +137,7 @@
547 # TODO compute this on create, enforce not-null for coverage fields
548 def compute_coverage(self):
549 project_list = CoverageProject.objects.filter(
550- stack=self.stack)
551+ product=self.product)
552 hit_sum = line_sum = taken_sum = total_sum = 0
553 for project in project_list:
554 try:
555
556=== modified file 'gaps/static/gaps/js/gaps_chart.js'
557--- gaps/static/gaps/js/gaps_chart.js 2014-04-04 21:10:20 +0000
558+++ gaps/static/gaps/js/gaps_chart.js 2014-05-27 13:57:11 +0000
559@@ -2,16 +2,24 @@
560 d3.json(data_url, function(data) {
561 nv.addGraph(function() {
562 var chart = nv.models.lineChart()
563- .width(1100).height(400);
564+ .width(900).height(400);
565+
566 chart.forceY([0, 1]);
567+
568 chart.xAxis
569- .axisLabel('Date')
570- .tickFormat(function(d){return d3.time.format('%Y%m%d')(new Date(d * 1000));})
571+ .axisLabel('Date')
572+ .tickFormat(function(d){return d3.time.format('%Y%m%d')(new Date(d * 1000));})
573+ .staggerLabels(true);
574
575 chart.yAxis
576- .axisLabel('%')
577- .tickFormat(d3.format('%'))
578- d3.select('#line-chart svg').datum(data).transition().duration(500).call(chart);
579+ .axisLabel('%')
580+ .tickFormat(d3.format('%'));
581+
582+ d3.select('#line-chart svg')
583+ .datum(data)
584+ .transition()
585+ .duration(500)
586+ .call(chart);
587
588 nv.utils.windowResize(chart.update);
589
590
591=== modified file 'gaps/templates/gaps/integration_list.html'
592--- gaps/templates/gaps/integration_list.html 2014-04-04 21:10:20 +0000
593+++ gaps/templates/gaps/integration_list.html 2014-05-27 13:57:11 +0000
594@@ -21,11 +21,11 @@
595 </div>
596 <div class='grid_2'>
597 <h3 class='nav-title'>
598- STACKS
599+ PRODUCTS
600 </h3>
601 <ul class='left_nav'>
602 <li>
603- <a href='{% url stacks %}'>All</a>
604+ <a href='{% url products %}'>All</a>
605 </li>
606 </ul>
607 </div>
608@@ -46,7 +46,7 @@
609 {% for test in test_list %}
610 <tr>
611 <td>
612- <a href='{% url stacks %}/{{ test.stack_name }}'>{{ test.stack_name }}</a>
613+ <a href='{% url products %}/{{ test.product_name }}'>{{ test.product_name }}</a>
614 </td>
615 <td>
616 {{ test.release }}
617
618=== renamed file 'gaps/templates/gaps/stack_detail.html' => 'gaps/templates/gaps/product_detail.html'
619--- gaps/templates/gaps/stack_detail.html 2014-04-07 22:20:18 +0000
620+++ gaps/templates/gaps/product_detail.html 2014-05-27 13:57:11 +0000
621@@ -22,7 +22,7 @@
622 //]]>
623 </script>
624 <script type="text/javascript">
625- d3.json('{% url 'gaps_api_stack' stack.name %}', function(data) {
626+ d3.json('{% url 'gaps_api_product' product.name %}', function(data) {
627 nv.addGraph(function() {
628 var chart = nv.models.lineChart()
629 .width(900).height(400);
630@@ -41,17 +41,17 @@
631 </script>
632 <div class='grid_15'>
633 <h2>
634- Stack: {{ stack.name }}
635+ Product: {{ product.name }}
636 </h2>
637 </div>
638 <div class='grid_2'>
639 <h3 class='nav-title'>
640- STACKS
641+ PRODUCTS
642 </h3>
643 <ul class='left_nav'>
644- {% for stack in stack_list %}
645+ {% for product in product_list %}
646 <li>
647- <a href='{% url 'stacks' %}{{ stack }}'>{{ stack }}</a>
648+ <a href='{% url 'products' %}{{ product }}'>{{ product }}</a>
649 </li>
650 {% endfor %}
651 </ul>
652
653=== renamed file 'gaps/templates/gaps/stack_list.html' => 'gaps/templates/gaps/product_list.html'
654--- gaps/templates/gaps/stack_list.html 2014-04-07 22:20:18 +0000
655+++ gaps/templates/gaps/product_list.html 2014-05-27 13:57:11 +0000
656@@ -22,15 +22,44 @@
657 //]]>
658 </script>
659 <script type="text/javascript">
660- var chart_url = '{% url "gaps_api_stack_list" %}'
661+ function create_gaps_line_chart(data_url) {
662+ d3.json(data_url, function(data) {
663+ nv.addGraph(function() {
664+ var chart = nv.models.lineChart()
665+ .width(900).height(400);
666+
667+ chart.forceY([0, 1]);
668+
669+ chart.xAxis
670+ .axisLabel('Date')
671+ .tickFormat(function(d){return d3.time.format('%Y%m%d')(new Date(d * 1000));})
672+ .staggerLabels(true);
673+
674+ chart.yAxis
675+ .axisLabel('%')
676+ .tickFormat(d3.format('%'));
677+
678+ d3.select('#line-chart svg')
679+ .datum(data)
680+ .transition()
681+ .duration(500)
682+ .call(chart);
683+
684+ nv.utils.windowResize(chart.update);
685+
686+ return chart;
687+ }); //d3.json
688+ }); //addGraph
689+ } //create_gaps_chart
690+ var chart_url = '{% url "gaps_api_product_list" %}'
691 create_gaps_line_chart(chart_url)
692 </script>
693 <div class='grid_15'>
694 <h2>
695 {% if show_filter %}
696- Stack : {{ show_filter }}
697+ Product : {{ show_filter }}
698 {% else %}
699- All Stacks
700+ All Products
701 {% endif %}
702 </h2>
703 </div>
704@@ -40,11 +69,11 @@
705 </h3>
706 <ul class='left_nav'>
707 <li>
708- <a href='{% url 'stacks' %}'>All</a>
709+ <a href='{% url 'products' %}'>All</a>
710 </li>
711 {% for release in release_list %}
712 <li>
713- <a href='{% url 'stacks' %}?show={{ release }} '>{{ release }}</a>
714+ <a href='{% url 'products' %}?show={{ release }} '>{{ release }}</a>
715 </li>
716 {% endfor %}
717 </ul>
718@@ -63,26 +92,26 @@
719 </tr>
720 </thead>
721 <tbody>
722- {% for stack in stack_list %}
723+ {% for product in product_list %}
724 <tr>
725 <td>
726- <a href='{% url 'stacks' %}{{ stack.name }}/'>{{ stack.name }}</a>
727+ <a href='{% url 'products' %}{{ product.name }}/'>{{ product.name }}</a>
728 </td>
729- <td class='{{ stack.percent_reporting|percent_reporting_color }}'>
730- {{ stack.percent_reporting|percentage }}
731+ <td class='{{ product.percent_reporting|percent_reporting_color }}'>
732+ {{ product.percent_reporting|percentage }}
733 </td>
734 <td class='num'>
735- {% if stack.last_coverage_observation.line_coverage == None %}
736+ {% if product.last_coverage_observation.line_coverage == None %}
737 N/A
738 {% else %}
739- {{ stack.last_coverage_observation.line_coverage|percentage }}%
740+ {{ product.last_coverage_observation.line_coverage|percentage }}%
741 {% endif %}
742 </td>
743 <td class='num'>
744- {% if stack.last_coverage_observation.branch_coverage == None %}
745+ {% if product.last_coverage_observation.branch_coverage == None %}
746 N/A
747 {% else %}
748- {{ stack.last_coverage_observation.branch_coverage|percentage }}%
749+ {{ product.last_coverage_observation.branch_coverage|percentage }}%
750 {% endif %}
751 </td>
752 </tr>
753
754=== modified file 'gaps/tests/__init__.py'
755--- gaps/tests/__init__.py 2014-05-22 14:13:44 +0000
756+++ gaps/tests/__init__.py 2014-05-27 13:57:11 +0000
757@@ -7,3 +7,8 @@
758 #from gaps.tests.test_add import ( # noqa
759 # GetLastBuildTestCase,
760 #)
761+from gaps.tests.test_cu2d import (
762+ TestRefreshCupstream2distroConfig,
763+ TestGenerateJobConfigsFromStackConfig,
764+ TestGenerateJenkinsJobNames,
765+)
766
767=== modified file 'gaps/tests/test_add.py'
768--- gaps/tests/test_add.py 2014-04-21 03:56:48 +0000
769+++ gaps/tests/test_add.py 2014-05-27 13:57:11 +0000
770@@ -1,8 +1,8 @@
771
772+from mock import patch, MagicMock
773+
774 from django.test import TestCase
775
776-from mock import patch, MagicMock
777-
778 from gaps.util.add import (
779 get_last_build_number_for_jenkins_job,
780 CoverageData,
781
782=== added file 'gaps/tests/test_cu2d.py'
783--- gaps/tests/test_cu2d.py 1970-01-01 00:00:00 +0000
784+++ gaps/tests/test_cu2d.py 2014-05-27 13:57:11 +0000
785@@ -0,0 +1,133 @@
786+
787+from django.test import TestCase
788+from mock import patch, MagicMock
789+
790+from gaps.util.cu2d import (
791+ generate_jenkins_job_names,
792+ refresh_cupstream2distro_config_repo,
793+ generate_job_configs_from_stack_config,
794+ UpdateCi,
795+)
796+
797+
798+class TestRefreshCupstream2distroConfig(TestCase):
799+
800+ @patch('os.path.exists')
801+ @patch('os.chdir')
802+ @patch('subprocess.call')
803+ def test_repo_already_exists(
804+ self,
805+ subprocess_call_mock,
806+ os_chdir_mock,
807+ os_path_exists_mock):
808+ os_path_exists_mock.return_value = True
809+ refresh_cupstream2distro_config_repo('/foo/bar')
810+ subprocess_call_mock.assert_called_with(['bzr', 'pull'])
811+
812+ @patch('os.path.split')
813+ @patch('os.chdir')
814+ @patch('subprocess.call')
815+ def test_repo_doesnt_exist(
816+ self,
817+ subprocess_call_mock,
818+ os_chdir_mock,
819+ os_path_split_mock):
820+ os_path_split_mock.return_value = ('fake_dir', 'fake_repo_name')
821+ refresh_cupstream2distro_config_repo('/fake/dir')
822+ subprocess_call_mock.assert_called_with(
823+ ['bzr',
824+ 'branch',
825+ 'lp:cupstream2distro-config',
826+ 'fake_repo_name']
827+ )
828+
829+
830+class TestGetStackListFromFilesystem(TestCase):
831+
832+ # TODO: more tests pls
833+ pass
834+
835+
836+class TestGenerateJobConfigsFromStackConfig(TestCase):
837+
838+ @patch('gaps.util.cu2d.load_default_cfg')
839+ @patch('gaps.util.cu2d.load_stack_cfg')
840+ def test_vanilla(self,
841+ load_stack_cfg_mock,
842+ load_default_cfg_mock):
843+ with patch.object(UpdateCi,
844+ '__init__',
845+ MagicMock(return_value=None)):
846+ stack_mock = MagicMock(config_filepath='fake_config_filepath')
847+ generate_job_configs_from_stack_config(
848+ '/fake/default/config/path',
849+ stack_mock
850+ )
851+ # TODO: get hold of the result and assert
852+
853+ @patch('gaps.util.cu2d.load_default_cfg')
854+ @patch('gaps.util.cu2d.load_stack_cfg')
855+ @patch('logging.error')
856+ def test_update_ci_process_stack_throws(
857+ self,
858+ logger_error_mock,
859+ load_stack_cfg_mock,
860+ load_default_cfg_mock):
861+ with patch.object(UpdateCi,
862+ '__init__',
863+ MagicMock(
864+ return_value=None,
865+ side_effect=Exception
866+ )):
867+ with self.assertRaises(Exception):
868+ stack_mock = MagicMock(config_filepath='fake_config_filepath')
869+ generate_job_configs_from_stack_config(
870+ '/fake/default/config/path',
871+ stack_mock
872+ )
873+
874+ @patch('gaps.util.cu2d.load_default_cfg')
875+ @patch('gaps.util.cu2d.load_stack_cfg')
876+ @patch('logging.error')
877+ def test_config_failed_to_load(
878+ self,
879+ logging_error_mock,
880+ load_stack_cfg_mock,
881+ load_default_cfg_mock):
882+ load_stack_cfg_mock.return_value = None
883+ stack_mock = MagicMock(config_filepath='fake_config_filepath')
884+ generate_job_configs_from_stack_config(
885+ '/fake/default/config/path',
886+ stack_mock
887+ )
888+ logging_error_mock.assert_called()
889+
890+
891+class TestGenerateJenkinsJobNames(TestCase):
892+
893+ @patch('gaps.util.cu2d.get_stack_list_from_filesystem')
894+ @patch('gaps.util.cu2d.generate_job_configs_from_stack_config')
895+ def test_vanilla(
896+ self,
897+ generate_job_configs_from_stack_config,
898+ get_stack_list_from_filesystem_mock):
899+ get_stack_list_from_filesystem_mock.return_value = [
900+ {'config_filepath': 'fake-filepath',
901+ 'pocket': 'fake-pocket',
902+ 'name': 'fake-name'}
903+ ]
904+ generate_job_configs_from_stack_config.return_value = [
905+ {'pocket': 'head',
906+ 'config_filepath': '/home/alesage/workspace/'
907+ 'qa-coverage-dashboard/aggregate-deux/qa_dashboard/..'
908+ '/gaps/util/cupstream2distro_config/stacks/head/mir.cfg',
909+ 'name': 'fake-jenkins-job-name'}
910+ ]
911+ result = generate_jenkins_job_names(
912+ 'fake-default-config-path',
913+ 'fake-stack-dir',
914+ 'fake-jenkins-job-name')
915+ self.assertEqual(
916+ ['fake-jenkins-job-name'],
917+ result
918+ )
919
920=== modified file 'gaps/tests/test_jenkins_pull_coverage.py'
921--- gaps/tests/test_jenkins_pull_coverage.py 2014-05-19 13:23:23 +0000
922+++ gaps/tests/test_jenkins_pull_coverage.py 2014-05-27 13:57:11 +0000
923@@ -4,7 +4,7 @@
924 CoverageBuild,
925 CoverageData,
926 CoverageProject,
927- CoverageStack,
928+ CoverageProduct,
929 )
930 from gaps.util import add
931
932@@ -98,10 +98,10 @@
933
934 self.assertEquals(project_result.name, 'libqtdbustest')
935
936- stack_result = CoverageStack.objects.filter(
937+ product_result = CoverageProduct.objects.filter(
938 name='qa').latest('name')
939
940- self.assertEquals(stack_result.name, 'qa')
941+ self.assertEquals(product_result.name, 'qa')
942
943 @mock.patch('gaps.util.add.jenkins_get')
944 @mock.patch('gaps.util.add.jenkins_pull')
945@@ -151,7 +151,7 @@
946
947 self.assertEquals(project_result.name, 'libqtdbustest')
948
949- stack_result = CoverageStack.objects.filter(
950+ product_result = CoverageProduct.objects.filter(
951 name='qa').latest('name')
952
953- self.assertEquals(stack_result.name, 'qa')
954+ self.assertEquals(product_result.name, 'qa')
955
956=== modified file 'gaps/urls.py'
957--- gaps/urls.py 2014-04-04 21:10:20 +0000
958+++ gaps/urls.py 2014-05-27 13:57:11 +0000
959@@ -22,8 +22,8 @@
960 'gaps.views',
961 url(r'^$', RedirectView.as_view(url='project', permanent=False),
962 name='gaps_overview'),
963- url(r'^stack/$', 'stack_list', name='stacks'),
964- url(r'^stack/(?P<name>[^/]+)/$', 'stack'),
965+ url(r'^product/$', 'product_list', name='products'),
966+ url(r'^product/(?P<name>[^/]+)/$', 'product'),
967 url(r'^project/$', 'project_list', name='projects'),
968 url(r'^project/(?P<name>[^/]+)/$', 'project'),
969 )
970
971=== modified file 'gaps/urls_api.py'
972--- gaps/urls_api.py 2014-03-09 01:05:53 +0000
973+++ gaps/urls_api.py 2014-05-27 13:57:11 +0000
974@@ -20,10 +20,10 @@
975 url(r'^project/(?P<name>[^/]+)/$',
976 'project',
977 name='gaps_api_project'),
978- url(r'^stack/$',
979- 'stack_list',
980- name='gaps_api_stack_list'),
981- url(r'^stack/(?P<name>[^/]+)/$',
982- 'stack',
983- name='gaps_api_stack'),
984+ url(r'^product/$',
985+ 'product_list',
986+ name='gaps_api_product_list'),
987+ url(r'^product/(?P<name>[^/]+)/$',
988+ 'product',
989+ name='gaps_api_product'),
990 )
991
992=== modified file 'gaps/util/__init__.py'
993--- gaps/util/__init__.py 2014-02-20 01:46:07 +0000
994+++ gaps/util/__init__.py 2014-05-27 13:57:11 +0000
995@@ -0,0 +1,35 @@
996+
997+import logging
998+
999+from gaps import PRODUCTS_TO_PROJECTS as products_to_projects
1000+from gaps.util.add import records
1001+from gaps.util.cu2d import generate_jenkins_job_names
1002+from qa_dashboard.settings import (
1003+ JENKINS_URL,
1004+ CU2D_DEFAULT_CONFIG_DIR,
1005+ CU2D_STACKS_DIR,
1006+)
1007+
1008+logger = logging.getLogger('qa_dashboard')
1009+
1010+
1011+def get_coverage(just_last_build=False):
1012+ # read the config and get a list of products
1013+ # TODO for the moment we'll just give a fake list
1014+ logger.info(len(products_to_projects))
1015+ for product in products_to_projects:
1016+ for lp_project_name in product['lp_project_names']:
1017+ jenkins_job_names = generate_jenkins_job_names(
1018+ CU2D_DEFAULT_CONFIG_DIR,
1019+ CU2D_STACKS_DIR,
1020+ lp_project_name
1021+ )
1022+ logger.info("FINISHED GENERATING NAMES FOR {}.".format(lp_project_name))
1023+ for jenkins_job_name in jenkins_job_names:
1024+ records(
1025+ product['name'],
1026+ lp_project_name,
1027+ jenkins_job_name,
1028+ 'coverage.xml',
1029+ just_last_build=just_last_build,
1030+ )
1031
1032=== modified file 'gaps/util/add.py'
1033--- gaps/util/add.py 2014-05-22 14:13:44 +0000
1034+++ gaps/util/add.py 2014-05-27 13:57:11 +0000
1035@@ -6,7 +6,7 @@
1036 CoverageBuild,
1037 CoverageData,
1038 CoverageProject,
1039- CoverageStack,
1040+ CoverageProduct,
1041 )
1042 from gaps.util import extractor, jenkins_pull
1043
1044@@ -14,7 +14,9 @@
1045
1046
1047 def get_last_build_number_for_jenkins_job(jenkins_job_name):
1048- logger.debug("getting the number of the last_build")
1049+ logger.debug(
1050+ "Getting the number of the last_build for {}.".format(
1051+ jenkins_job_name))
1052 try:
1053 last_coveragedata = CoverageData.objects.filter(
1054 coverage_build__job__name__exact=jenkins_job_name
1055@@ -29,10 +31,11 @@
1056 return [0]
1057
1058
1059-def records(stack_name, project_name, jenkins_job_name, file_name, just_last_build=False):
1060+def records(product_name, project_name, jenkins_job_name,
1061+ file_name, just_last_build=False):
1062 """Adds coverage data to the database
1063
1064- :param stack_name: stack to get artifacts from
1065+ :param product_name: product to get artifacts from
1066 :param project name: name of project to get artifact from
1067 :param jenkins_job_name: job to get artifacts from
1068 :param file_name: file name of artifact
1069@@ -51,7 +54,7 @@
1070 if just_last_build:
1071 pulled_list = [pulled_list[-1]]
1072 for build_info in pulled_list:
1073- logger.debug('build info{}'.format(build_info))
1074+ logger.debug('build info {}'.format(build_info))
1075 if build_info['url'].endswith('xml'):
1076 url_file = build_info['url']
1077 url = '/'.join(build_info['url'].split('/')[:5])
1078@@ -60,15 +63,15 @@
1079 url = build_info['url']
1080 build_id = build_info['build_id']
1081 build_ran_at = build_info['ran_at'].replace(tzinfo=None)
1082- stack, stack_created = CoverageStack.objects.get_or_create(
1083- name=stack_name)
1084+ product, product_created = CoverageProduct.objects.get_or_create(
1085+ name=product_name)
1086 project, project_created = CoverageProject.objects.get_or_create(
1087 name=project_name)
1088- # make sure the project is linked to this stack
1089+ # make sure the project is linked to this product
1090 try:
1091- CoverageProject.objects.get(name=project_name, stack=stack)
1092+ CoverageProject.objects.get(name=project_name, product=product)
1093 except CoverageProject.DoesNotExist:
1094- project.stack.add(stack)
1095+ project.product.add(product)
1096 (jenkins_job,
1097 jenkins_job_created) = JenkinsJob.objects.get_or_create(
1098 name=jenkins_job_name, url=url)
1099@@ -95,4 +98,4 @@
1100 total_count=coverage_info.total,
1101 coverage_build=build,
1102 )
1103- stack.refresh_coverage_observations()
1104+ product.refresh_coverage_observations()
1105
1106=== added directory 'gaps/util/cu2d'
1107=== added file 'gaps/util/cu2d/__init__.py'
1108--- gaps/util/cu2d/__init__.py 1970-01-01 00:00:00 +0000
1109+++ gaps/util/cu2d/__init__.py 2014-05-27 13:57:11 +0000
1110@@ -0,0 +1,128 @@
1111+
1112+import logging
1113+import os
1114+import subprocess
1115+
1116+from gaps.util.cu2d.cupstream2distro_config.c2dconfigutils.cu2dUpdateCi import UpdateCi # noqa
1117+from gaps.util.cu2d.cupstream2distro_config.c2dconfigutils.c2dconfigutils import (
1118+ load_default_cfg,
1119+ load_stack_cfg,
1120+)
1121+from qa_dashboard.settings import CU2D_LP_REPO
1122+
1123+logger = logging.getLogger('qa_dashboard')
1124+
1125+
1126+def refresh_cupstream2distro_config_repo(cu2d_config_repo_dir):
1127+ """Refresh our local lp:cupstream2distro-config repo.
1128+
1129+ NOTE: also establish the repo if it's not found.
1130+
1131+ :param cu2d_config_repo_dir: path to the repo
1132+
1133+ """
1134+ if os.path.exists(cu2d_config_repo_dir):
1135+ with os.chdir(cu2d_config_repo_dir):
1136+ subprocess.call(['bzr', 'pull'])
1137+ else:
1138+ (enclosing_dir, repo_name) = os.path.split(cu2d_config_repo_dir)
1139+ with os.chdir(enclosing_dir):
1140+ subprocess.call(
1141+ ['bzr',
1142+ 'branch',
1143+ "lp:{}".format(CU2D_LP_REPO),
1144+ repo_name]
1145+ )
1146+ # touch __init__.py here so that we can treat as a module ;)
1147+ open('__init__.py', 'a').close()
1148+
1149+
1150+def get_stack_list_from_filesystem(stacks_dir):
1151+ """Construct a list of stacks from cu2d *.cfg files in the filesystem.
1152+
1153+ :param stacks_dir: dir containing stacks (<repo>/stacks typically)
1154+ :returns: list of dicts: 'config_filepath', 'pocket', and 'name' for stack
1155+ """
1156+ stack_list = []
1157+ pocket_dirs = os.listdir(stacks_dir)
1158+ for pocket_dir in pocket_dirs:
1159+ stack_config_filenames = os.listdir(
1160+ os.path.join(
1161+ stacks_dir,
1162+ pocket_dir
1163+ )
1164+ )
1165+ for stack_config_filename in stack_config_filenames:
1166+ (stack_name, _) = os.path.splitext(stack_config_filename)
1167+ stack_config_filepath = os.path.join(
1168+ stacks_dir,
1169+ pocket_dir,
1170+ stack_config_filename
1171+ )
1172+ stack_list.append({'config_filepath': stack_config_filepath,
1173+ 'pocket': pocket_dir,
1174+ 'name': stack_name})
1175+ return stack_list
1176+
1177+
1178+def generate_job_configs_from_stack_config(
1179+ default_config_path,
1180+ stack,
1181+ project_name=None):
1182+ """Given a stack config file, use cu2d to generate a list of job configs.
1183+
1184+ :param default_config_path: dir containing 'default.cfg' (<repo>/ci ATM)
1185+ :param stack: 'config_filepath', 'pocket', 'name' dict representing stack
1186+ :param project_name: Launchpad name; generate only relevant configs
1187+ """
1188+ # yes this is an object
1189+ update_ci = UpdateCi()
1190+ # this is necessary due to a deeply buried reference in _get_build_script
1191+ update_ci.default_config_path = default_config_path
1192+ logger.debug("Generating jobs for stack {} for project {} from {}".format(
1193+ stack, project_name, default_config_path))
1194+ default_config = load_default_cfg(default_config_path)
1195+ stack_cfg = load_stack_cfg(stack['config_filepath'], default_config)
1196+ if not stack_cfg:
1197+ logging.error('Stack configuration failed to load. Aborting!')
1198+ return
1199+ job_configs = []
1200+ try:
1201+ update_ci.process_stack(job_configs, stack_cfg, target_project=project_name)
1202+ # TODO: seeing various and sundry errors, nail down then differentiate
1203+ except TypeError, e:
1204+ # this happens when the 'projects' field in the config is empty
1205+ pass
1206+ except Exception, e:
1207+ logger.error(
1208+ "{}: failed to process stack {}: {}.".format(
1209+ type(e), stack, e
1210+ )
1211+ )
1212+ return job_configs
1213+
1214+
1215+def generate_jenkins_job_names(
1216+ default_config_path,
1217+ stacks_dir,
1218+ project_name=None):
1219+ """Use cu2d tools to generate Jenkins job names.
1220+
1221+ :param default_config_path: dir containing 'default.cfg' (<repo>/ci ATM)
1222+ :param stacks_dir: directory containing stacks from which to generate
1223+ :param project_name: Launchpad name; return only relevant names
1224+ :returns: a list of Jenkins job names
1225+ """
1226+
1227+ stack_config_list = get_stack_list_from_filesystem(stacks_dir)
1228+ job_config_list = []
1229+ for stack_config in stack_config_list:
1230+ job_config_list.extend(
1231+ generate_job_configs_from_stack_config(
1232+ default_config_path,
1233+ stack_config,
1234+ project_name
1235+ )
1236+ )
1237+ job_names = [job_config['name'] for job_config in job_config_list]
1238+ return job_names
1239
1240=== modified file 'gaps/util/jenkins_pull.py'
1241--- gaps/util/jenkins_pull.py 2014-05-21 14:32:58 +0000
1242+++ gaps/util/jenkins_pull.py 2014-05-27 13:57:11 +0000
1243@@ -4,7 +4,13 @@
1244
1245 from jenkinsapi.api import Jenkins
1246 from jenkinsapi.custom_exceptions import JenkinsAPIException
1247-from qa_dashboard.settings import JENKINS_URL
1248+
1249+from gaps.util.cu2d import generate_jenkins_job_names
1250+from qa_dashboard.settings import (
1251+ JENKINS_URL,
1252+ CU2D_DEFAULT_CONFIG_DIR,
1253+ CU2D_STACKS_DIR,
1254+)
1255
1256 logger = logging.getLogger('qa_dashboard')
1257
1258@@ -33,7 +39,10 @@
1259 return urls
1260
1261
1262-def url_artifact_list(jenkins_job_name, artifact_name, last_build=0):
1263+def url_artifact_list(jenkins_job_name,
1264+ artifact_name,
1265+ last_build=0,
1266+ just_last_build=False):
1267 """
1268 gets a specific artifact for all successful builds from
1269 specified job_name
1270@@ -42,6 +51,7 @@
1271 :param string artifact_name: the file name of the artifact in jenkins
1272 :param integer last_build: the last build already stored in the database
1273 use this to return a list of only artifacts that are not in the database
1274+ :param just_last_build: get just the last build?
1275
1276 example:
1277 >>>jenkins_pull.url_list(
1278@@ -56,17 +66,20 @@
1279 job = jenkapi.get_job(jenkins_job_name)
1280 except:
1281 logger.debug(
1282- "job with name {} does not exist".format(jenkins_job_name))
1283+ "Job with name {} does not exist.".format(jenkins_job_name))
1284 return False
1285 try:
1286 build_ids = job.get_build_ids()
1287- except JenkinsAPIException:
1288+ except JenkinsAPIException, e:
1289+ logger.debug("JenkinsAPIException: {}.".format(e))
1290 return False
1291+ if just_last_build:
1292+ build_ids = [list(build_ids)[-1]]
1293 url_list = []
1294- for id in build_ids:
1295- if int(last_build) < int(id):
1296+ for build_id in build_ids:
1297+ if int(last_build) < int(build_id):
1298 try:
1299- build = job.get_build(id)
1300+ build = job.get_build(build_id)
1301 except:
1302 continue
1303 ran_at = build.get_timestamp()
1304@@ -80,13 +93,13 @@
1305 logger.debug("adding url {}".format(url))
1306 url_list.append({
1307 'url': ''.join(url),
1308- 'build_id': id,
1309+ 'build_id': build_id,
1310 'ran_at': ran_at
1311 })
1312 else:
1313 url_list.append({
1314 'url': job.baseurl,
1315- 'build_id': id,
1316+ 'build_id': build_id,
1317 'ran_at': ran_at
1318 })
1319 logger.debug(url_list)
1320
1321=== modified file 'gaps/views.py'
1322--- gaps/views.py 2014-04-04 21:40:40 +0000
1323+++ gaps/views.py 2014-05-27 13:57:11 +0000
1324@@ -10,33 +10,33 @@
1325 from gaps.models import (CoverageProject,
1326 CoverageBuild,
1327 CoverageData,
1328- CoverageStack,
1329- StackCoverageObservation)
1330+ CoverageProduct,
1331+ ProductCoverageObservation)
1332
1333 logger = logging.getLogger('qa_dashboard')
1334
1335
1336-@BreadCrumb("Stacks")
1337-def stack_list(request):
1338- stack_list_ = CoverageStack.objects.all()
1339- t = loader.get_template('gaps/stack_list.html')
1340- c = Context({'stack_list': stack_list_,
1341+@BreadCrumb("Products")
1342+def product_list(request):
1343+ product_list_ = CoverageProduct.objects.all()
1344+ t = loader.get_template('gaps/product_list.html')
1345+ c = Context({'product_list': product_list_,
1346 'bread_crumb_trail': BreadCrumbTrail.leading_to(
1347- stack_list)})
1348+ product_list)})
1349 return HttpResponse(t.render(c))
1350
1351-@BreadCrumb("{name}", parent=stack_list, needs=['name'])
1352-def stack(request, name):
1353- stack_ = CoverageStack.objects.get(name=name)
1354- stack_list = CoverageStack.objects.all()
1355+@BreadCrumb("{name}", parent=product_list, needs=['name'])
1356+def product(request, name):
1357+ product_ = CoverageProduct.objects.get(name=name)
1358+ product_list = CoverageProduct.objects.all()
1359 project_list = CoverageProject.objects.filter(
1360- stack__name=name)
1361- t = loader.get_template('gaps/stack_detail.html')
1362- c = Context({'stack': stack_,
1363- 'stack_list': stack_list,
1364+ product__name=name)
1365+ t = loader.get_template('gaps/product_detail.html')
1366+ c = Context({'product': product_,
1367+ 'product_list': product_list,
1368 'project_list': project_list,
1369 'bread_crumb_trail': BreadCrumbTrail.leading_to(
1370- stack, name=stack_.name)})
1371+ product, name=product_.name)})
1372 return HttpResponse(t.render(c))
1373
1374 @BreadCrumb("Projects")
1375
1376=== modified file 'qa_dashboard/settings.py'
1377--- qa_dashboard/settings.py 2014-05-21 12:47:08 +0000
1378+++ qa_dashboard/settings.py 2014-05-27 13:57:11 +0000
1379@@ -42,6 +42,18 @@
1380 PROJECT_PATH = os.path.dirname(os.path.abspath(__file__))
1381 LAUNCHPADLIB_CACHE = ".cache/launchpadlib"
1382
1383+CU2D_CONFIG_REPO_DIR = os.path.join(
1384+ PROJECT_PATH,
1385+ '..',
1386+ 'gaps',
1387+ 'util',
1388+ 'cu2d',
1389+ 'cupstream2distro_config'
1390+)
1391+CU2D_DEFAULT_CONFIG_DIR = os.path.join(CU2D_CONFIG_REPO_DIR, 'ci')
1392+CU2D_STACKS_DIR = os.path.join(CU2D_CONFIG_REPO_DIR, 'stacks')
1393+CU2D_LP_REPO = 'cupstream2distro-config'
1394+
1395 JENKINS_URL = "http://jenkins.qa.ubuntu.com"
1396
1397 MANAGERS = ADMINS

Subscribers

People subscribed via source and target branches

to all changes: