qakit

Merge lp:~canonical-platform-qa/qakit/add-vis into lp:qakit

  1. add-vis
  2. Merge into trunk
Proposed by Allan LeSage
Status: Merged
Approved by: Allan LeSage
Approved revision: 117
Merged at revision: 127
Proposed branch: lp:~canonical-platform-qa/qakit/add-vis
Merge into: lp:qakit
Diff against target: 1346 lines (+1273/-0)
14 files modified
qakit/vis/dashboard/main.js (+55/-0)
qakit/vis/dashboard/ust_rc-proposed_arale_regression.html (+48/-0)
qakit/vis/dashboard/ust_rc-proposed_arale_sanity.html (+48/-0)
qakit/vis/dashboard/ust_rc-proposed_krillin_regression.html (+48/-0)
qakit/vis/dashboard/ust_rc-proposed_krillin_sanity.html (+48/-0)
qakit/vis/dashboard/ust_rc_arale_regression.html (+48/-0)
qakit/vis/dashboard/ust_rc_arale_sanity.html (+48/-0)
qakit/vis/dashboard/ust_rc_krillin_regression.html (+48/-0)
qakit/vis/dashboard/ust_rc_krillin_sanity.html (+48/-0)
qakit/vis/jenkins_results.py (+70/-0)
qakit/vis/retrieve_jenkins_results.py (+229/-0)
qakit/vis/tests/test_retrieve_jenkins_results.py (+47/-0)
qakit/vis/tests/test_visualize.py (+194/-0)
qakit/vis/visualize.py (+294/-0)
To merge this branch: bzr merge lp:~canonical-platform-qa/qakit/add-vis
Reviewer Review Type Date Requested Status
Canonical Platform QA Team Pending
Review via email: mp+299606@code.launchpad.net

Commit message

Add a vis tool to visualize test suite performance.

Description of the change

Add a 'vis' tool for visualizing test suite status.

Presently we're using a few html files to present this datatables/bootstrap version, could be nicer with tabs but we'll save that for a later version.

To post a comment you must log in.
Revision history for this message
Sergio Cazzolato (sergio-j-cazzolato) wrote :

I made a high level review, I left some comments inline.

lp:~canonical-platform-qa/qakit/add-vis updated
113. By Allan LeSage

Move dashboard-type files to a dashboard dir.

114. By Allan LeSage

Context manager for json writer.

115. By Allan LeSage

Use statistics to compute mean.

116. By Allan LeSage

Move to dao, fix tests.

117. By Allan LeSage

Typo!

Revision history for this message
Allan LeSage (allanlesage) wrote :

Made all the requested changes, thanks cachio for a further review!

Revision history for this message
Sergio Cazzolato (sergio-j-cazzolato) wrote :

Organized now! go ahead with this change

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'qakit/vis'
2=== added directory 'qakit/vis/dashboard'
3=== added file 'qakit/vis/dashboard/main.js'
4--- qakit/vis/dashboard/main.js 1970-01-01 00:00:00 +0000
5+++ qakit/vis/dashboard/main.js 2016-07-13 18:36:50 +0000
6@@ -0,0 +1,55 @@
7+
8+function createdCellFtn(cell, cellData, rowData, rowIndex, colIndex) {
9+ if (cellData.skipped == true) {
10+ // skipped tests get dashed borders
11+ $(cell).css('border-style', 'dashed');
12+ $(cell).css('border-width', 'medium');
13+ } else {
14+ // tests get colors
15+ $(cell).css('background-color', cellData.color);
16+ };
17+ if (colIndex > 2) {
18+ $(cell).html("<a href=" + cellData.link + ">&nbsp;</a>");
19+ } else if (colIndex == 2) {
20+ // no links in the trend column pls
21+ $(cell).html("");
22+ };
23+}
24+
25+function visTable(data, selector) {
26+ var tableData = data.data.map(function(obj) {
27+ var row = [ obj.section, obj.test_name ];
28+ row.push(obj.trend);
29+ obj.builds.forEach(function(build) {
30+ row.push(build);
31+ });
32+ return row;
33+ });
34+ var tableColumns = data.header.map(function(obj) {
35+ return {
36+ title: obj,
37+ createdCell: createdCellFtn,
38+ orderable: false,
39+ searchable: false,
40+ // hide Section column
41+ visible: (obj == 'Section' ? false : true),
42+ };
43+ });
44+ $(selector).dataTable({
45+ data: tableData,
46+ columns: tableColumns,
47+ searching: false,
48+ order: [[0, 'dsc']],
49+ bDestroy: true,
50+ lengthChange: false,
51+ lengthMenu: [-1],
52+ });
53+}
54+
55+function draw(filename) {
56+ $.getJSON(filename, function(data) {
57+ visTable(data, "#visTable");
58+ }).fail(function(data, status) {
59+ console.log("Faled to load vis JSON.");
60+ });
61+}
62
63=== added file 'qakit/vis/dashboard/ust_rc-proposed_arale_regression.html'
64--- qakit/vis/dashboard/ust_rc-proposed_arale_regression.html 1970-01-01 00:00:00 +0000
65+++ qakit/vis/dashboard/ust_rc-proposed_arale_regression.html 2016-07-13 18:36:50 +0000
66@@ -0,0 +1,48 @@
67+<!DOCTYPE html>
68+<html>
69+<head>
70+ <meta charset="utf-8">
71+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
72+ <meta name="viewport" content="width=device-width, initial-scale=1">
73+
74+ <title>Platform QA - VIS</title>
75+
76+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
77+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
78+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
79+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
80+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
81+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
82+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
83+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
84+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
85+ <script src="main.js"></script>
86+ <style>
87+ .ar {
88+ text-align: right;
89+ }
90+ .vertical{
91+ writing-mode:tb-rl;
92+ white-space:nowrap;
93+ display:block;
94+ bottom:0;
95+ height:20px;
96+ }
97+ td a {
98+ display:block;
99+ width:100%;
100+ text-decoration:none;
101+ }
102+ </style>
103+</head>
104+<body>
105+ <div class="col-md-12">
106+ <table id="visTable" class="table table-striped table-bordered"></table>
107+ </div>
108+ <script>
109+ window.onload = function() {
110+ draw("ust_rc-proposed_arale_regression.json");
111+ }
112+ </script>
113+</body>
114+</html>
115
116=== added file 'qakit/vis/dashboard/ust_rc-proposed_arale_sanity.html'
117--- qakit/vis/dashboard/ust_rc-proposed_arale_sanity.html 1970-01-01 00:00:00 +0000
118+++ qakit/vis/dashboard/ust_rc-proposed_arale_sanity.html 2016-07-13 18:36:50 +0000
119@@ -0,0 +1,48 @@
120+<!DOCTYPE html>
121+<html>
122+<head>
123+ <meta charset="utf-8">
124+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
125+ <meta name="viewport" content="width=device-width, initial-scale=1">
126+
127+ <title>Platform QA - VIS</title>
128+
129+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
130+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
131+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
132+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
133+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
134+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
135+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
136+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
137+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
138+ <script src="main.js"></script>
139+ <style>
140+ .ar {
141+ text-align: right;
142+ }
143+ .vertical{
144+ writing-mode:tb-rl;
145+ white-space:nowrap;
146+ display:block;
147+ bottom:0;
148+ height:20px;
149+ }
150+ td a {
151+ display:block;
152+ width:100%;
153+ text-decoration:none;
154+ }
155+ </style>
156+</head>
157+<body>
158+ <div class="col-md-12">
159+ <table id="visTable" class="table table-striped table-bordered"></table>
160+ </div>
161+ <script>
162+ window.onload = function() {
163+ draw("ust_rc-proposed_arale_sanity.json");
164+ }
165+ </script>
166+</body>
167+</html>
168
169=== added file 'qakit/vis/dashboard/ust_rc-proposed_krillin_regression.html'
170--- qakit/vis/dashboard/ust_rc-proposed_krillin_regression.html 1970-01-01 00:00:00 +0000
171+++ qakit/vis/dashboard/ust_rc-proposed_krillin_regression.html 2016-07-13 18:36:50 +0000
172@@ -0,0 +1,48 @@
173+<!DOCTYPE html>
174+<html>
175+<head>
176+ <meta charset="utf-8">
177+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
178+ <meta name="viewport" content="width=device-width, initial-scale=1">
179+
180+ <title>Platform QA - VIS</title>
181+
182+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
183+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
184+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
185+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
186+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
187+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
188+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
189+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
190+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
191+ <script src="main.js"></script>
192+ <style>
193+ .ar {
194+ text-align: right;
195+ }
196+ .vertical{
197+ writing-mode:tb-rl;
198+ white-space:nowrap;
199+ display:block;
200+ bottom:0;
201+ height:20px;
202+ }
203+ td a {
204+ display:block;
205+ width:100%;
206+ text-decoration:none;
207+ }
208+ </style>
209+</head>
210+<body>
211+ <div class="col-md-12">
212+ <table id="visTable" class="table table-striped table-bordered"></table>
213+ </div>
214+ <script>
215+ window.onload = function() {
216+ draw("ust_rc-proposed_krillin_regression.json");
217+ }
218+ </script>
219+</body>
220+</html>
221
222=== added file 'qakit/vis/dashboard/ust_rc-proposed_krillin_sanity.html'
223--- qakit/vis/dashboard/ust_rc-proposed_krillin_sanity.html 1970-01-01 00:00:00 +0000
224+++ qakit/vis/dashboard/ust_rc-proposed_krillin_sanity.html 2016-07-13 18:36:50 +0000
225@@ -0,0 +1,48 @@
226+<!DOCTYPE html>
227+<html>
228+<head>
229+ <meta charset="utf-8">
230+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
231+ <meta name="viewport" content="width=device-width, initial-scale=1">
232+
233+ <title>Platform QA - VIS</title>
234+
235+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
236+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
237+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
238+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
239+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
240+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
241+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
242+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
243+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
244+ <script src="main.js"></script>
245+ <style>
246+ .ar {
247+ text-align: right;
248+ }
249+ .vertical{
250+ writing-mode:tb-rl;
251+ white-space:nowrap;
252+ display:block;
253+ bottom:0;
254+ height:20px;
255+ }
256+ td a {
257+ display:block;
258+ width:100%;
259+ text-decoration:none;
260+ }
261+ </style>
262+</head>
263+<body>
264+ <div class="col-md-12">
265+ <table id="visTable" class="table table-striped table-bordered"></table>
266+ </div>
267+ <script>
268+ window.onload = function() {
269+ draw("ust_rc-proposed_krillin_sanity.json");
270+ }
271+ </script>
272+</body>
273+</html>
274
275=== added file 'qakit/vis/dashboard/ust_rc_arale_regression.html'
276--- qakit/vis/dashboard/ust_rc_arale_regression.html 1970-01-01 00:00:00 +0000
277+++ qakit/vis/dashboard/ust_rc_arale_regression.html 2016-07-13 18:36:50 +0000
278@@ -0,0 +1,48 @@
279+<!DOCTYPE html>
280+<html>
281+<head>
282+ <meta charset="utf-8">
283+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
284+ <meta name="viewport" content="width=device-width, initial-scale=1">
285+
286+ <title>Platform QA - VIS</title>
287+
288+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
289+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
290+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
291+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
292+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
293+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
294+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
295+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
296+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
297+ <script src="main.js"></script>
298+ <style>
299+ .ar {
300+ text-align: right;
301+ }
302+ .vertical{
303+ writing-mode:tb-rl;
304+ white-space:nowrap;
305+ display:block;
306+ bottom:0;
307+ height:20px;
308+ }
309+ td a {
310+ display:block;
311+ width:100%;
312+ text-decoration:none;
313+ }
314+ </style>
315+</head>
316+<body>
317+ <div class="col-md-12">
318+ <table id="visTable" class="table table-striped table-bordered"></table>
319+ </div>
320+ <script>
321+ window.onload = function() {
322+ draw("ust_rc_arale_regression.json");
323+ }
324+ </script>
325+</body>
326+</html>
327
328=== added file 'qakit/vis/dashboard/ust_rc_arale_sanity.html'
329--- qakit/vis/dashboard/ust_rc_arale_sanity.html 1970-01-01 00:00:00 +0000
330+++ qakit/vis/dashboard/ust_rc_arale_sanity.html 2016-07-13 18:36:50 +0000
331@@ -0,0 +1,48 @@
332+<!DOCTYPE html>
333+<html>
334+<head>
335+ <meta charset="utf-8">
336+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
337+ <meta name="viewport" content="width=device-width, initial-scale=1">
338+
339+ <title>Platform QA - VIS</title>
340+
341+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
342+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
343+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
344+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
345+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
346+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
347+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
348+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
349+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
350+ <script src="main.js"></script>
351+ <style>
352+ .ar {
353+ text-align: right;
354+ }
355+ .vertical{
356+ writing-mode:tb-rl;
357+ white-space:nowrap;
358+ display:block;
359+ bottom:0;
360+ height:20px;
361+ }
362+ td a {
363+ display:block;
364+ width:100%;
365+ text-decoration:none;
366+ }
367+ </style>
368+</head>
369+<body>
370+ <div class="col-md-12">
371+ <table id="visTable" class="table table-striped table-bordered"></table>
372+ </div>
373+ <script>
374+ window.onload = function() {
375+ draw("ust_rc_arale_sanity.json");
376+ }
377+ </script>
378+</body>
379+</html>
380
381=== added file 'qakit/vis/dashboard/ust_rc_krillin_regression.html'
382--- qakit/vis/dashboard/ust_rc_krillin_regression.html 1970-01-01 00:00:00 +0000
383+++ qakit/vis/dashboard/ust_rc_krillin_regression.html 2016-07-13 18:36:50 +0000
384@@ -0,0 +1,48 @@
385+<!DOCTYPE html>
386+<html>
387+<head>
388+ <meta charset="utf-8">
389+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
390+ <meta name="viewport" content="width=device-width, initial-scale=1">
391+
392+ <title>Platform QA - VIS</title>
393+
394+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
395+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
396+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
397+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
398+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
399+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
400+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
401+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
402+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
403+ <script src="main.js"></script>
404+ <style>
405+ .ar {
406+ text-align: right;
407+ }
408+ .vertical{
409+ writing-mode:tb-rl;
410+ white-space:nowrap;
411+ display:block;
412+ bottom:0;
413+ height:20px;
414+ }
415+ td a {
416+ display:block;
417+ width:100%;
418+ text-decoration:none;
419+ }
420+ </style>
421+</head>
422+<body>
423+ <div class="col-md-12">
424+ <table id="visTable" class="table table-striped table-bordered"></table>
425+ </div>
426+ <script>
427+ window.onload = function() {
428+ draw("ust_rc_krillin_regression.json");
429+ }
430+ </script>
431+</body>
432+</html>
433
434=== added file 'qakit/vis/dashboard/ust_rc_krillin_sanity.html'
435--- qakit/vis/dashboard/ust_rc_krillin_sanity.html 1970-01-01 00:00:00 +0000
436+++ qakit/vis/dashboard/ust_rc_krillin_sanity.html 2016-07-13 18:36:50 +0000
437@@ -0,0 +1,48 @@
438+<!DOCTYPE html>
439+<html>
440+<head>
441+ <meta charset="utf-8">
442+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
443+ <meta name="viewport" content="width=device-width, initial-scale=1">
444+
445+ <title>Platform QA - VIS</title>
446+
447+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
448+ <link href="http://fonts.googleapis.com/css?family=Ubuntu" rel="stylesheet" type="text/css">
449+ <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
450+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.css"/>
451+ <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
452+ <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/1.0.2/Chart.min.js"></script>
453+ <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
454+ <script type="text/javascript" src="https://cdn.datatables.net/s/bs/dt-1.10.10,b-1.1.0,fh-3.1.0,r-2.0.0/datatables.min.js"></script>
455+ <script src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
456+ <script src="main.js"></script>
457+ <style>
458+ .ar {
459+ text-align: right;
460+ }
461+ .vertical{
462+ writing-mode:tb-rl;
463+ white-space:nowrap;
464+ display:block;
465+ bottom:0;
466+ height:20px;
467+ }
468+ td a {
469+ display:block;
470+ width:100%;
471+ text-decoration:none;
472+ }
473+ </style>
474+</head>
475+<body>
476+ <div class="col-md-12">
477+ <table id="visTable" class="table table-striped table-bordered"></table>
478+ </div>
479+ <script>
480+ window.onload = function() {
481+ draw("ust_rc_krillin_sanity.json");
482+ }
483+ </script>
484+</body>
485+</html>
486
487=== added file 'qakit/vis/jenkins_results.py'
488--- qakit/vis/jenkins_results.py 1970-01-01 00:00:00 +0000
489+++ qakit/vis/jenkins_results.py 2016-07-13 18:36:50 +0000
490@@ -0,0 +1,70 @@
491+#!/usr/bin/env python3
492+# UEQA Vis
493+# Copyright (C) 2016 Canonical
494+#
495+# This program is free software: you can redistribute it and/or modify
496+# it under the terms of the GNU General Public License as published by
497+# the Free Software Foundation, either version 3 of the License, or
498+# (at your option) any later version.
499+#
500+# This program is distributed in the hope that it will be useful,
501+# but WITHOUT ANY WARRANTY; without even the implied warranty of
502+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
503+# GNU General Public License for more details.
504+#
505+# You should have received a copy of the GNU General Public License
506+# along with this program. If not, see <http://www.gnu.org/licenses/>.
507+
508+
509+import pymongo
510+
511+
512+class JenkinsResults:
513+
514+ def __init__(self, db):
515+ self.db = db
516+
517+ def get_build_test_results(self, job_name, build_number):
518+ """Query db for test results.
519+
520+ NOTE that we don't return skipped tests.
521+
522+ """
523+ return self.db.tests.find(
524+ {'job_name': job_name,
525+ 'build_number': build_number,
526+ 'status': {'$ne': 'SKIPPED'}}
527+ )
528+
529+ def get_test_result(self, job_name, build_number, test_name):
530+ """Query db for a single test result."""
531+ return self.db.tests.find_one(
532+ {'test_name': test_name,
533+ 'job_name': job_name,
534+ 'build_number': build_number})
535+
536+ def get_trailing_test_results(self, job_name, test_name, limit=10):
537+ """Return last 'limit' test_name results."""
538+ return self.db.tests.find(
539+ {'test_name': test_name,
540+ 'job_name': job_name,
541+ 'status': {'$ne': 'SKIPPED'}}).sort(
542+ 'build_number', pymongo.DESCENDING).limit(limit)
543+
544+ def get_test_names(self, job_name):
545+ """Get all test names associated with a Jenkins job from db."""
546+ return self.db.tests.find(
547+ {'job_name': job_name}).distinct('test_name')
548+
549+ def get_build_numbers(self, job_name):
550+ """Get all build numbers associated with a Jenkins job from db."""
551+ return sorted(
552+ self.db.builds.find({'job_name': job_name}).distinct('number'),
553+ reverse=True)
554+
555+ def get_build(self, job_name, build_number):
556+ """Return a specific build."""
557+ return self.db.builds.find_one(
558+ {'job_name': job_name,
559+ 'number': build_number}
560+ )
561
562=== added file 'qakit/vis/retrieve_jenkins_results.py'
563--- qakit/vis/retrieve_jenkins_results.py 1970-01-01 00:00:00 +0000
564+++ qakit/vis/retrieve_jenkins_results.py 2016-07-13 18:36:50 +0000
565@@ -0,0 +1,229 @@
566+#!/usr/bin/python3
567+
568+# QA KPI Utilities
569+# Copyright (C) 2015-2016 Canonical
570+#
571+# This program is free software: you can redistribute it and/or modify
572+# it under the terms of the GNU General Public License as published by
573+# the Free Software Foundation, either version 3 of the License, or
574+# (at your option) any later version.
575+#
576+# This program is distributed in the hope that it will be useful,
577+# but WITHOUT ANY WARRANTY; without even the implied warranty of
578+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
579+# GNU General Public License for more details.
580+#
581+# You should have received a copy of the GNU General Public License
582+# along with this program. If not, see <http://www.gnu.org/licenses/>.
583+
584+import argparse
585+import logging
586+import sys
587+
588+import bson
589+import jenkinsapi
590+import pymongo
591+
592+logger = logging.getLogger(__name__)
593+handler = logging.StreamHandler()
594+handler.setLevel(logging.DEBUG)
595+logger.addHandler(handler)
596+logger.setLevel(logging.DEBUG)
597+
598+
599+def filter_job_names(job_names, pattern):
600+ return list(filter(lambda x: pattern in x, job_names))
601+
602+
603+class JenkinsMongoSession():
604+
605+ def __init__(self, db, baseurl, username=None, password=None):
606+ super().__init__()
607+ self.db = db
608+ self.jenkins = jenkinsapi.jenkins.Jenkins(
609+ baseurl, username, password)
610+
611+ def get_job(self, job_name):
612+ return self.jenkins[job_name]
613+
614+ def retrieve_build_resultset(self, db_build_id):
615+ """Retrieve the resultset of a known build in our db."""
616+ db_build = self.db.builds.find_one(
617+ {'_id': bson.ObjectId(db_build_id)})
618+ job_name = db_build['job_name']
619+ jenkins_job = self.get_job(job_name)
620+ build_number = db_build['number']
621+ jenkins_build = jenkins_job.get_build(build_number)
622+ try:
623+ resultset = jenkins_build.get_resultset()
624+ except jenkinsapi.custom_exceptions.NoResults:
625+ return {}
626+ test_results = []
627+ for test_name in resultset.keys():
628+ result = resultset[test_name]
629+ test_result = {
630+ 'test_name': test_name,
631+ 'status': result.status,
632+ 'build_number': build_number,
633+ 'build_timestamp': db_build['timestamp'],
634+ 'build_name': db_build['description'],
635+ 'job_name': job_name,
636+ }
637+ test_results.append(test_result)
638+ return test_results
639+
640+ def retrieve_and_insert_build_resultset(self, db_build_id):
641+ """Retrieve test results from Jenkins, inserting into our db."""
642+ build_resultset = self.retrieve_build_resultset(db_build_id)
643+ return self.db.tests.insert(build_resultset)
644+
645+ def retrieve_build(self, job_name, build_number):
646+ """Retrieve build from Jenkins."""
647+ logger.info(
648+ 'Retrieving {} build #{}.'.format(job_name, build_number)
649+ )
650+ job = self.get_job(job_name)
651+ build = job.get_build(build_number)
652+ data = build.get_data(build.python_api_url(build.baseurl))
653+ actions = build.get_actions()
654+ total = failed = skipped = 0
655+ try:
656+ total = actions['totalCount']
657+ failed = actions['failCount']
658+ skipped = actions['skipCount']
659+ except KeyError:
660+ # we don't have a real result here but we'll still save
661+ pass
662+ return dict({
663+ 'job_name': job_name,
664+ 'total': total,
665+ 'failed': failed,
666+ 'skipped': skipped},
667+ **data)
668+
669+ def retrieve_and_insert_build(self, job_name, build_number):
670+ """Retrieve build from Jenkins and deposit in db."""
671+ build_results = self.retrieve_build(job_name, build_number)
672+ return self.db.builds.insert(build_results)
673+
674+ def get_db_build_ids_for_job(self, job_name):
675+ """Get a list of known build ids for the given job for comparison."""
676+ return [
677+ build['number'] for build in self.db.builds.find(
678+ {'job_name': job_name},
679+ {'number': 1, '_id': 0})
680+ ]
681+
682+ def get_missing_builds(self, job_name):
683+ """Return a list of build ids in Jenkins we're missing in db."""
684+ job = self.get_job(job_name)
685+ jenkins_build_ids = set(job.get_build_ids())
686+ db_build_ids = set(self.get_db_build_ids_for_job(job_name))
687+ missing_builds = [
688+ build_id for build_id in
689+ # take the set-difference
690+ list(jenkins_build_ids.difference(db_build_ids))
691+ # filter out running builds
692+ if not job.get_build(build_id).is_running()]
693+ logger.info(
694+ 'For job {}, missing builds {}.'.format(job_name, missing_builds))
695+ return missing_builds
696+
697+ def retrieve_data_for_job(self, job_name):
698+ """Retrieve builds and results from Jenkins, inserting into our db."""
699+ logger.debug('Retrieving data for job {}.'.format(job_name))
700+ build_numbers = self.get_missing_builds(job_name)
701+ for build_number in reversed(build_numbers):
702+ db_build_id = self.retrieve_and_insert_build(
703+ job_name, build_number)
704+ self.retrieve_and_insert_build_resultset(db_build_id)
705+
706+ def retrieve_jenkins_results(self, job_names=None, pattern=None):
707+ """Retrieve builds and results either by name or matching a pattern."""
708+ if pattern is not None:
709+ all_job_names = self.jenkins.get_jobs_list()
710+ job_names = filter_job_names(all_job_names, pattern)
711+ for job_name in job_names:
712+ self.retrieve_data_for_job(job_name)
713+
714+
715+def _parse_arguments():
716+ parser = argparse.ArgumentParser(
717+ "Retrieve Jenkins build results, storing in a mongo db.")
718+ parser.add_argument(
719+ '-v',
720+ '--verbose',
721+ help='Verbose logging.',
722+ default=False,
723+ action='store_true',
724+ )
725+ parser.add_argument(
726+ '--vis-db',
727+ help='MongoDB collection in which to store data.',
728+ type=str,
729+ default='vis',
730+ required=False,
731+ )
732+ parser.add_argument(
733+ '--jenkins-url',
734+ help='Jenkins URL from which to collect data.',
735+ type=str,
736+ required=True,
737+ )
738+ parser.add_argument(
739+ '--jenkins-user',
740+ help='Jenkins username.',
741+ type=str,
742+ required=False,
743+ )
744+ parser.add_argument(
745+ '--jenkins-password',
746+ help='Jenkins API token for Jenkins user.',
747+ type=str,
748+ required=False,
749+ )
750+ parser.add_argument(
751+ '--mongo-host',
752+ help='MongoDB host IP.',
753+ type=str,
754+ default='127.0.0.1',
755+ required=False,
756+ )
757+ parser.add_argument(
758+ nargs='*',
759+ help='Job names from which to collect data.',
760+ type=str,
761+ dest='job_names',
762+ )
763+ parser.add_argument(
764+ '-p',
765+ '--pattern',
766+ help='Job name search pattern.',
767+ type=str,
768+ )
769+ args = parser.parse_args()
770+ if args.pattern and args.job_names:
771+ print("Please specify either job names or pattern.")
772+ parser.print_help()
773+ sys.exit(1)
774+ return args
775+
776+
777+def main():
778+ args = _parse_arguments()
779+ if args.verbose:
780+ logger.setLevel(logging.DEBUG)
781+ conn = pymongo.MongoClient(host=args.mongo_host)
782+ db = getattr(conn, args.vis_db)
783+ session = JenkinsMongoSession(
784+ db,
785+ args.jenkins_url,
786+ args.jenkins_user,
787+ args.jenkins_password,
788+ )
789+ session.retrieve_jenkins_results(
790+ args.job_names, args.pattern)
791+
792+
793+if __name__ == '__main__':
794+ sys.exit(main())
795
796=== added directory 'qakit/vis/tests'
797=== added file 'qakit/vis/tests/__init__.py'
798=== added file 'qakit/vis/tests/test_retrieve_jenkins_results.py'
799--- qakit/vis/tests/test_retrieve_jenkins_results.py 1970-01-01 00:00:00 +0000
800+++ qakit/vis/tests/test_retrieve_jenkins_results.py 2016-07-13 18:36:50 +0000
801@@ -0,0 +1,47 @@
802+#!/usr/bin/env python3
803+# UEQA VIS
804+# Copyright (C) 2016 Canonical
805+#
806+# This program is free software: you can redistribute it and/or modify
807+# it under the terms of the GNU General Public License as published by
808+# the Free Software Foundation, either version 3 of the License, or
809+# (at your option) any later version.
810+#
811+# This program is distributed in the hope that it will be useful,
812+# but WITHOUT ANY WARRANTY; without even the implied warranty of
813+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
814+# GNU General Public License for more details.
815+#
816+# You should have received a copy of the GNU General Public License
817+# along with this program. If not, see <http://www.gnu.org/licenses/>.
818+
819+import unittest
820+import unittest.mock as mock
821+
822+import qakit.vis.retrieve_jenkins_results as retrieve
823+
824+
825+JOB_NAMES = [
826+ 'ust_rc-proposed_krillin_sanity',
827+ 'ust_rc-proposed_krillin_regression',
828+ 'ust_rc-proposed_arale_sanity',
829+ 'foo',
830+]
831+
832+
833+class RegexTestCase(unittest.TestCase):
834+
835+ def test_vanilla(self):
836+ result = retrieve.filter_job_names(JOB_NAMES, 'ust_')
837+ self.assertEqual(
838+ ['ust_rc-proposed_krillin_sanity',
839+ 'ust_rc-proposed_krillin_regression',
840+ 'ust_rc-proposed_arale_sanity'],
841+ result)
842+
843+ def test_just_krillin(self):
844+ result = retrieve.filter_job_names(JOB_NAMES, 'krillin')
845+ self.assertEqual(
846+ ['ust_rc-proposed_krillin_sanity',
847+ 'ust_rc-proposed_krillin_regression'],
848+ result)
849
850=== added file 'qakit/vis/tests/test_visualize.py'
851--- qakit/vis/tests/test_visualize.py 1970-01-01 00:00:00 +0000
852+++ qakit/vis/tests/test_visualize.py 2016-07-13 18:36:50 +0000
853@@ -0,0 +1,194 @@
854+#!/usr/bin/env python3
855+# UEQA VIS
856+# Copyright (C) 2014-2016 Canonical
857+#
858+# This program is free software: you can redistribute it and/or modify
859+# it under the terms of the GNU General Public License as published by
860+# the Free Software Foundation, either version 3 of the License, or
861+# (at your option) any later version.
862+#
863+# This program is distributed in the hope that it will be useful,
864+# but WITHOUT ANY WARRANTY; without even the implied warranty of
865+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
866+# GNU General Public License for more details.
867+#
868+# You should have received a copy of the GNU General Public License
869+# along with this program. If not, see <http://www.gnu.org/licenses/>.
870+
871+import unittest
872+import unittest.mock as mock
873+
874+import qakit.vis.visualize as visualize
875+
876+
877+class StatusToFloatTestCase(unittest.TestCase):
878+
879+ def test_passed(self):
880+ result = visualize.status_to_float('PASSED')
881+ self.assertEqual(1.0, result)
882+
883+ def test_failed(self):
884+ result = visualize.status_to_float('FAILED')
885+ self.assertEqual(0.0, result)
886+
887+ def test_fixed(self):
888+ result = visualize.status_to_float('FIXED')
889+ self.assertEqual(1.0, result)
890+
891+ def test_regression(self):
892+ result = visualize.status_to_float('REGRESSION')
893+ self.assertEqual(0.0, result)
894+
895+ def test_gibberish(self):
896+ result = visualize.status_to_float('GIBBERISH')
897+ self.assertEqual(0.0, result)
898+
899+
900+class ComputeColorTestCase(unittest.TestCase):
901+
902+ def test_compute_color_pass(self):
903+ self.assertEqual('rgb(0, 255, 0)',
904+ visualize.compute_color(1.0))
905+
906+ def test_compute_color_fail(self):
907+ self.assertEqual('rgb(255, 0, 0)',
908+ visualize.compute_color(0.0))
909+
910+ def test_compute_color_none(self):
911+ self.assertEqual('rgb(255, 255, 255)',
912+ visualize.compute_color(None))
913+
914+
915+class DbVisTestCase(unittest.TestCase):
916+
917+ def setUp(self):
918+ super().setUp()
919+ self.db = mock.Mock(
920+ vis=mock.Mock())
921+
922+
923+@unittest.skip('Temporarily deprecated.')
924+class GetPassRateTestCase(unittest.TestCase):
925+
926+ def test_pass_rate(self):
927+ pass_rate = 0.9
928+ pass_rate_dict = {'pass_rate': pass_rate}
929+ db = mock.Mock(
930+ dashboard_tests_pass_rates=mock.Mock(
931+ find_one=mock.Mock(
932+ return_value=pass_rate_dict)))
933+ self.assertEqual(
934+ pass_rate,
935+ visualize.get_pass_rate(
936+ 'fake_suite_name',
937+ 'fake_build_name',
938+ db))
939+
940+ def test_pass_rate_no_results(self):
941+ db = mock.Mock(
942+ dashboard_tests_pass_rates=mock.Mock(
943+ find_one=mock.Mock(
944+ side_effect=TypeError)))
945+ self.assertEqual(
946+ None,
947+ visualize.get_pass_rate(
948+ 'fake_suite_name',
949+ 'fake_build_name',
950+ db))
951+
952+
953+class GetUrlPathFromTestNameTestCase(unittest.TestCase):
954+
955+ def test_vanilla(self):
956+ result = visualize.get_url_path_from_test_name(
957+ 'ubuntu_system_tests.tests.test_calls.CallsTestCase.'
958+ 'test_receive_incoming_call_from_contact')
959+ self.assertEqual(
960+ 'ubuntu_system_tests.tests.test_calls/CallsTestCase/'
961+ 'test_receive_incoming_call_from_contact',
962+ result)
963+
964+ def test_more_nesting(self):
965+ result = visualize.get_url_path_from_test_name(
966+ 'ubuntu_system_tests.tests.webapps.test_here.HereTestCase.'
967+ 'test_retrieve_location')
968+ self.assertEqual(
969+ 'ubuntu_system_tests.tests.webapps.test_here/HereTestCase/'
970+ 'test_retrieve_location',
971+ result)
972+
973+ def test_odd_test_name_directs_to_build_test_result(self):
974+ result = visualize.get_url_path_from_test_name(
975+ 'subunit.parser')
976+ self.assertEqual('', result)
977+
978+
979+class ComputePassRateFromListOfResultsTestCase(unittest.TestCase):
980+
981+ def test_vanilla(self):
982+ result = visualize.compute_pass_rate_from_statuses(
983+ ['PASSED', 'PASSED'])
984+ self.assertEqual(1.0, result)
985+
986+ def test_half_passed(self):
987+ result = visualize.compute_pass_rate_from_statuses(
988+ ['PASSED', 'FAILED'])
989+ self.assertEqual(0.5, result)
990+
991+
992+class SplitTestNameTestCase(unittest.TestCase):
993+
994+ def test_vanilla(self):
995+ result = visualize.split_test_name(
996+ 'foo.bar.FooTestCase.test_something')
997+ self.assertEqual(
998+ ('foo.bar', 'FooTestCase.test_something'),
999+ result,
1000+ )
1001+
1002+ def test_deeper_nesting(self):
1003+ result = visualize.split_test_name(
1004+ 'foo.bar.baz.FooTestCase.test_something')
1005+ self.assertEqual(
1006+ ('foo.bar.baz', 'FooTestCase.test_something'),
1007+ result,
1008+ )
1009+
1010+ def test_gibberish(self):
1011+ result = visualize.split_test_name(
1012+ 'gibberish')
1013+ self.assertEqual(
1014+ ('gibberish', 'gibberish'),
1015+ result,
1016+ )
1017+
1018+ def test_gibberish_with_dots(self):
1019+ result = visualize.split_test_name(
1020+ 'gibberish.test_something')
1021+ self.assertEqual(
1022+ ('gibberish.test_something', 'gibberish.test_something'),
1023+ result,
1024+ )
1025+
1026+
1027+class GetColorFromStatus(unittest.TestCase):
1028+
1029+ def test_skipped(self):
1030+ result = visualize.get_color_from_status('SKIPPED')
1031+ self.assertEqual('white', result)
1032+
1033+ def test_passed(self):
1034+ result = visualize.get_color_from_status('PASSED')
1035+ self.assertEqual('lime', result)
1036+
1037+ def test_failed(self):
1038+ result = visualize.get_color_from_status('FAILED')
1039+ self.assertEqual('red', result)
1040+
1041+ def test_regression(self):
1042+ result = visualize.get_color_from_status('REGRESSION')
1043+ self.assertEqual('red', result)
1044+
1045+ def test_fixed(self):
1046+ result = visualize.get_color_from_status('FIXED')
1047+ self.assertEqual('lime', result)
1048
1049=== added file 'qakit/vis/visualize.py'
1050--- qakit/vis/visualize.py 1970-01-01 00:00:00 +0000
1051+++ qakit/vis/visualize.py 2016-07-13 18:36:50 +0000
1052@@ -0,0 +1,294 @@
1053+#!/usr/bin/env python3
1054+# UEQA Vis
1055+# Copyright (C) 2014-2016 Canonical
1056+#
1057+# This program is free software: you can redistribute it and/or modify
1058+# it under the terms of the GNU General Public License as published by
1059+# the Free Software Foundation, either version 3 of the License, or
1060+# (at your option) any later version.
1061+#
1062+# This program is distributed in the hope that it will be useful,
1063+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1064+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1065+# GNU General Public License for more details.
1066+#
1067+# You should have received a copy of the GNU General Public License
1068+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1069+
1070+import argparse
1071+import json
1072+import math
1073+import statistics
1074+import sys
1075+
1076+import pymongo
1077+
1078+import jenkins_results
1079+
1080+
1081+def get_color_from_status(status):
1082+ """Return the color for the given cell.
1083+
1084+ PASSED (or FIXED) = green,
1085+ FAILED (or REGRESSED) = red,
1086+ SKIPPED = white
1087+
1088+ """
1089+ red = 'red'
1090+ white = 'white'
1091+ # lime happens to help with trend color-calculations
1092+ green = 'lime'
1093+ return {
1094+ 'SKIPPED': white,
1095+ 'PASSED': green,
1096+ 'FIXED': green,
1097+ 'REGRESSION': red,
1098+ 'FAILED': red,
1099+ }[status]
1100+
1101+
1102+def compute_color(pass_rate):
1103+ """Return a red-green color representing the given pass rate.
1104+
1105+ Use a sqrt curve to highlight the difference between 0.0 and
1106+ 0.01: invert the value, apply sqrt, then invert the value
1107+ again.
1108+
1109+ """
1110+ if pass_rate is None:
1111+ # white indicates "no result"
1112+ return 'rgb(255, 255, 255)'
1113+ p = abs(
1114+ 1.0 - math.sqrt(
1115+ abs(1.0 - float(pass_rate))
1116+ )
1117+ ) * 255.0
1118+ return 'rgb({}, {}, 0)'.format(
1119+ int(255 - p),
1120+ int(p),
1121+ )
1122+
1123+
1124+def status_to_float(status):
1125+ """Convert JUnit test status to float for averaging."""
1126+ if status == 'PASSED' or status == 'FIXED':
1127+ return 1.0
1128+ else:
1129+ return 0.0
1130+
1131+
1132+def compute_pass_rate_from_statuses(results):
1133+ """Compute pass rate from a list of JUnit test statuses."""
1134+ float_results = [status_to_float(result) for result in results]
1135+ try:
1136+ return statistics.mean(float_results)
1137+ except statistics.StatisticsError:
1138+ return None
1139+
1140+
1141+def compute_color_from_statuses(statuses):
1142+ """Compute a red-green color from a list of JUnit test statuses."""
1143+ return compute_color(compute_pass_rate_from_statuses(statuses))
1144+
1145+
1146+def get_build_pass_rate(dao, job_name, build_number):
1147+ """Calculate pass rate for a specified build in the given db."""
1148+ results = dao.get_build_test_results(job_name, build_number)
1149+ try:
1150+ statuses = [result['status'] for result in results]
1151+ return compute_pass_rate_from_statuses(statuses)
1152+ except TypeError:
1153+ # no result in db
1154+ return None
1155+
1156+
1157+def get_url_path_from_test_name(test_name):
1158+ """Return url path of Jenkins report for test.
1159+
1160+ Jenkins displays the result (incl. e.g. traceback) for a given
1161+ test at specific URL path under the build URL.
1162+
1163+ """
1164+ split_test_name = test_name.split('.')
1165+ if len(split_test_name) < 5:
1166+ return ''
1167+ try:
1168+ url_path = '/'.join(
1169+ ['.'.join(
1170+ split_test_name[:-2]),
1171+ split_test_name[-2],
1172+ split_test_name[-1]])
1173+ except IndexError:
1174+ url_path = ''
1175+ return url_path
1176+
1177+
1178+def get_test_link_for_build(build_url, test_name):
1179+ """Return a link to the test under the given Jenkins build."""
1180+ link = get_url_path_from_test_name(test_name)
1181+ return build_url + 'testReport/' + link
1182+
1183+
1184+def get_cell_data(dao, job_name, build_number, test_name):
1185+ """Return a dict representing the given test for rendering."""
1186+ cell = {'test_name': test_name,
1187+ 'job_name': job_name,
1188+ 'build_number': build_number}
1189+ test_result = dao.get_test_result(**cell)
1190+ build = dao.get_build(job_name, build_number)
1191+ if not test_result:
1192+ return dict({'status': None, 'color': None, 'link': None}, **cell)
1193+ cell['status'] = test_result['status']
1194+ try:
1195+ cell['color'] = get_color_from_status(test_result['status'])
1196+ except TypeError:
1197+ # no run for test, default to white
1198+ cell['color'] = 'rgb{255, 255, 255}'
1199+ cell['skipped'] = test_result['status'] == 'SKIPPED'
1200+ cell['link'] = get_test_link_for_build(build['url'], test_name)
1201+ return cell
1202+
1203+
1204+def get_trend_cell(dao, job_name, test_name, limit=10):
1205+ """Return a dict representing the pass rate trend."""
1206+ cell = {'test_name': test_name,
1207+ 'job_name': job_name}
1208+ test_results = dao.get_trailing_test_results(**cell)
1209+ cell['color'] = compute_color_from_statuses(
1210+ [test_result['status'] for test_result in test_results]
1211+ )
1212+ return cell
1213+
1214+
1215+def split_test_name(test_name):
1216+ """Return a tuple of (section, test_name) for given test name.
1217+
1218+ Typically tests are named as
1219+
1220+ foo.bar.baz.TestCaseClass.test_something
1221+
1222+ so return a "section" as 'foo.bar.baz', test_name as
1223+ TestCaseClass.test_something.
1224+
1225+ Non-conforming test names (e.g. 'gibberish') are granted their own
1226+ section names: ('gibberish', 'gibberish').
1227+
1228+ """
1229+ split_test_name = test_name.split('.')
1230+ if len(split_test_name) <= 2:
1231+ return (test_name, test_name)
1232+ test_name = '.'.join(split_test_name[-2:])
1233+ section_name = '.'.join(split_test_name[:-2])
1234+ return (section_name, test_name)
1235+
1236+
1237+def get_test_row(dao, job_name, test_name, build_numbers):
1238+ """Return a dict of test results for rendering."""
1239+ test_name_split = split_test_name(test_name)
1240+ row = {
1241+ 'section': test_name_split[0],
1242+ 'test_name': test_name_split[1],
1243+ }
1244+ row['trend'] = get_trend_cell(dao, job_name, test_name)
1245+ builds = []
1246+ for build_number in build_numbers:
1247+ builds.append(get_cell_data(
1248+ dao, job_name, build_number, test_name))
1249+ row['builds'] = builds
1250+ return row
1251+
1252+
1253+def get_report(
1254+ dao,
1255+ job_name):
1256+ """Generate a JSON representation from db data."""
1257+ build_numbers = dao.get_build_numbers(job_name)
1258+ pass_rates = []
1259+ for build_number in build_numbers:
1260+ pass_rates.append(get_build_pass_rate(dao, job_name, build_number))
1261+
1262+ report = {}
1263+ report['header'] = ['Section', 'Test Name', 'Trend']
1264+
1265+ # TODO: this may belong in the DataTables js, easier to compose here
1266+ def get_pass_rate_build_number_header(pass_rate, build_number):
1267+ try:
1268+ # express pass_rate as a percentage
1269+ return "{} ({:2.1f}%)".format(build_number, pass_rate*100)
1270+ except TypeError:
1271+ # when we encounter a None:
1272+ return "{}".format(build_number)
1273+
1274+ report['header'].extend([
1275+ get_pass_rate_build_number_header(pass_rate, build_number)
1276+ for build_number, pass_rate in
1277+ zip(build_numbers, pass_rates)
1278+ ])
1279+
1280+ test_names = dao.get_test_names(job_name)
1281+ data = []
1282+ for test_name in test_names:
1283+ data.append(
1284+ get_test_row(
1285+ dao,
1286+ job_name,
1287+ test_name,
1288+ build_numbers,
1289+ )
1290+ )
1291+ report['data'] = data
1292+ return report
1293+
1294+
1295+def write_json_report(
1296+ dao,
1297+ job_name,
1298+ output_filepath=None):
1299+ """Write a JSON report of test performance to the given filepath."""
1300+ json_report = get_report(dao, job_name)
1301+ with open(output_filepath, 'w') as f:
1302+ f.write(json.dumps(json_report, indent=4))
1303+
1304+
1305+def _parse_arguments():
1306+ parser = argparse.ArgumentParser(
1307+ "Write a JSON report of test performance.")
1308+ parser.add_argument(
1309+ help='Jenkins job to visualize.',
1310+ type=str,
1311+ dest='jenkins_job')
1312+ parser.add_argument(
1313+ '-o',
1314+ help="Output filepath for JSON report.",
1315+ type=str,
1316+ dest='output_filepath',
1317+ required=True)
1318+ parser.add_argument(
1319+ '--mongo-host',
1320+ help='MongoDB host IP.',
1321+ type=str,
1322+ default='127.0.0.1',
1323+ required=False)
1324+ parser.add_argument(
1325+ '--vis-db',
1326+ help='MongoDB database from which to generate report.',
1327+ type=str,
1328+ default='vis',
1329+ required=False)
1330+ return parser.parse_args()
1331+
1332+
1333+def main():
1334+ args = _parse_arguments()
1335+ conn = pymongo.MongoClient(host=args.mongo_host)
1336+ db = getattr(conn, args.vis_db)
1337+ dao = jenkins_results.JenkinsResults(db)
1338+ return write_json_report(
1339+ dao,
1340+ args.jenkins_job,
1341+ args.output_filepath,
1342+ )
1343+
1344+
1345+if __name__ == '__main__':
1346+ sys.exit(main())

Subscribers

People subscribed via source and target branches

to all changes: