Merge lp:~roadmr/checkbox/setup_and_suites_tests into lp:checkbox

[Daniel Manrique]

I added some tests that validate:

- All job files in jobs/ are also declared in setup.cfg (so they are included when building a package) and po/POTFILES.in (so they are considered when generating translations).
- All job files are listed in jobs/local.txt.in (so no job files are "left behind" and end up suiteless and lonely).

I get the feeling the tests are a bit brittle because they have a few hardcoded paths and make a few assumptions about which files should and should not be verified (the local.txt test arbitrarily removes a couple of files), so if you can think of a way to improve this it would be welcome. The hardcoded paths are not so much of a problem for checkbox, as this is mainly meant to be run at build time, but may be a problem for e.g. checkbox-oem.

[Brendan Donegan (brendan-donegan)]

Should this have [FEATURE] as it's not a bug? Also I see an unused import of 'os' in the test source file which you may want to remove.

review: Needs Information

[lp:~roadmr/checkbox/setup_and_suites_tests]

1683. By Daniel Manrique on 2012-09-21

Removed stray 'import os'

1684. By Daniel Manrique on 2012-09-21

  po/POTFILES.in and included in jobs/local.txt.in. (LP: #1052986)
  (LP: #1052986)

[Daniel Manrique (roadmr)]

Thanks, nice catches! I added the bug number that this code fixes (there's a bug for that!). I also removed the stray import os call.

review: Needs Resubmitting

[Brendan Donegan (brendan-donegan)]

Let's merge this and worry about how to make all of the tests refer to job files in oem-qa-checkbox later

review: Approve

=== added file 'checkbox/tests/setup_files.py'
--- checkbox/tests/setup_files.py	1970-01-01 00:00:00 +0000
+++ checkbox/tests/setup_files.py	2012-09-21 14:45:25 +0000
@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+#
+# This file is part of Checkbox.
+#
+# Copyright 2012 Canonical Ltd.
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+#
+import configparser
+import glob
+import re
+import unittest
+
+class SetupFiles(unittest.TestCase):
+
+    def test_job_files_in_setup_cfg(self):
+        #setup_cfg is a python config file
+        config = configparser.ConfigParser()
+        config.read('setup.cfg')
+        #config.get returns a string, so we have to eval it and then get the second
+        #element of the first element. The values are declared as an array of nested
+        #tuples.
+        rfc822deb_from_config = sorted(eval(config.get('build_i18n', 
+                                                       'rfc822deb_files'))[0][1])
+        existing_files = sorted(glob.glob('jobs/*.in'))
+        self.assertEqual(rfc822deb_from_config, existing_files)
+
+    def test_job_files_in_potfiles(self):
+        potfile_lines = [line.strip() for line in open('po/POTFILES.in','r')]
+        potfile_jobfiles = [file for file in potfile_lines if "rfc822deb" in file]
+        potfile_jobfiles = [line.split(']')[1].strip() 
+                            for line in potfile_jobfiles if ']' in line]
+
+        potfile_jobfiles = sorted(potfile_jobfiles)
+        existing_files = sorted(glob.glob('jobs/*.in'))
+        self.assertEqual(potfile_jobfiles, existing_files)
+
+    def test_job_files_in_local_txt(self):
+        local_lines = [line.strip() for line in open('jobs/local.txt.in','r')]
+        local_jobfiles = [file for file in local_lines if "$CHECKBOX_SHARE/jobs" in file]
+        local_jobfiles = [re.search('\$CHECKBOX_SHARE\/(?P<job_file>jobs\/(.+)\.txt)\?.*',
+                          file).group('job_file')+".in" for file in local_jobfiles]
+
+        local_jobfiles = sorted(local_jobfiles)
+        existing_files = sorted(glob.glob('jobs/*.in'))
+        #We need to remove local.txt.in from existing_files since it doesn't contain 
+        #itself, also removing resource.txt.in which does not need
+        #to be included in a suite.
+        existing_files.remove("jobs/local.txt.in")
+        existing_files.remove("jobs/resource.txt.in")
+        self.assertEqual(local_jobfiles, existing_files)
+

=== modified file 'debian/changelog'
--- debian/changelog	2012-09-21 13:33:06 +0000
+++ debian/changelog	2012-09-21 14:45:25 +0000
@@ -50,6 +50,10 @@
   * scripts/audio_test: made the default mode verbose, it now sends all
     output to stderr (but still exits a proper return value). Jobs using it 
     are updated to remove the now-unneeded -v parameter.
+  * Added tests to ensure all job files are declared in setup.cfg, 
+    po/POTFILES.in and included in jobs/local.txt.in. (LP: #1052986)
+  * setup.cfg, po/POTFILES.in, jobs/local.txt.in: Fixed so the tests pass.
+    (LP: #1052986)
 
   [Alberto Milone]
   * [FEATURE] scripts/window_test, jobs/graphics.txt.in: Added script
@@ -141,7 +145,7 @@
     the 'device' argument (bus type) and require at least one value
   * [FEATURE] scripts/removable_storage_watcher: add support for debugging
 
- -- Jeff Lane <[e-mail]>  Wed, 19 Sep 2012 14:20:15 -0400
+ -- Daniel Manrique <[e-mail]>  Wed, 19 Sep 2012 16:26:56 -0400
 
 checkbox (0.14.5) quantal; urgency=low
 

=== modified file 'jobs/local.txt.in'
--- jobs/local.txt.in	2012-08-28 13:35:03 +0000
+++ jobs/local.txt.in	2012-09-21 14:45:25 +0000
@@ -187,6 +187,13 @@
   shopt -s extglob
   cat $CHECKBOX_SHARE/jobs/panel_clock_test.txt?(.in)
 
+name: __panel_reboot__
+plugin: local
+_description: Panel Reboot Verification tests
+command:
+  shopt -s extglob
+  cat $CHECKBOX_SHARE/jobs/panel_reboot.txt?(.in)
+
 name: __pcmcia-pcix__
 plugin: local
 _description: PCMCIA/ExpressCard tests

=== modified file 'po/POTFILES.in'
--- po/POTFILES.in	2012-09-19 18:17:45 +0000
+++ po/POTFILES.in	2012-09-21 14:45:25 +0000
@@ -31,6 +31,7 @@
 [type: gettext/rfc822deb] jobs/panel_reboot.txt.in
 [type: gettext/rfc822deb] jobs/pcmcia-pcix.txt.in
 [type: gettext/rfc822deb] jobs/peripheral.txt.in
+[type: gettext/rfc822deb] jobs/piglit.txt.in
 [type: gettext/rfc822deb] jobs/power-management.txt.in
 [type: gettext/rfc822deb] jobs/rendercheck.txt.in
 [type: gettext/rfc822deb] jobs/resource.txt.in

=== modified file 'setup.cfg'
--- setup.cfg	2012-09-19 18:17:45 +0000
+++ setup.cfg	2012-09-21 14:45:25 +0000
@@ -37,6 +37,7 @@
          "jobs/panel_reboot.txt.in",
          "jobs/pcmcia-pcix.txt.in",
          "jobs/peripheral.txt.in",
+         "jobs/piglit.txt.in",
          "jobs/power-management.txt.in",
          "jobs/rendercheck.txt.in",
          "jobs/resource.txt.in",