Merge ~kissiel/hwcert-tools:add-beta-snap-build-step into hwcert-tools:master

[Maciej Kisielewski]

Snapcraft Testing: bring testing of beta branch and tidy up the scripts

[Taihsiang Ho (tai271828)]

We need this story https://trello.com/c/zzQ1cetn/552-when-i-work-on-tools-scripts-for-the-tpe-lab-i-want-them-to-be-organized-under-a-sub-project-like-checkbox-project-so-i-can-know
to avoid using branches as repos ; )

Unmerged commits

7391692... by Maciej Kisielewski on 2017-03-22

add checking of the beta branch of checkbox-snappy

Also:
  use lp:checkbox-snappy instead of the packaging branch from ~checkbox-dev
  make failing one branch not stop the whole script
  invoke checkbox-snappy from ~/ to avoid problems with calling from tmps

Signed-off-by: Maciej Kisielewski <email address hidden>

0f7c180... by Taihsiang Ho on 2017-03-15

Ask preseed.cgi to remove old grub loader.

a04a106... by Taihsiang Ho on 2017-02-24

crontab: staging is stable. mv them to production.

edbd241... by Taihsiang Ho on 2017-02-24

routine update of crontab

de81f15... by Taihsiang Ho on 2017-02-21

add one more step to update snapcraft.

34caf44... by Taihsiang Ho on 2017-02-21

fix typo

555bac8... by Taihsiang Ho on 2016-12-23

specify correct staging conditions

a7463a1... by Taihsiang Ho on 2016-12-23

use rtncode to check the checkbox-snappy build status.

fb28361... by Taihsiang Ho on 2016-12-23

add staging switch

a168be9... by Taihsiang Ho on 2016-12-23

use rtncode to check the checkbox-snappy build status.

diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..4d1bea3
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2016, taipei-cert-lab
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..f848402
--- /dev/null
+++ b/README.md
@@ -0,0 +1,2 @@
+# mini-ci-legacy
+legacy mini CI
diff --git a/src/cgi/plainbox-submit.cgi b/src/cgi/plainbox-submit.cgi
new file mode 100755
index 0000000..be6f5ad
--- /dev/null
+++ b/src/cgi/plainbox-submit.cgi
@@ -0,0 +1,422 @@
+#!/usr/bin/env python
+#
+# put this file in:
+# /srv/enablement/cgi-bin/
+#
+
+import os
+import posixpath
+import re
+import cgi
+import smtplib
+from email.mime.text import MIMEText
+from email.utils import formataddr
+from tempfile import NamedTemporaryFile
+from subprocess import Popen, PIPE
+from time import gmtime, strftime
+import pickle
+import shutil
+import sys
+from collections import defaultdict
+
+
+# delete the summary after it was sent?
+# WARNING: you have to remove the summary file
+# otherwise it will append and append the new results.
+DELETE_SUMMARY = True
+# 1 or True for debug mode on
+# False for off
+# if True, ce-qa mailing list will not get emails.
+DEBUG = False
+# send real email when debug mode is on
+# set True to get the email
+# only effective when using together with DEBUG
+DEBUG_EMAIL = True
+
+form = cgi.FieldStorage()
+if DEBUG:
+    SUMMARY_FILE_NAME = "/tmp/plainbox-ci-summary.debug"
+    SUMMARY_AUTOPILOT = "/tmp/minici-autopilot-summary.debug"
+    SUMMARY_SNAPCRAFT = "/tmp/minici-snapcraft-summary.debug"
+else:
+    SUMMARY_FILE_NAME = "/tmp/plainbox-ci-summary"
+    SUMMARY_AUTOPILOT = "/tmp/minici-autopilot-summary"
+    SUMMARY_SNAPCRAFT = "/tmp/minici-snapcraft-summary"
+PICKLE_FILE_NAME = "/tmp/plainbox-ci-old-status"
+INSTALLED_FILE_NAME = "/tmp/plainbox-ci-installed"
+
+print "Content-type: text/plain"
+print "\n"
+
+def rtn_code_to_status(code):
+    if code == '0' or code == 0:
+        return "PASS"
+    else:
+        return "FAIL"
+
+def get_subject(message):
+    subject = ""
+    if DEBUG: subject += "[TEST AND PLEASE IGNORE]"
+    #subject += if_missing_run(message)
+    subject += if_worse(message)
+    subject += if_checkbox_crash(message)
+    subject += if_fail_to_run(message)
+    return subject
+
+def get_subject_build_snap(message, status):
+    subject = ""
+    if DEBUG: subject += "[TEST AND PLEASE IGNORE]"
+    subject += '[' + rtn_code_to_status(status) + ']'
+    return subject
+
+def if_missing_run(message):
+    # TODO:
+    # for fail to run jobs
+    # the string should be the same
+    subject = ""
+    list_cid = ["201302-12681",
+                "201212-12224",
+                "201212-12226"] 
+    list_distribute = ["Ubuntu 12.04.5 LTS desktop",
+                       "Ubuntu 14.04.1 LTS desktop",
+                       "Ubuntu Vivid Vervet (development branch) desktop",
+                       "Ubuntu 12.04.5 LTS server",
+                       "Ubuntu 14.04.1 LTS server",
+                       "Ubuntu Vivid Vervet (development branch) server"]
+    for i in list_cid + list_distribute:
+        if i not in message:
+             subject = "[run missing]"
+    return subject
+
+def if_worse(message):
+    subject = ""
+    if "Overall trend:  WORSE" in message:
+        subject = "[run getting worse]"
+    return subject
+
+def if_checkbox_crash(message):
+    # TODO
+    subject = ""
+    return subject
+
+def if_fail_to_run(message):
+    # TODO: will warn after failing to run in the recent 3 loops.
+    subject = ""
+    if "pass: 0, fail: 0" in message:
+        subject = "[run failing]"
+    return subject
+
+
+def create_data_obj(lines):
+    """
+    create a data object
+
+    return a data object to host the data of mini-CI reports.
+    This report entries are separated by an empty lines for each cluster.
+
+    :param lines: the lines given by the opened file readlines()
+    :return mini-CI report data object
+    """
+    obj = []
+    obj_sub = []
+    for line in lines:
+        if line == "\n":
+            obj.append(obj_sub)
+            obj_sub = []
+        else:
+            obj_sub.append(line)
+
+    return obj
+
+
+def dump_to_string(minici_rpt_obj):
+    """
+    dump mini-CI report data object to a string
+
+    dump the data as a string so MIMEText could use it
+    to send an email.
+
+    :param minici_rpt_obj: mini-CI report data object
+    :return string
+    """
+    rtn_string = ""
+    for entry in minici_rpt_obj:
+        for line in entry:
+            rtn_string += line
+        rtn_string += "\n"
+
+    return rtn_string
+
+################  begin of the mini-CI mailing flow ##############
+if 'emailit' in form:
+    to_addr = []
+    try:
+        summary_f = open(SUMMARY_FILE_NAME, "r")
+        mini_ci_rpt_obj = create_data_obj(summary_f.readlines())
+        # sort the report so it is more readable.
+        mini_ci_rpt_obj.sort()
+        msg = MIMEText(dump_to_string(mini_ci_rpt_obj))
+        summary_f.close()
+        if DELETE_SUMMARY:
+            os.unlink(summary_f.name)
+    except (IOError, OSError):
+        print("Error trying to open the file %s" % SUMMARY_FILE_NAME)
+        print(sys.exc_info())
+        sys.exit(1)
+    msg_subject = get_subject(msg.as_string()) + " Summary of Plainbox mini-CI test runs"
+    if 'dailyimage' in form:
+        msg_subject =  msg_subject + ' (daily build)'
+    if 'amdgpu' in form:
+        msg_subject =  msg_subject + ' (AMD GPU on Xenial)'
+    msg['Subject'] = msg_subject
+    msg['From'] = formataddr(('mini-CI', '[e-mail]'))
+    msg['reply-to'] = '[e-mail]'
+    if DEBUG:
+        msg['To'] = '[e-mail]'
+        to_addr.append(msg['To'])
+        # add secondary email for testing:
+        to_addr.append("[e-mail]")
+        print(msg['Subject'])
+        if not DEBUG_EMAIL:
+            sys.exit(0)
+    else:
+        #msg['To'] = '[e-mail]'
+        #msg['To'] = '[e-mail]'
+        msg['To'] = '[e-mail]'
+        to_addr.append(msg['To'])
+        to_addr.append("[e-mail]")
+    s = smtplib.SMTP('localhost')
+    s.sendmail(msg['From'], to_addr, msg.as_string())
+    s.quit()
+elif 'plainbox_output' in form and 'subject' in form:
+    pickle_file = None
+    previous_status = {}
+    try:
+        pickle_file = open(PICKLE_FILE_NAME, "r")
+        previous_status = pickle.load(pickle_file)
+    except:
+        pass
+    finally:
+        if pickle_file:
+            pickle_file.close()
+
+    preamble = "This is output of the plainbox mini-ci run in Taipei\n\n"
+    text = preamble + form['plainbox_output'].value
+    f = NamedTemporaryFile()
+    f.write(text)
+    f.flush()
+    #Curl it here and get the URL
+    command=('curl -v -H "Expect:" --data-urlencode '
+             '"content@%s" -d "poster=[e-mail]" '
+             '-d "syntax=text" http://paste.ubuntu.com') % f.name
+    process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
+    (output, sterr) = process.communicate()
+    regex = re.compile(r"Location: (.+)")
+    match = regex.search(sterr)
+    if match:
+        url = match.group(1)
+    else:
+        url = "No url, I probably had a problem pastebinning the log :("
+    f.close() # Auto deletes after closing
+
+    status_tuples = re.findall(r"- (.+): (fail|pass)", text)
+    status_dict = {}
+    for (test, status) in status_tuples:
+         status_dict[test] = status
+    pass_count = len([s for s in status_dict.values() if s == 'pass'])
+    fail_count = len([s for s in status_dict.values() if s == 'fail'])
+    #Write to summary file
+    sum_f = open(SUMMARY_FILE_NAME, "a")
+    sum_f.write("%s at %s: results here:  %s\n" % 
+                (form['subject'].value,
+                 strftime("%Y-%m-%d %H:%M UTC", gmtime()),
+                 url, ))
+    sum_f.write("pass: %s, fail: %s" % (pass_count, fail_count))
+    # Figure out a trend
+    old_status_dict = previous_status.get(form['subject'].value)
+    if old_status_dict:
+        # We need to report:
+        # Tests that transitioned from pass to fail
+        # Tests that transitioned from fail to pass
+        # Tests that appeared (added)
+        # Tests that disappeared (removed)
+        # Overall trend (HOW TO DEFINE THIS?)
+        trends = defaultdict(list)
+        for (test, current_status) in status_dict.items():
+            old_status = old_status_dict.get(test)
+            if old_status:
+                # Existing test, trend it
+                if old_status == "pass" and current_status == "fail":
+                     trends['pass_to_fail'].append(test)
+                if old_status == "fail" and current_status == "pass":
+                     trends['fail_to_pass'].append(test)
+            else:
+                 trends['added'].append(test)
+        overall = 0             
+        if 'pass_to_fail' in trends:
+            overall = overall - len(trends['pass_to_fail'])
+            sum_f.write("\n%s Tests got worse (%s)" % 
+                        (len(trends['pass_to_fail']),
+                        ", ".join(trends['pass_to_fail'])))
+        if 'fail_to_pass' in trends:
+            overall = overall + len(trends['fail_to_pass'])
+            sum_f.write("\n%s Tests got better (%s)" % 
+                            (len(trends['fail_to_pass']),
+                            ", ".join(trends['fail_to_pass'])))
+        if (overall < 0): overall_sign = " WORSE "
+        elif (overall > 0): overall_sign = " BETTER "
+        else: overall_sign = " STABLE "
+        sum_f.write("\nOverall trend: %s" % overall_sign)
+    sum_f.write("\n\n")
+    sum_f.close()
+
+    #Now pickle the fail/pass counts for later trending
+    previous_status[form['subject'].value] = status_dict 
+    pickle_file = None
+    try:
+        pickle_file = open(PICKLE_FILE_NAME, "w")
+        pickle.dump(previous_status, pickle_file)
+    except:
+        pass
+    finally:
+        if pickle_file:
+             pickle_file.close()
+elif 'mini_ci_notification_installed' in form:
+    installed_file = None
+    try:
+        installed_file = open(INSTALLED_FILE_NAME, "a")
+        installed_file.write("%s at %s\n" %
+                            (form['mini_ci_notification_installed'].value,
+                            strftime("%Y-%m-%d %H:%M UTC", gmtime())))
+    except:
+        pass
+    finally:
+        if installed_file:
+            installed_file.close()
+elif 'mini_ci_notification_installed_reset' in form:
+    installed_file = None
+    try:
+        installed_file = open(INSTALLED_FILE_NAME, "w")
+        installed_file.write("%s at %s\n" %
+                            (form['mini_ci_notification_installed_reset'].value,
+                            strftime("%Y-%m-%d %H:%M UTC", gmtime())))
+    except:
+        pass
+    finally:
+        if installed_file:
+            installed_file.close()
+elif 'rm_summary_report' in form:
+    # usage:
+    # curl -F rm_summary_report="yes" http://localhost/cgi-bin/plainbox-submit.cgi
+    try:
+        os.unlink(SUMMARY_FILE_NAME)
+    except:
+        print "something is wrong when removing %s" % SUMMARY_FILE_NAME
+elif 'create_test_file' in form:
+    # usage:
+    # curl -F create_test_file="yes" http://localhost/cgi-bin/plainbox-submit.cgi
+    test_file = open("/tmp/plainbox-ci-test", "w")
+    test_file.close()
+elif 'email_mini_ci_autopilot' in form:
+    try:
+        summary_f = open(SUMMARY_AUTOPILOT, "r")
+        msg = MIMEText(summary_f.read())
+        summary_f.close()
+        os.unlink(summary_f.name)
+    except (IOError, OSError):
+        print("Error trying to open the file")
+        print(sys.exc_info())
+        sys.exit(1)
+    msg['Subject'] = get_subject(msg.as_string()) + " Summary of autopilot mini-CI test runs"
+    msg['From'] = formataddr(('autopilot mini-CI', '[e-mail]'))
+    msg['reply-to'] = '[e-mail]'
+    if DEBUG:
+        msg['To'] = '[e-mail]'
+        print(msg['Subject'])
+        if not DEBUG_EMAIL:
+            sys.exit(0)
+    else:
+        #msg['To'] = '[e-mail]'
+        #msg['To'] = '[e-mail]'
+        msg['To'] = '[e-mail]'
+    s = smtplib.SMTP('localhost')
+    s.sendmail(msg['From'],[msg['To']], msg.as_string())
+    s.quit()
+elif 'autopilot_output' in form and 'autopilot_subject' in form:
+    preamble = "This is output of the auto mini-ci run in Taipei\n\n"
+    text = preamble + form['autopilot_output'].value
+    if DEBUG:
+        print text
+    f = NamedTemporaryFile()
+    f.write(text)
+    f.flush()
+    #Curl it here and get the URL
+    command=('curl -v -H "Expect:" --data-urlencode '
+             '"content@%s" -d "poster=[e-mail]" '
+             '-d "syntax=text" http://paste.ubuntu.com') % f.name
+    process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
+    (output, sterr) = process.communicate()
+    regex = re.compile(r"Location: (.+)")
+    match = regex.search(sterr)
+    if match:
+        url = match.group(1)
+        if DEBUG:
+            print url
+    else:
+        url = "No url, I probably had a problem pastebinning the log :("
+    f.close() # Auto deletes after closing
+
+    #Write to summary file
+    sum_f = open(SUMMARY_AUTOPILOT, "a")
+    sum_f.write("%s at %s: results here:  %s\n" % 
+                (form['autopilot_subject'].value,
+                 strftime("%Y-%m-%d %H:%M UTC", gmtime()),
+                 url, ))
+    overall_result = text.splitlines()[-3:-1]
+    sum_f.write("Overall result: %s" % str(overall_result))
+elif 'email_mini_ci_snapcraft' in form and 'snapcraft_status' in form:
+    try:
+        summary_f = open(SUMMARY_SNAPCRAFT, "r")
+        msg = MIMEText(summary_f.read())
+        summary_f.close()
+        os.unlink(summary_f.name)
+    except (IOError, OSError):
+        print("Error trying to open the file")
+        print(sys.exc_info())
+        sys.exit(1)
+    msg['Subject'] = get_subject_build_snap(msg.as_string(), form['snapcraft_status'].value) + "[mini-CI] snapcraft build checkbox-snappy.snap"
+    msg['From'] = formataddr(('Taipei Lab mini-CI', '[e-mail]'))
+    msg['reply-to'] = '[e-mail]'
+    if DEBUG:
+        msg['To'] = '[e-mail]'
+        print(msg['Subject'])
+        if not DEBUG_EMAIL:
+            sys.exit(0)
+    else:
+        #msg['To'] = '[e-mail]'
+        #msg['To'] = '[e-mail]'
+        msg['To'] = '[e-mail]'
+    s = smtplib.SMTP('localhost')
+    s.sendmail(msg['From'],[msg['To']], msg.as_string())
+    if DEBUG:
+        print "emamil sent."
+    s.quit()
+elif 'snapcraft_output' in form and 'snapcraft_subject' in form and 'snapcraft_status' in form:
+    preamble = "=========================\n"
+    preamble += "mini-CI Run in Taipei Lab\n"
+    preamble += "=========================\n"
+    preamble += "SUBMITTING TIME:\n"
+    preamble += strftime("%Y-%m-%d %H:%M UTC", gmtime()) + "\n"
+    preamble += "=========================\n"
+    preamble += form['snapcraft_subject'].value + "\n"
+    preamble += "=========================\n"
+    preamble += "OVERALL STATUS: " + rtn_code_to_status(form['snapcraft_status'].value) + "\n"
+    preamble += "=========================\n"
+    text = preamble + form['snapcraft_output'].value
+    if DEBUG:
+        print text
+    sum_f = open(SUMMARY_SNAPCRAFT, "w")
+    sum_f.write(text + "\n")
+    sum_f.close()
+else:
+    print("unknown curl data")
diff --git a/src/cgi/preseed.cgi b/src/cgi/preseed.cgi
new file mode 100755
index 0000000..51ed809
--- /dev/null
+++ b/src/cgi/preseed.cgi
@@ -0,0 +1,56 @@
+#!/usr/bin/env python
+
+import os
+import posixpath
+import logging
+FORMATTER = '[%(asctime)s] {%(lineno)d} %(message)s'
+logging.basicConfig(filename='/tmp/preseed-cgi.log',level=logging.INFO, format=FORMATTER)
+
+# Required environment variables
+SERVER_ADDR = os.environ["SERVER_ADDR"]
+REMOTE_ADDR = os.environ["REMOTE_ADDR"]
+
+# Optional environment variables
+CHECKBOX_PXELINUX_CFG = os.environ.get("CHECKBOX_PXELINUX_CFG",
+    "/srv/enablement/tftpboot/pxelinux.cfg")
+CHECKBOX_GRUB_CFG = os.environ.get("CHECKBOX_GRUB_CFG",
+    "/srv/enablement/tftpboot/grub")
+CHECKBOX_PRESEEDS = os.environ.get("CHECKBOX_PRESEEDS",
+    "preseeds")
+
+# Convert the remote address to hex format
+hex_address = "".join("%02X" % int(x) for x in REMOTE_ADDR.split("."))
+
+logging.info(SERVER_ADDR)
+logging.info(CHECKBOX_PRESEEDS)
+logging.info(REMOTE_ADDR)
+logging.info(hex_address)
+
+# Remove pxelinux configuration file for the host
+# 1. for legacy boot
+pxelinux_filename = posixpath.join(CHECKBOX_PXELINUX_CFG, hex_address)
+old_pxelinux_filename = pxelinux_filename + ".old"
+if posixpath.exists(pxelinux_filename):
+    logging.info('boot over legacy.')
+    if posixpath.exists(old_pxelinux_filename):
+        os.unlink(old_pxelinux_filename)
+        logging.info('done unlink.')
+    os.rename(pxelinux_filename, old_pxelinux_filename)
+    logging.info('done rename to be old.')
+# 2. for secure boot (using grub.cfg instead of hex_address preseed)
+# this naming style is defined by checkbox-satellite
+grubcfg_filename = posixpath.join(CHECKBOX_GRUB_CFG, REMOTE_ADDR + ".grub.cfg")
+old_grubcfg_filename = grubcfg_filename + ".old"
+if posixpath.exists(grubcfg_filename):
+    logging.info('boot over grub.')
+    if posixpath.exists(old_grubcfg_filename):
+        logging.info(old_grubcfg_filename)
+        os.unlink(old_grubcfg_filename)
+        logging.info('done unlink')
+    os.rename(grubcfg_filename, old_grubcfg_filename)
+    logging.info('done rename to be old.')
+
+# Redirect to the preseed file
+print "Location: http://%s/%s/%s" % (SERVER_ADDR, CHECKBOX_PRESEEDS, hex_address)
+print ""
+
diff --git a/src/cgi/unit-test.sh b/src/cgi/unit-test.sh
new file mode 100755
index 0000000..31db330
--- /dev/null
+++ b/src/cgi/unit-test.sh
@@ -0,0 +1,14 @@
+test_log="$HOME/mini-ci-log/for-dev"
+rm ${test_log}/plainbox-ci-summary
+ln -s ${test_log}/plainbox-ci-summary-01-ok ${test_log}/plainbox-ci-summary
+./plainbox-submit.cgi
+
+rm ${test_log}/plainbox-ci-summary
+ln -s ${test_log}/plainbox-ci-summary-02-fail-to-run ${test_log}/plainbox-ci-summary
+./plainbox-submit.cgi
+
+rm ${test_log}/plainbox-ci-summary
+ln -s ${test_log}/plainbox-ci-summary-03-worse-better ${test_log}/plainbox-ci-summary
+./plainbox-submit.cgi
+
+rm ${test_log}/plainbox-ci-summary
diff --git a/src/crontab/autopilot/run-autopilot b/src/crontab/autopilot/run-autopilot
new file mode 100755
index 0000000..9f671ff
--- /dev/null
+++ b/src/crontab/autopilot/run-autopilot
@@ -0,0 +1,93 @@
+#!/bin/bash
+#
+# $0          $1               $2      $3     $4  $5      $6
+# run-mini-ci launch-reset     12.04.5 server CID
+# run-mini-ci installed-reboot 12.04.5 server CID
+# run-mini-ci installed-launch 12.04.5 server CID
+# run-mini-ci launch-done      12.04.5 server CID 12.04.4 desktop
+#
+#set -x
+#
+# alias
+# lt_minici 12.04.5 server CID
+
+if [ -z "$STAGING" ]; then
+    export CHECKBOX_SRC=${CHECKBOX_SRC:-${HOME}/src}
+else
+    export CHECKBOX_SRC=${CHECKBOX_SRC:-${HOME}/src.staging}
+fi
+
+lt_minici="$CHECKBOX_SRC/checkbox-message/trunk/bin/launch_testrun --autopilot --ppa=dev"
+dir_checkbox_message_outlet="$CHECKBOX_SRC/checkbox-message/trunk/outlets"
+command_manage_power="$CHECKBOX_SRC/checkbox-satellite/trunk/scripts/manage_power"
+pdu_setting_off='--retry=6 --sleep=0 --timeout=30 off apc://apc:[e-mail]'
+pdu_setting_on='--retry=6 --sleep=0 --timeout=30 on apc://apc:[e-mail]'
+cgi_url="http://localhost/cgi-bin/plainbox-submit.cgi"
+subject="CheckBox NG CI testing run for"
+message="after several reboots, it seems not to begin to run or completed the run."
+
+function reset_previous_results {
+    curl -F mini_ci_notification_installed_reset="New mini ci loop cycle" $cgi_url
+}
+
+function launch_mini_ci {
+    case $2 in
+        "desktop")
+        echo "$lt_minici $1 $2 $3"
+        $lt_minici $1 $2 $3 > /tmp/mini-ci-${1}-${2}-${3}-autopilot-`date +%y%m%d` 2>&1
+        ;;
+        "server")
+        echo "$lt_minici $1 $2 $3"
+        $lt_minici $1 $2 $3 > /tmp/mini-ci-${1}-${2}-${3}-`date +%y%m%d` 2>&1
+        esac
+}
+
+function pdu_off_on {
+    local outlet=`grep outlet ${dir_checkbox_message_outlet}/$1  | awk -F: {'print $2'} | sed -e 's/^ *//'`
+    $command_manage_power $pdu_setting_off "$outlet"
+    # for some machines especially laptops with big or stable adapters
+    # you have to off the power for longer time to wait the voltage is exactly down.
+    sleep 60
+    $command_manage_power $pdu_setting_on "$outlet"
+}
+
+case $1 in
+    "launch-reset")
+        # reset the installed status summary and launch the machine
+        # usually used in the beginning of a mini ci cycle
+        reset_previous_results
+        launch_mini_ci $2 $3 $4
+    ;; 
+    "launch")
+        # launch <point-release> <desktop-server> CID
+        launch_mini_ci $2 $3 $4
+    ;;  
+    "installed-reboot")
+        # check the installed status of <desktop-server> CID
+        # if no installed record is found,
+        # power off and on the machine
+        [ `grep -i $2 /tmp/plainbox-ci-installed | grep $3 | grep $4 | wc -l` -eq 0 ] && pdu_off_on $4
+    ;;  
+    "installed-launch")
+        # check the installed status of <desktop-server> CID
+        # if no installed record is found,
+        # relaunch the machine
+        [ `grep -i $2 /tmp/plainbox-ci-installed | grep $3 | grep $4 | wc -l` -eq 0 ] && launch_mini_ci $2 $3 $4
+    ;;  
+    "done-desktop")
+        # to see whether the previous testing for desktop completed or not
+        # if not, mark it as failure run.
+        done_number=`grep $5 /tmp/plainbox-ci-summary  | grep $6 | grep $4 | grep results | grep -v FAILED | wc -l`
+        [ $done_number -eq 0 ] && curl -F subject="${subject} Ubuntu $5 $6 on $4 FAILED to run." -F plainbox_output="$message" $cgi_url
+    ;;  
+    "done-server")
+        # to see whether the previous testing for server completed or not
+        # if not, mark it as failure run.
+        # a workaround for server release that not using etho0 as its ethernet device,
+        # so in plainbox-ci-summary the CID would not be shown.
+        # Skip to grep CID otherwise it will always regard the result as FAILED.
+        # Thus this command should be used specifically to server log.
+        #done_number=`grep $5 /tmp/plainbox-ci-summary  | grep $6 | grep $4 | grep results | grep -v FAILED | wc -l`
+        done_number=`grep $5 /tmp/plainbox-ci-summary  | grep $6 | grep $6 | grep results | grep -v FAILED | wc -l`
+        [ $done_number -eq 0 ] && curl -F subject="${subject} Ubuntu $5 $6 on $4 FAILED to run." -F plainbox_output="$message" $cgi_url
+esac
diff --git a/src/crontab/autopilot/run-autopilot-staging b/src/crontab/autopilot/run-autopilot-staging
new file mode 100755
index 0000000..f2ad17a
--- /dev/null
+++ b/src/crontab/autopilot/run-autopilot-staging
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# if STAGING is not empty, $HOME/src.staging will be used instead of $HOME/src
+export STAGING='.src'
+dir=`dirname $0`
+$dir/run-autopilot "$@"
+
diff --git a/src/crontab/backup_mini_ci_result b/src/crontab/backup_mini_ci_result
new file mode 100755
index 0000000..c50920c
--- /dev/null
+++ b/src/crontab/backup_mini_ci_result
@@ -0,0 +1,10 @@
+#set -x
+for var in today yesterday
+do
+    var_date_h=`date --date=$var +%y%m%d-%H`
+    var_date=`date --date=$var +%y%m%d`
+    dir_dest=/home/$USER/mini-ci-log/$var_date_h
+    mkdir -p $dir_dest
+    cp -a /tmp/plainbox-ci-* /tmp/mini-ci-*-$var_date $dir_dest
+    rm /tmp/mini-ci-*-$var_date
+done
diff --git a/src/crontab/checkbox/run-mini-ci b/src/crontab/checkbox/run-mini-ci
new file mode 100755
index 0000000..ee0a366
--- /dev/null
+++ b/src/crontab/checkbox/run-mini-ci
@@ -0,0 +1,146 @@
+#!/bin/bash
+#
+# $0          $1               $2      $3     $4  $5      $6        $7
+# run-mini-ci launch-reset     12.04.5 server CID
+# run-mini-ci installed-reboot 12.04.5 server CID
+# run-mini-ci installed-launch 12.04.5 server CID
+# run-mini-ci launch-done      12.04.5 server CID 12.04.4 desktop
+# run-mini-ci launch-done      12.04.5 server CID 12.04.4 desktop   -p
+#
+# $4 $5 $6 are key words used to can the plainbox summary report.
+#
+#set -x
+#
+# alias
+# lt_minici 12.04.5 server CID
+
+if [ -z "$STAGING" ]; then
+    export CHECKBOX_SRC=${CHECKBOX_SRC:-${HOME}/src}
+else
+    export CHECKBOX_SRC=${CHECKBOX_SRC:-${HOME}/src.staging}
+fi
+
+lt_minici="$CHECKBOX_SRC/checkbox-message/trunk/bin/launch_testrun --mini-ci --ppa=dev --plainbox"
+dir_checkbox_message_outlet="$CHECKBOX_SRC/checkbox-message/trunk/outlets"
+command_manage_power="$CHECKBOX_SRC/checkbox-satellite/trunk/scripts/manage_power"
+pdu_setting_off='--retry=6 --sleep=0 --timeout=30 off apc://apc:[e-mail]'
+pdu_setting_on='--retry=6 --sleep=0 --timeout=30 on apc://apc:[e-mail]'
+cgi_url="http://localhost/cgi-bin/plainbox-submit.cgi"
+subject="CheckBox NG CI testing run for"
+message="after several reboots, it seems not to begin to run or completed the run."
+# if you want to debug $cgi_url
+# make sure this file name is set up accordingly.
+if [ "$MINICIDEBUG" == "" ]; then
+    summary_report="/tmp/plainbox-ci-summary"
+else
+    summary_report="/tmp/plainbox-ci-summary.debug"
+fi
+
+
+function reset_previous_results {
+    # clean old installed-flag-file and create a new one
+    curl -F mini_ci_notification_installed_reset="New mini ci loop cycle" $cgi_url
+    # remove the existing summary report
+    curl -F rm_summary_report="yes" $cgi_url
+}
+
+function launch_mini_ci {
+    case $2 in
+        "desktop")
+        echo "$lt_minici -w autotesting.whitelist $1 $2 $3 $4"
+        $lt_minici -w autotesting.whitelist $1 $2 $3 $4 > /tmp/mini-ci-${1}-${2}-${3}-`date +%y%m%d` 2>&1
+        ;;
+        "server")
+        echo "$lt_minici $1 $2 $3 $4"
+        $lt_minici $1 $2 $3 $4 > /tmp/mini-ci-${1}-${2}-${3}-`date +%y%m%d` 2>&1
+        esac
+}
+
+
+# tweak sleep time manually
+# so we could make the on-off switch longer
+# and the power supply of machines
+# could fully discharge and then wake up on power resume.
+#
+# pdu_off_on <cid>
+#
+function pdu_off_on {
+    local outlet=`grep outlet ${dir_checkbox_message_outlet}/$1  | awk -F: {'print $2'} | sed -e 's/^ *//'`
+    $command_manage_power $pdu_setting_off "$outlet"
+    # for some machines especially laptops with big or stable adapters
+    # you have to off the power for longer time to wait the voltage is exactly down.
+    sleep 60
+    $command_manage_power $pdu_setting_on "$outlet"
+}
+
+
+function check_summary_rpt_name {
+    number_of_rpt=`ls {/tmp/plainbox-ci-summary,/tmp/plainbox-ci-summary.debug} | grep -v "No such file or directory" | wc -l`
+}
+
+case $1 in
+    "launch-reset")
+        # reset the installed status summary and launch the machine
+        # usually used in the beginning of a mini ci cycle
+        reset_previous_results
+        launch_mini_ci $2 $3 $4
+    ;; 
+    "launch")
+        # launch <point-release> <desktop-server> CID
+        launch_mini_ci $2 $3 $4 $7
+    ;;  
+    "installed-reboot")
+        # seems no need to use this because we add kernel parameter no_hung to reboot the system.
+        # 
+        # check the installed status of <desktop-server> CID
+        # if no installed record is found,
+        # power off and on the machine
+        #
+        # $0          $1               $2      $3     $4  $5      $6
+        # run-mini-ci launch-reset     12.04.5 server CID
+        #
+        # debug command:
+        # grep -i 12.04.5 ./plainbox-ci-installed | grep server | grep 201209-11625
+        #
+        # TODO: for server, the hostname is not shown as CID so check CID will always fail.
+        # refer to the option "done-server"
+        [ `grep -i $2 /tmp/plainbox-ci-installed | grep $3 | grep $4 | wc -l` -eq 0 ] && pdu_off_on $4
+    ;;  
+    "installed-launch")
+        # seems no need to use this because we add kernel parameter no_hung to reboot the system.
+        #
+        # check the installed status of <desktop-server> CID
+        # if no installed record is found,
+        # relaunch the machine
+        [ `grep -i $2 /tmp/plainbox-ci-installed | grep $3 | grep $4 | wc -l` -eq 0 ] && launch_mini_ci $2 $3 $4
+    ;;  
+    "done-desktop")
+        # to see whether the previous testing for desktop completed or not
+        # if not, mark it as failure run.
+        #
+        # $0          $1               $2      $3     $4  $5      $6
+        # run-mini-ci launch-done      12.04.5 server CID 12.04.4 desktop
+        # grep 12.04.5 plainbox-ci-summary | grep desktop | grep 201302-12843 | grep results | grep -v FAILED | wc -l
+        done_number=`grep $5 $summary_report  | grep $6 | grep $4 | grep results | grep -v FAILED | wc -l`
+        [ $done_number -eq 0 ] && curl -F subject="${subject} Ubuntu $5 $6 on $4 FAILED to run." -F plainbox_output="$message" $cgi_url
+    ;;  
+    "done-server")
+        # to see whether the previous testing for server completed or not
+        # if not, mark it as failure run.
+        # a workaround for server release that not using eth0 as its ethernet device,
+        # so in plainbox-ci-summary the CID would not be shown.
+        # Skip to grep CID otherwise it will always regard the result as FAILED.
+        # Thus this command should be used specifically to server log.
+        #done_number=`grep $5 /tmp/plainbox-ci-summary  | grep $6 | grep $4 | grep results | grep -v FAILED | wc -l`
+        done_number=`grep $5 $summary_report  | grep $6 | grep $6 | grep results | grep -v FAILED | wc -l`
+        [ $done_number -eq 0 ] && curl -F subject="${subject} Ubuntu $5 $6 on $4 FAILED to run." -F plainbox_output="$message" $cgi_url
+    ;;
+    "dry")
+        # for testing
+        #
+        check_summary_rpt_name
+        # one and only one summary report could exist
+        [ $number_of_rpt -eq 1 ] || echo "plainbox-ci-summary name setting is incorrect."
+        echo $summary_report
+        echo $MINICIDEBUG
+esac
diff --git a/src/crontab/checkbox/run-mini-ci-staging b/src/crontab/checkbox/run-mini-ci-staging
new file mode 100755
index 0000000..5cde4ff
--- /dev/null
+++ b/src/crontab/checkbox/run-mini-ci-staging
@@ -0,0 +1,7 @@
+#!/bin/bash
+#
+# if STAGING is not empty, $HOME/src.staging will be used instead of $HOME/src
+export STAGING='.src'
+dir=`dirname $0`
+$dir/run-mini-ci "$@"
+
diff --git a/src/crontab/crontab b/src/crontab/crontab
new file mode 100644
index 0000000..cebad1f
--- /dev/null
+++ b/src/crontab/crontab
@@ -0,0 +1,233 @@
+# Edit this file to introduce tasks to be run by cron.
+# 
+# Each task to run has to be defined through a single line
+# indicating with different fields when the task will be run
+# and what command to run for the task
+# 
+# To define the time you can provide concrete values for
+# minute (m), hour (h), day of month (dom), month (mon),
+# and day of week (dow) or use '*' in these fields (for 'any').# 
+# Notice that tasks will be started based on the cron's system
+# daemon's notion of time and timezones.
+# 
+# Output of the crontab jobs (including errors) is sent through
+# email to the user the crontab file belongs to (unless redirected).
+# 
+# For example, you can run a backup of all your user accounts
+# at 5 a.m every week with:
+# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
+# 
+# For more information see the manual pages of crontab(5) and cron(8)
+# 
+# m h  dom mon dow   command
+
+# yantok time + 8hr = Taipei lab time
+
+# crontab action notifications
+MAILTO="[e-mail]"
+
+################ MINI CI LOOP SETTING #############################
+# for launch_testrun
+# please use your own yantok username, replace following "certification"s to your login
+USER=taihsiang
+# setting PATH is necessary to use
+# sudo /usr/bin/service isc-dhcp-server restart
+# (requested by checkbox-satellite)
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
+# alias
+lt=/home/taihsiang/src/checkbox-message/trunk/bin/launch_testrun
+lt_staging=/home/taihsiang/src.staging/checkbox-message/trunk/bin/launch_testrun
+run_mini_ci=/srv/enablement/www/mini-ci-legacy/src/crontab/checkbox/run-mini-ci
+run_mini_ci_staging=/srv/enablement/www/mini-ci-legacy/src/crontab/checkbox/run-mini-ci-staging
+run_autopilot=/srv/enablement/www/mini-ci-legacy/src/crontab/autopilot/run-autopilot
+run_autopilot_staging=/srv/enablement/www/mini-ci-legacy/src/crontab/autopilot/run-autopilot-staging
+run_snapcraft=/srv/enablement/www/mini-ci-legacy/src/crontab/snapcraft/run-snapcraft
+run_snapcraft_staging=/srv/enablement/www/mini-ci-legacy/src/crontab/snapcraft/run-snapcraft-staging
+backup_mini_ci_result=/srv/enablement/www/mini-ci-legacy/src/crontab/backup_mini_ci_result
+# local variables
+satellite_ip=10.101.47.1
+
+############ test hardware CID ####################################
+# fake
+#mini_ci_machine_c=111111-11111
+
+# Lenovo ThinkCentre Edge 91z
+# back to OEM
+mini_ci_machine_a=201212-12224
+
+# HP Pavilion Slimline 400 Desktop PC certied with 14.04
+#mini_ci_machine_b=201305-13517
+
+# Dell OptiPlex 3011
+# 14.04 certificated
+mini_ci_machine_b=201302-12681
+
+# Lenovo ThinkCentre M62z
+# BCM failed
+mini_ci_machine_c=201212-12226
+
+# Acer AT110 F2
+mini_ci_machine_d=201209-11625
+
+#HP Pavilion P2
+mini_ci_machine_e=201206-11407
+
+# Dell Optiplex 3030
+# 14.04.1 will fail to have ethernet at the last stage.
+#mini_ci_machine_d=201401-14498
+
+#system broken
+#mini_ci_machine_f=201109-9789 swap to below:
+mini_ci_machine_f=201209-11784
+
+# Dell Optiplex 9020 AIO
+# needs BCM proprietary driver
+mini_ci_machine_g=201211-12157
+
+# HP Pavilion 23 AIO - Heineken
+# needs BCM proprietary driver
+mini_ci_machine_h=201206-11396
+
+# SRU precise 3.13 test unit
+# no bluetooth, so sad
+mini_ci_machine_i=201302-12728
+
+# Dell Latitude 3330 - Austin
+# a laptop without bt
+# 12.04.5 certified
+mini_ci_machine_j=201302-12843
+
+# SRU trusty 3.13 test unit
+mini_ci_machine_k=201301-12632
+
+# was for 15.04, but now for xenial desktop only
+mini_ci_machine_l=201401-14498
+
+
+###########################################
+#
+# machine usage list
+#
+# convention: m for machine
+#
+###########################################
+m_12_04_5_desktop=201302-12843
+m_14_04_4_desktop=201401-14498
+m_16_04_desktop=201302-12843
+m_16_10_desktop=201302-12681
+m_zesty_daily_desktop=201302-12681
+
+m_12_04_5_server=201209-11625
+m_14_04_4_server=201209-11625
+m_16_04_server=201209-11625
+m_yakkety_daily_server=201302-12681
+
+m_16_04_amdgpu_1=201302-12843
+m_16_04_amdgpu_2=201302-12681
+#m_16_04_amdgpu_2=$mini_ci_machine_a
+m_16_04_amdgpu_4=$m_14_04_4_desktop
+
+m_autopilot=201401-14498
+m_snapcraft=201401-14498
+
+############# daily schedule #################
+#
+# Taipei time
+# 2000 - 1400: mini-ci running
+#   1200 - 0600 (yantok)
+# 1400 - 2000: debug time lol
+#   0600 - 1200 (yantok)
+#
+#
+# Time required:
+# desktop precise: 00h 55m --> 02h
+# desktop trusty:  01h 00m --> 02h
+# desktop Xenial:  01h 10m --> 02h
+#
+# server precise:  03h 30m --> 05h
+# server trusty:   03h 30m --> 05h
+# server Xenial:   05h 50m --> 07h
+##############################################
+
+
+###########################################
+# yantok/C3    Taipei
+# 12:00/13:00  20:00
+#
+# mini-ci begins on 20:00 (Taipei) each day
+#
+# Mon Wen Fri (Taipei)
+# desktop and server
+#
+# Tue Thr Sat (Taipei)
+# AMD GPU systems
+#
+# Sunday does nothing! (Bible says)
+#
+# Every day: autopilot for checkbox-coverged
+#
+###########################################
+
+
+########################################################################
+# dont launch_testrun at the same time otherwise dhcp table will mess up
+#######################################################################u
+############################### desktop ##################################################
+01 12 * * 2,4,6 $run_mini_ci launch-reset     12.04.5 desktop   $m_12_04_5_desktop
+00 14 * * 2,4,6 $run_mini_ci done-desktop     12.04.5 desktop   $m_12_04_5_desktop 12.04.5 desktop
+
+01 14 * * 2,4,6 $run_mini_ci launch           14.04.4 desktop   $m_14_04_4_desktop
+00 16 * * 2,4,6 $run_mini_ci done-desktop     14.04.4 desktop   $m_14_04_4_desktop 14.04.4 desktop
+
+# mini_ci_machine_l did not suspend successfully. - 160512
+# a failure entry in checkbox test result - 1605xx
+01 16 * * 2,4,6 $run_mini_ci launch           16.04   desktop   $m_16_04_desktop
+00 18 * * 2,4,6 $run_mini_ci done-desktop     16.04   desktop   $m_16_04_desktop 16.04 desktop
+
+01 19 * * 2,4,6 $run_mini_ci launch           16.10   desktop   $m_16_10_desktop
+00 21 * * 2,4,6 $run_mini_ci done-desktop     16.10   desktop   $m_16_10_desktop 16.10 desktop
+# daily image so we need to sync up it each time before testing
+01 15 * * 2,4,6 $lt -S 17.04 desktop
+01 22 * * 2,4,6 $run_mini_ci launch           17.04   desktop   $m_zesty_daily_desktop
+00 23 * * 2,4,6 $run_mini_ci done-desktop     17.04   desktop   $m_zesty_daily_desktop Zesty desktop
+
+############################## server ####################################################
+05 12 * * 2,4,6 $run_mini_ci launch           12.04.5 server    $m_12_04_5_server
+04 17 * * 2,4,6 $run_mini_ci done-server      12.04.5 server    $m_12_04_5_server 12.04.5 server
+
+05 17 * * 2,4,6 $run_mini_ci launch           14.04.4 server    $m_14_04_4_server
+04 22 * * 2,4,6 $run_mini_ci done-server      14.04.4 server    $m_14_04_4_server 14.04.4   server
+
+05 22 * * 2,4,6 $run_mini_ci launch           16.04   server    $m_16_04_server
+# another day
+04 05 * * 3,5,7 $run_mini_ci done-server      16.04   server    $m_16_04_server 16.04  server
+
+# FIX ME:
+# Dont test completely
+#05 13 * * 2,4,6 $run_mini_ci launch           16.10   server   $m_yakkety_daily_desktop
+#04 18 * * 2,4,6 $run_mini_ci done-desktop     16.10   server   $m_yakkety_daily_desktop 16.10  server
+
+
+################# amd gpu test for Xenial #################################################
+00 12 * * 1,3,5 $run_mini_ci launch           16.04   desktop   $m_16_04_amdgpu_1 dummy  dummy   -p
+00 14 * * 1,3,5 $run_mini_ci done-desktop     16.04   desktop   $m_16_04_amdgpu_1 16.04  desktop -p
+
+10 14 * * 1,3,5 $run_mini_ci launch           16.04   desktop   $m_16_04_amdgpu_2 dummy  dummy   -p
+10 16 * * 1,3,5 $run_mini_ci done-desktop     16.04   desktop   $m_16_04_amdgpu_2 16.04  desktop -p
+#############################################################################################
+
+
+################# checkbox-converged autopilot on Xenial ###################################
+00 18 * * * $run_autopilot launch            16.04   desktop   $m_autopilot
+
+################# checkbox-converged snapcraft on Xenial ###################################
+00 20 * * * $run_snapcraft 16.04 desktop $m_snapcraft
+
+############################### email ######################################################
+59 05 * * * $backup_mini_ci_result
+# 14:00 (Taipei)
+00 06 * * 3,5,7 /usr/bin/curl -d "emailit=yes" http://$satellite_ip/cgi-bin/plainbox-submit.cgi
+# separate official release and daily build test
+#00 06 * * 3,5 /usr/bin/curl -d "emailit=yes" -d "dailyimage=yes" http://$satellite_ip/cgi-bin/plainbox-submit.cgi
+# separate official release and amd gpu test
+00 06 * * 2,4,6 /usr/bin/curl -d "emailit=yes" -d "amdgpu=yes" http://$satellite_ip/cgi-bin/plainbox-submit.cgi
diff --git a/src/crontab/snapcraft/run-snapcraft b/src/crontab/snapcraft/run-snapcraft
new file mode 100755
index 0000000..98a9b68
--- /dev/null
+++ b/src/crontab/snapcraft/run-snapcraft
@@ -0,0 +1,9 @@
+#!/bin/bash
+if [ -z "$STAGING" ]; then
+    export CHECKBOX_SRC=${CHECKBOX_SRC:-${HOME}/src}
+else
+    export CHECKBOX_SRC=${CHECKBOX_SRC:-${HOME}/src.staging}
+fi
+
+lt_minici="$CHECKBOX_SRC/checkbox-message/trunk/bin/launch_testrun --snapcraft --ppa=dev"
+$lt_minici $1 $2 $3 > /tmp/mini-ci-${1}-${2}-${3}-snapcraft-`date +%y%m%d` 2>&1
diff --git a/src/crontab/snapcraft/run-snapcraft-staging b/src/crontab/snapcraft/run-snapcraft-staging
new file mode 100755
index 0000000..eebb578
--- /dev/null
+++ b/src/crontab/snapcraft/run-snapcraft-staging
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# if STAGING is not empty, $HOME/src.staging will be used instead of $HOME/src
+export STAGING='.src'
+dir=`dirname $0`
+$dir/run-snapcraft "$@"
+
diff --git a/src/www/autopilot/run-autopilot-auto.desktop b/src/www/autopilot/run-autopilot-auto.desktop
new file mode 100644
index 0000000..25d4e58
--- /dev/null
+++ b/src/www/autopilot/run-autopilot-auto.desktop
@@ -0,0 +1,8 @@
+[Desktop Entry]
+Encoding=UTF-8
+Name=Autopilot
+Comment=Autopilot can run after login to test a system automatically.
+Exec=sh /home/ubuntu/run-autopilot-test.sh
+Terminal=false
+Type=Application
+Categories=
diff --git a/src/www/autopilot/run-autopilot-test.sh b/src/www/autopilot/run-autopilot-test.sh
new file mode 100644
index 0000000..37652c7
--- /dev/null
+++ b/src/www/autopilot/run-autopilot-test.sh
@@ -0,0 +1,33 @@
+# script to make the autopilot test automatical
+cgi_url="http://10.101.47.1/cgi-bin/plainbox-submit.cgi"
+log_autopilot="/tmp/minici_make-autopilot.log"
+log_autopilot_xvfb="/tmp/minici_make-autopilot-xvfb-run.log"
+log_pastebinit="/tmp/minici_pastebinit.log"
+# to wait and make sure the network connects after xdg lauched.
+sleep 60
+# prepare
+cd $HOME
+mkdir checkbox && cd checkbox
+git clone git://git.launchpad.net/plainbox
+git clone git://git.launchpad.net/checkbox-converged
+cd checkbox-converged/
+# for running on local machine
+#make autopilot > $log_autopilot 2>&1
+# for virtual framebuffer analysis
+echo "xvfb-run --server-args='-screen 0 1024x768x24 -ac +render -noreset' make autopilot" > $log_autopilot_xvfb
+xvfb-run --server-args='-screen 0 1024x768x24 -ac +render -noreset' make autopilot >> $log_autopilot_xvfb 2>&1
+# pass the test log files to pastebin and ready for further using
+#pastebinit $log_autopilot > $log_pastebinit 2>&1
+#pastebinit $log_autopilot_xvfb >> $log_pastebinit 2>&1
+#curl -F autopilot_subject="make autopilot" -F autopilot_output=@$log_autopilot $cgi_url
+curl -F autopilot_subject="make autopilot xvfb-run" -F autopilot_output=@$log_autopilot_xvfb $cgi_url
+
+# send the nofification email
+# in this case we only run the test on one machine within one day
+# if we would like to wait for many test from different fresh installation
+# we need to apply this command on a server which waiting for other test results.
+curl -d "email_mini_ci_autopilot=yes" $cgi_url
+
+# save the polar bear and protect the hardware!
+sudo shutdown -h now
+
diff --git a/src/www/snapcraft/run-snapcraft-test-launcher.sh b/src/www/snapcraft/run-snapcraft-test-launcher.sh
new file mode 100755
index 0000000..1d76ffe
--- /dev/null
+++ b/src/www/snapcraft/run-snapcraft-test-launcher.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+cgi_url="http://10.101.47.1/cgi-bin/plainbox-submit.cgi"
+minici_snapcraft_log="/home/ubuntu/minici-snapcraft/minici-snapcraft.log"
+# wait for the ethernet getting ready
+sleep 60
+# run the real test and fetch the log
+cd /home/ubuntu/minici-snapcraft
+./run-snapcraft-test.sh  > ./minici-snapcraft.log 2>&1
+execute_status=$?
+
+# TODO: submit the hostname to cgi
+
+curl -F snapcraft_subject="./run-snapcraft-test.sh" -F snapcraft_output=@${minici_snapcraft_log} -F snapcraft_status="$execute_status" $cgi_url
+echo "snapcraft_status=$execute_status"
+curl -d "email_mini_ci_snapcraft=yes" -d "snapcraft_status=$execute_status" $cgi_url
+# protect the test machine and polar bears!
+shutdown -h now
diff --git a/src/www/snapcraft/run-snapcraft-test.desktop b/src/www/snapcraft/run-snapcraft-test.desktop
new file mode 100644
index 0000000..6b59c3e
--- /dev/null
+++ b/src/www/snapcraft/run-snapcraft-test.desktop
@@ -0,0 +1,8 @@
+[Desktop Entry]
+Encoding=UTF-8
+Name=Snapcraft P-P-S test
+Comment=Test whether plainbox-provider-snappy is snappable with snapcraft
+Exec=sh /home/ubuntu/run-snapcraft-test-launcher.sh
+Terminal=false
+Type=Application
+Categories=
diff --git a/src/www/snapcraft/run-snapcraft-test.sh b/src/www/snapcraft/run-snapcraft-test.sh
new file mode 100755
index 0000000..c54522c
--- /dev/null
+++ b/src/www/snapcraft/run-snapcraft-test.sh
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# This file is part of Checkbox.
+#
+# Copyright 2016-2017 Canonical Ltd.
+# Written by:
+#   Maciej Kisielewski <[e-mail]>
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+
+SAVED_DIR=$PWD
+WORK_DIR=`mktemp -d`
+
+cd $WORK_DIR
+mkdir -p logs
+LOG_DIR=$WORK_DIR/logs
+
+function clean_up {
+    cd $SAVED_DIR
+    rm -rf $WORK_DIR
+}
+
+function time_it {
+    /usr/bin/time -f "%E elapsed. %P CPU" -o $LOG_DIR/timing.dat $@
+}
+
+function pastebin_logs {
+    if [ -s $LOG_DIR/stdout ]; then
+        echo stdout: $(pastebinit $LOG_DIR/stdout)
+    else
+        echo stdout: empty
+    fi
+    if [ -s $LOG_DIR/stderr ]; then
+        echo stderr: $(pastebinit $LOG_DIR/stderr)
+    else
+        echo stderr: empty
+    fi
+}
+
+function run_step {
+    echo -n Step \"$1\"
+    if ! time_it $2 >>$LOG_DIR/stdout 2>>$LOG_DIR/stderr; then
+        echo -n "- FAIL - "
+        outcome=1
+    else
+        echo -n "- PASS - "
+        outcome=0
+    fi
+    echo -n elapsed: ; cat $LOG_DIR/timing.dat
+    if [ $outcome -ne 0 ]; then
+        pastebin_logs
+        return 1
+    fi
+}
+
+# master branch test
+function test_master {
+    run_step "git-clone" "git clone -b master https://git.launchpad.net/checkbox-snappy packaging" || return 1
+    run_step "update-snapcraft" "snapcraft update" || return 1
+    (cd packaging && run_step "run-snapcraft" "snapcraft snap -o ../checkbox-snappy.snap") || return 1
+    run_step "verify-build" "ls -al checkbox-snappy.snap" || return 1
+    run_step "install-checkbox-snappy" "sudo snap install --devmode ./checkbox-snappy.snap" || return 1
+    (cd && run_step "run smoke-test" "checkbox-snappy.smoke-test") || return 1
+    return 0
+}
+test_master
+master_result=$?
+if [[ $master_result -eq 1 ]]; then
+    echo "Testing Checkbox snap for branch master failed!"
+fi
+pastebin_logs
+
+# beta branch test
+function test_beta {
+    run_step "git-clone" "git clone -b beta https://git.launchpad.net/checkbox-snappy packaging-beta" || return 1
+    run_step "update-snapcraft" "snapcraft update" || return 1
+    (cd packaging-beta && run_step "run-snapcraft" "snapcraft snap -o ../checkbox-snappy-beta.snap") || return 1
+    run_step "verify-build" "ls -al checkbox-snappy-beta.snap" || return 1
+    run_step "install-checkbox-snappy" "sudo snap install --devmode ./checkbox-snappy-beta.snap" || return 1
+    (cd && run_step "run smoke-test" "checkbox-snappy.smoke-test") || return 1
+    return 0
+}
+test_beta
+beta_result=$?
+if [[ $beta_result -eq 1 ]]; then
+    echo "Testing Checkbox snap for branch beta failed!"
+fi
+pastebin_logs
+
+clean_up
+echo "$master_result"
+echo "$beta_result"
+if [[ $master_result -eq 0 && $beta_result -eq 0 ]]; then
+    echo "ALL STEPS PASSED"
+    exit 0
+else
+    exit 1
+fi