Merge lp:~sergiusens/phablet-tools/server_options into lp:phablet-tools

[Sergio Schvezov]

phablet-flash: Add an option for an alternate server for system images

[Matt Fischer (mfisch)]

Not working for me:

mfisch@caprica:/tmp/server_options$ ./phablet-flash ubuntu-system --channel devel-proposed --alternate-server http://23.21.157.252/~ubuntu/touch-mirror/system-image.ubuntu.com --no-backup
INFO:phablet-flash:Device detected as mako
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): system-image.ubuntu.com
INFO:phablet-flash:Download directory set to /home/mfisch/Downloads/phablet-flash/imageupdates
INFO:phablet-flash:Downloading https://system-image.ubuntu.com/pool/ubuntu-4a26dee7e03f278bb3e2c2ce48b065e3e68f2c0ec88649959ecaff7cddb43b4c.tar.xz to /home/mfisch/Downloads/phablet-flash/imageupdates/pool/ubuntu-4a26dee7e03f278bb3e2c2ce48b065e3e68f2c0ec88649959ecaff7cddb43b4c.tar.xz
--2013-10-15 12:46:03-- https://system-image.ubuntu.com/pool/ubuntu-4a26dee7e03f278bb3e2c2ce48b065e3e68f2c0ec88649959ecaff7cddb43b4c.tar.xz
Resolving system-image.ubuntu.com (system-image.ubuntu.com)..

[lp:~sergiusens/phablet-tools/server_options]

209. By Sergio Schvezov on 2013-10-15

Using uri that comes from arguments.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:208
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/211/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-amd64-ci/154
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-armhf-ci/154
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-i386-ci/155

Click here to trigger a rebuild:
http://10.97.0.26:8080/job/phablet-tools-ci/211/rebuild

review: Approve (continuous-integration)

[lp:~sergiusens/phablet-tools/server_options]

210. By Sergio Schvezov on 2013-10-15

Using the right args

[Matt Fischer (mfisch)]

Still broken. I see 2 issues, first it has curl issues and second it still claims to be reading system-image.

mfisch@caprica:/tmp/server_options$ ./phablet-flash ubuntu-system --channel devel-proposed --alternate-server http://23.21.157.252/~ubuntu/touch-mirror/system-image.ubuntu.com --no-backup
INFO:phablet-flash:Device detected as mako
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): system-image.ubuntu.com
INFO:phablet-flash:Download directory set to /home/mfisch/Downloads/phablet-flash/imageupdates
INFO:phablet-flash:Downloading http://23.21.157.252/~ubuntu/touch-mirror/system-image.ubuntu.com/pool/ubuntu-1c82da4133142a3ace9105bceb63488a55752a77b3044575e643ac73286e7b44.tar.xz to /home/mfisch/Downloads/phablet-flash/imageupdates/pool/ubuntu-1c82da4133142a3ace9105bceb63488a55752a77b3044575e643ac73286e7b44.tar.xz
** Resuming transfer from byte position 405
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 405 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.
ERROR:phablet-flash:Command '['curl', '-L', '-C', '-', u'http://23.21.157.252/~ubuntu/touch-mirror/system-image.ubuntu.com/pool/ubuntu-1c82da4133142a3ace9105bceb63488a55752a77b3044575e643ac73286e7b44.tar.xz', '-o', u'/home/mfisch/Downloads/phablet-flash/imageupdates/pool/ubuntu-1c82da4133142a3ace9105bceb63488a55752a77b3044575e643ac73286e7b44.tar.xz']' returned non-zero exit status 33
Removing directory /tmp/tmpp0ykfe

[Matt Fischer (mfisch)]

That URL curl is trying to use is bad, i Think because line 28 in ubuntuimage.py still uses the wrong URL.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:210
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/213/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-amd64-ci/156
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-armhf-ci/156
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-i386-ci/157

Click here to trigger a rebuild:
http://10.97.0.26:8080/job/phablet-tools-ci/213/rebuild

review: Approve (continuous-integration)

[lp:~sergiusens/phablet-tools/server_options]

211. By Sergio Schvezov on 2013-10-15

sending uri to get json index and using uri to generate hash for download dir

212. By Sergio Schvezov on 2013-10-15

Adding get_sha256_hash funtion and preferring wget over everything else.

213. By Sergio Schvezov on 2013-10-15

Adding uri parameter for getting indexes

[Matt Fischer (mfisch)]

Working great. Thanks

review: Approve

[Andy Doan (doanac)]

8 + parser.add_argument('--alternate-server',
9 + default=settings.system_image_uri,
10 + help='Use an alternate system server, the default '
11 + 'server is %s' % settings.system_image_uri)

nitpick, but i think:
  help='Use an alternate system server, the default '
       'server is %(default)s')

is more common.

I'm not a big fan of the download function logic:
 + if uri.startswith('http://10.97.0.26'):

but adding some "--curl" option and then passing it around everyone is a bit cumbersome also.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:213
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/215/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-amd64-ci/158/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-armhf-ci/158/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-i386-ci/159/console

Click here to trigger a rebuild:
http://10.97.0.26:8080/job/phablet-tools-ci/215/rebuild

review: Needs Fixing (continuous-integration)

[Andy Doan (doanac)]

> I'm not a big fan of the download function logic:
> + if uri.startswith('http://10.97.0.26'):
>
> but adding some "--curl" option and then passing it around everyone is a bit
> cumbersome also.

I'm starting to think we should make this optional. For example, we have an upcoming lab move and we probably don't want to tie phablet-tools to our current lab configuration. I think there are two alternatives you could try:

1) add a new "--curl" option that says to use curl.

2) add a new "--download-cmd" option that allows you to specify what command to download with. The default command would then be: "wget -c {uri} -O {path}".

[Stéphane Graber (stgraber)]

Looks good here, just ran against https://phablet.stgraber.org (channel test-customized) and it downloaded everything as expected.

review: Approve

[lp:~sergiusens/phablet-tools/server_options]

214. By Sergio Schvezov on 2013-10-15

Fixing help

[Andy Doan (doanac)]

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:214
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/216/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-amd64-ci/159/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-armhf-ci/159/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-i386-ci/160/console

Click here to trigger a rebuild:
http://10.97.0.26:8080/job/phablet-tools-ci/216/rebuild

review: Needs Fixing (continuous-integration)

[lp:~sergiusens/phablet-tools/server_options]

215. By Sergio Schvezov on 2013-10-16

Updating tests

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:215
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/218/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-amd64-ci/161
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-armhf-ci/161
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-saucy-i386-ci/162

Click here to trigger a rebuild:
http://10.97.0.26:8080/job/phablet-tools-ci/218/rebuild

review: Approve (continuous-integration)

[Ricardo Salveti (rsalveti)]

Tested, worked fine with stgraber's server.

review: Approve

=== modified file 'phabletutils/arguments.py'
--- phabletutils/arguments.py	2013-09-17 12:51:01 +0000
+++ phabletutils/arguments.py	2013-10-16 01:55:23 +0000
@@ -135,6 +135,10 @@
                         type=int,
                         help='''Download a relative revision from current
                                 (-1, -2, ...) or a specific version.''')
+    parser.add_argument('--alternate-server',
+                        default=settings.system_image_uri,
+                        help='''Use an alternate system server, the default
+                                server is %(default)s''')
     group = parser.add_mutually_exclusive_group()
     group.add_argument('--no-backup',
                        action='store_false',

=== modified file 'phabletutils/downloads.py'
--- phabletutils/downloads.py	2013-09-06 13:29:01 +0000
+++ phabletutils/downloads.py	2013-10-16 01:55:23 +0000
@@ -76,6 +76,10 @@
     return directory
 
 
+def get_sha256_hash(string):
+    return hashlib.sha256(string).hexdigest()
+
+
 def checksum_verify(file_path, file_hash, sum_method=hashlib.sha256):
     '''Returns the checksum for a file with a specified algorightm.'''
     file_sum = sum_method()
@@ -96,14 +100,7 @@
 
 
 def _download(uri, path):
-    if uri.startswith('http://cdimage.ubuntu.com') or \
-       uri.startswith('https://system-image.ubuntu.com'):
-        subprocess.check_call(['wget',
-                               '-c',
-                               uri,
-                               '-O',
-                               path])
-    else:
+    if uri.startswith('http://10.97.0.26'):
         subprocess.check_call(['curl',
                                '-L',
                                '-C',
@@ -111,6 +108,12 @@
                                uri,
                                '-o',
                                path])
+    else:
+        subprocess.check_call(['wget',
+                               '-c',
+                               uri,
+                               '-O',
+                               path])
 
 
 def download_sig(artifact):

=== modified file 'phabletutils/environment.py'
--- phabletutils/environment.py	2013-09-17 23:17:37 +0000
+++ phabletutils/environment.py	2013-10-16 01:55:23 +0000
@@ -146,17 +146,23 @@
 
 def setup_ubuntu_system(args):
     device = detect_device(args.serial, args.device)
+    uri = args.alternate_server
     if args.revision <= 0:
         json = ubuntuimage.get_json_from_index(device,
                                                args.channel,
-                                               args.revision)
+                                               args.revision,
+                                               uri)
     else:
         json = ubuntuimage.get_json_from_version(device,
                                                  args.channel,
-                                                 args.revision)
-    download_dir = downloads.get_full_path(os.path.join(
-        settings.download_dir, args.project))
-    uri = settings.system_image_uri
+                                                 args.revision,
+                                                 uri)
+    download_dir_part = os.path.join(settings.download_dir, args.project)
+    if uri != settings.system_image_uri:
+        base_uri = downloads.get_sha256_hash(uri)
+        download_dir_part = os.path.join(download_dir_part, base_uri)
+    download_dir = downloads.get_full_path(download_dir_part)
+
     files, command_part = ubuntuimage.get_files(download_dir, uri, json)
     return projects.UbuntuTouchSystem(
         file_list=files,

=== modified file 'phabletutils/ubuntuimage.py'
--- phabletutils/ubuntuimage.py	2013-09-17 23:17:37 +0000
+++ phabletutils/ubuntuimage.py	2013-10-16 01:55:23 +0000
@@ -20,30 +20,29 @@
 from phabletutils import downloads
 from phabletutils import fileutils
 from phabletutils import resources
-from phabletutils import settings
-
-
-def _get_json(device, channel):
+
+
+def _get_json(device, channel, uri):
     json_index_uri = '%s/%s/%s/index.json' % \
-        (settings.system_image_uri, channel, device)
+        (uri, channel, device)
     json_content = downloads.get_content(json_index_uri)
     if not json_content:
         raise RuntimeError('%s cannot be retrieved' % json_index_uri)
     return json.loads(json_content)
 
 
-def get_json_from_index(device, channel, index):
+def get_json_from_index(device, channel, index, uri):
     """Returns json index for device"""
     index -= 1
-    json_index = _get_json(device, channel)
+    json_index = _get_json(device, channel, uri)
     json_dict = sorted([entry for entry in json_index['images']
                        if entry['type'] == "full"],
                        key=lambda entry: entry['version'])[index]
     return json_dict
 
 
-def get_json_from_version(device, channel, version):
-    for entry in _get_json(device, channel)['images']:
+def get_json_from_version(device, channel, version, uri):
+    for entry in _get_json(device, channel, uri)['images']:
         if entry['type'] == 'full' and entry['version'] == version:
             return entry
     raise RuntimeError('Version %s cannot be retrieved' % version)

=== modified file 'tests/test_ubuntuimage.py'
--- tests/test_ubuntuimage.py	2013-08-28 14:44:14 +0000
+++ tests/test_ubuntuimage.py	2013-10-16 01:55:23 +0000
@@ -38,7 +38,8 @@
         # when
         json_dict = ubuntuimage.get_json_from_index(self.device,
                                                     self.channel,
-                                                    0)
+                                                    0,
+                                                    'http://server.com')
         # then
         self.assertThat(json_dict['version'], Equals(20130808))
         self.assertThat(json_dict['description'], Equals('20130807'))
@@ -51,7 +52,8 @@
         # when
         json_dict = ubuntuimage.get_json_from_index(self.device,
                                                     self.channel,
-                                                    -1)
+                                                    -1,
+                                                    'http://server.com')
         # then
         self.assertThat(json_dict['version'], Equals(20130807))
         self.assertThat(json_dict['description'], Equals('20130806.1'))