Merge lp:~caio1982/ubuntu-system-image/sis-skip404 into lp:ubuntu-system-image/server

[Caio Begotti]

Image server goes crazy if the configured image is not there at the time of importing. What happens is that it fetches an Apache error page (HTML) and try to import it, which surprisingly works. Then when the image is finally there it tries to generate a delta, and that obviously fails because one of the files (the source one) is not an image.

The only way to recover from this state is to manually intervene and remove the first processed files, json, images etc. This is very problematic for us in PES.

Although this MR is kind of hackish it does the job for running image server on Prodstack. I understand a more robust solution would be needed to handle corner cases (perhaps using Requests instead of urllib, to start with).

[lp:~caio1982/ubuntu-system-image/sis-skip404]

262. By Caio Begotti on 2015-03-11

catch all http errors above 400 instead and make the logging message more generic to reflect it

263. By Caio Begotti on 2015-03-12

transform all calls to urlopen and urlretrieve in wrapper calls so we catch all timeouts and >= 400 http errors better

264. By Caio Begotti on 2015-03-12

fix flake errors due to leftovers from previous commit

265. By Caio Begotti on 2015-03-12

add more debug logging to the methods we are wrapping around

Unmerged revisions

265. By Caio Begotti on 2015-03-12

add more debug logging to the methods we are wrapping around

264. By Caio Begotti on 2015-03-12

fix flake errors due to leftovers from previous commit

263. By Caio Begotti on 2015-03-12

transform all calls to urlopen and urlretrieve in wrapper calls so we catch all timeouts and >= 400 http errors better

262. By Caio Begotti on 2015-03-11

catch all http errors above 400 instead and make the logging message more generic to reflect it

261. By Caio Begotti on 2015-03-11

make s-i-s a bit smarted by returning none if the actual file was not found, otherwise it will process a dummy html page with a 404 error message inside it... this is more like a hack because using urllib these days seems already wrong and i dont have the time to fully fix how all files are urlopened or urlretrieved in its code

=== modified file 'lib/systemimage/generators.py'
--- lib/systemimage/generators.py	2015-02-24 22:24:16 +0000
+++ lib/systemimage/generators.py	2015-03-11 13:28:58 +0000
@@ -890,6 +890,12 @@
             environment['version_detail'].append(version_detail)
             return path
 
+    # Make sure it's there before retrieval
+    http_code = urlopen(url).getcode()
+    if http_code == 404:
+        logger.debug('Not found on the server, skipping %s' % url)
+        return None
+
     # Grab the real thing
     tempdir = tempfile.mkdtemp()
     old_timeout = socket.getdefaulttimeout()