Merge lp:~xnox/ubuntu-release-upgrader/lp1409555 into lp:ubuntu-release-upgrader

[Dimitri John Ledkov]

Not sure how to test this "for-real".

Upgrade from utopic -> vivid should work with this upgrader, sources.list after the upgrade should not have neither the comment nor the extras.ubuntu.com repository.

I do not know if the comment in the sources.list is translated or not into other languages, if yes I might need a few more changes to completely remove traces of extras.ubuntu.com.

[Brian Murray (brian-murray)]

I've search my local archive of bug attachments, specifically sources.list files and VarLogDistupgradeAptclonesystemstate.tar.gz (contains sources.list), and couldn't find the comments you are looking for translated at all so that's good.

[Brian Murray (brian-murray)]

To test it "for-real" we could build the package, copy over the vivid.tar.gz to a utopic system and try upgrading.

[Brian Murray (brian-murray)]

There was a failure with the test that I've commented on in-line.

[Dimitri John Ledkov (xnox)]

On 13 January 2015 at 22:50, Brian Murray <email address hidden> wrote:
> There was a failure with the test that I've commented on in-line.
>
>> +
>> + sources_file = apt_pkg.config.find_file("Dir::Etc::sourcelist")
>
> The test was failing for me and it seems to because the sources_file isn't saved when updateSourcesList is called. This is why the other tests use self._verifySources() whcih I've switched to using for your test.
>

Hm, this could be due to environment changes. I'm developing this on Trusty.

_verifySources() is not sufficient for this test. _verifySources only
checks that "these lines are in the file", I however need to test for
"extras.ubuntu.com lines _are removed from the file_" Thus
_verifySources was passing the test for me, with or without, the
patch. I don't believe wee have done outright removal ever before
(only commenting things out).

I'll check the branch / and test on utopic/vivid in chroot to see what
is going on.

>> + self.assertEqual(open(sources_file).read(),"""deb http://archive.ubuntu.com/ubuntu gutsy main restricted
>> +deb http://archive.ubuntu.com/ubuntu gutsy-updates main restricted
>> +deb http://security.ubuntu.com/ubuntu/ gutsy-security main restricted
>> +
>> +""")
>> +
>> def test_powerpc_transition(self):
>> """
>> test transition of powerpc to ports.ubuntu.com
>>

--
Regards,

Dimitri.

[Brian Murray (brian-murray)]

In the process of tracking down the test failure I also found a method for removing entries from sources.list, sources.list.remove(), and that seems like a better idea to me than creating a new list.

[Dimitri John Ledkov (xnox)]

On 13 January 2015 at 23:00, Brian Murray <email address hidden> wrote:
> In the process of tracking down the test failure I also found a method for removing entries from sources.list, sources.list.remove(), and that seems like a better idea to me than creating a new list.

That was my original implementation, however it's no good. As it
iterates the list for each removal invokes comparators to remove the
first item with the matching value from the list. Given that list is
"objects" rather than strings, the value for the "commented" string
did not match. Furthermore in python one is not allowed to modify a
list whilst iterating over it, thus a new list (or copy will be
required). This single pass is cumbersome, but most straight-forward
way to implement it. Or e.g. python3 on trusty has some subtle object
comparison error.

If it works and passes the upgrade test with both trusty & utopic
python3, I'm all happy to use whichever algorithm that gives correct
end-result.

--
Regards,

Dimitri.

[Dimitri John Ledkov (xnox)]

I've now pushes fixes for all open() calls and logging.warn.

The build ends up using system installation of the ubuntu-release-upgrader.... one cannot remove it as the source code has circular depenency on itself (system update-notifier must be installed, which depends on the system ubuntu-release-upgrader)

...

$ sudo rm /usr/lib/python3/dist-packages/DistUpgrade
$ bzr bd

Above two commands make the build complete including the full test suite, in the current branch.

Note that in the pre-build.sh:

 "(cd utils && ./demotions.py utopic vivid > demoted.cfg)"

passes for me on Trusty, but fails on Vivid, which may be related to changes/regressions in vivid's python-apt. However, I believe that is not related to my changes.

[Michael Vogt (mvo)]

Hi Dimitri, thanks a lot for this work. The branch looks good, whats puzzling is that I see a test failure when running bzr-buildpackage. It looks like nosetests3 is not using the current directory and not honoring PYTHONPATH here which is a bit confusing. Only when python3-distupgrade is not instaleld do the tests work for me when run with nosetest3. Beside this the diff looks fine and I think we should upload it.

review: Approve

=== modified file 'DistUpgrade/DistUpgradeAptCdrom.py'
--- DistUpgrade/DistUpgradeAptCdrom.py	2014-06-26 06:43:50 +0000
+++ DistUpgrade/DistUpgradeAptCdrom.py	2015-01-20 22:44:23 +0000
@@ -63,9 +63,11 @@
         diskname = self._readDiskName()
         pentry = self._generateSourcesListLine(diskname, self.packages)
         sourceslist = apt_pkg.config.find_file("Dir::Etc::sourcelist")
-        content = open(sourceslist).read()
+        with open(sourceslist) as f:
+            content = f.read()
         content = content.replace(pentry, "# %s" % pentry)
-        open(sourceslist, "w").write(content)
+        with open(sourceslist, "w") as f:
+            f.write(content)
 
     def _scanCD(self):
         """ 
@@ -106,9 +108,9 @@
         cdrom = apt_pkg.Cdrom()
         id = cdrom.ident(apt.progress.base.CdromProgress())
         label = self._readDiskName()
-        out = open(dbfile, "a")
-        out.write('CD::%s "%s";\n' % (id, label))
-        out.write('CD::%s::Label "%s";\n' % (id, label))
+        with open(dbfile, "a") as out:
+            out.write('CD::%s "%s";\n' % (id, label))
+            out.write('CD::%s::Label "%s";\n' % (id, label))
 
     def _dropArch(self, packages):
         """ drop architectures that are not ours """
@@ -127,7 +129,8 @@
         diskname = self.cdrompath
         info = os.path.join(self.cdrompath, ".disk", "info")
         if os.path.exists(info):
-            diskname = open(info).read()
+            with open(info) as f:
+                diskname = f.read()
             for special in ('"', ']', '[', '_'):
                 diskname = diskname.replace(special, '_')
         return diskname
@@ -162,17 +165,13 @@
                 "cdrom:[%s]/%s" % (diskname, f[f.find("dists"):]))
             outf = os.path.join(targetdir, os.path.splitext(fname)[0])
             if f.endswith(".gz"):
-                g = gzip.open(f)
-                try:
-                    with open(outf, "wb") as out:
-                        # uncompress in 64k chunks
-                        while True:
-                            s = g.read(64000)
-                            out.write(s)
-                            if s == b"":
-                                break
-                finally:
-                    g.close()
+                with gzip.open(f) as g, open(outf, "wb") as out:
+                    # uncompress in 64k chunks
+                    while True:
+                        s = g.read(64000)
+                        out.write(s)
+                        if s == b"":
+                            break
             else:
                 shutil.copy(f, outf)
         return True
@@ -187,17 +186,13 @@
                 "cdrom:[%s]/%s" % (diskname, f[f.find("dists"):]))
             outf = os.path.join(targetdir, os.path.splitext(fname)[0])
             if f.endswith(".gz"):
-                g = gzip.open(f)
-                try:
-                    with open(outf, "wb") as out:
-                        # uncompress in 64k chunks
-                        while True:
-                            s = g.read(64000)
-                            out.write(s)
-                            if s == b"":
-                                break
-                finally:
-                    g.close()
+                with gzip.open(f) as g, open(outf, "wb") as out:
+                    # uncompress in 64k chunks
+                    while True:
+                        s = g.read(64000)
+                        out.write(s)
+                        if s == b"":
+                            break
             else:
                 shutil.copy(f, outf)
         return True
@@ -218,15 +213,18 @@
             if not (ret == 0):
                 return False
             # now do the hash sum checks
-            t = apt_pkg.TagFile(open(releasef))
-            t.step()
-            for entry in t.section["SHA256"].split("\n"):
+            with open(releasef) as f:
+                t = apt_pkg.TagFile(f)
+                t.step()
+                sha256_section = t.section["SHA256"]
+            for entry in sha256_section.split("\n"):
                 (hash, size, name) = entry.split()
                 f = os.path.join(basepath, name)
                 if not os.path.exists(f):
                     logging.info("ignoring missing '%s'" % f)
                     continue
-                sum = apt_pkg.sha256sum(open(f))
+                with open(f) as fp:
+                    sum = apt_pkg.sha256sum(open(fp))
                 if not (sum == hash):
                     logging.error(
                         "hash sum mismatch expected %s but got %s" % (
@@ -279,9 +277,12 @@
         
         # prepend to the sources.list
         sourceslist = apt_pkg.config.find_file("Dir::Etc::sourcelist")
-        content = open(sourceslist).read()
-        open(sourceslist, "w").write(
-            "# added by the release upgrader\n%s\n%s" % (debline, content))
+        with open(sourceslist) as f:
+            content = f.read()
+        with open(sourceslist, "w") as f:
+            f.write(
+                "# added by the release upgrader\n%s\n%s" %
+                (debline, content))
         self._writeDatabase()
 
         return True

=== modified file 'DistUpgrade/DistUpgradeAufs.py'
--- DistUpgrade/DistUpgradeAufs.py	2014-06-26 06:43:50 +0000
+++ DistUpgrade/DistUpgradeAufs.py	2015-01-20 22:44:23 +0000
@@ -117,10 +117,11 @@
 
 def is_aufs_mount(dir):
     " test if the given dir is already mounted with aufs overlay "
-    for line in open("/proc/mounts"):
-        (device, mountpoint, fstype, options, a, b) = line.split()
-        if device == "none" and fstype == "aufs" and mountpoint == dir:
-            return True
+    with open("/proc/mounts") as f:
+        for line in f:
+            (device, mountpoint, fstype, options, a, b) = line.split()
+            if device == "none" and fstype == "aufs" and mountpoint == dir:
+                return True
     return False
 
 
@@ -182,7 +183,8 @@
     # create something vaguely rollbackable
 
     # get the mount points before the aufs buisiness starts
-    mounts = open("/proc/mounts").read()
+    with open("/proc/mounts") as f:
+        mounts = f.read()
     from .DistUpgradeMain import SYSTEM_DIRS
     systemdirs = SYSTEM_DIRS
     
@@ -234,16 +236,18 @@
     # include sub mounts)
     needs_bind_mount = set()
     needs_bind_mount.add("/var/cache/apt/archives")
-    for line in open("/proc/mounts"):
-        (device, mountpoint, fstype, options, a, b) = line.split()
-        if is_real_fs(fstype) and is_submount(mountpoint, systemdirs):
-            logging.warning("mountpoint %s submount of systemdir" % mountpoint)
-            return False
-        if (fstype != "aufs" and
-                not is_real_fs(fstype) and
-                is_submount(mountpoint, systemdirs)):
-            logging.debug("found %s that needs bind mount", mountpoint)
-            needs_bind_mount.add(mountpoint)
+    with open("/proc/mounts") as f:
+        for line in f:
+            (device, mountpoint, fstype, options, a, b) = line.split()
+            if is_real_fs(fstype) and is_submount(mountpoint, systemdirs):
+                logging.warning("mountpoint %s submount of systemdir" % 
+                                mountpoint)
+                return False
+            if (fstype != "aufs" and
+                    not is_real_fs(fstype) and
+                    is_submount(mountpoint, systemdirs)):
+                logging.debug("found %s that needs bind mount", mountpoint)
+                needs_bind_mount.add(mountpoint)
 
     # aufs mounts do not support stacked filesystems, so
     # if we mount /var we will loose the tmpfs stuff

=== modified file 'DistUpgrade/DistUpgradeCache.py'
--- DistUpgrade/DistUpgradeCache.py	2014-05-02 13:07:42 +0000
+++ DistUpgrade/DistUpgradeCache.py	2015-01-20 22:44:23 +0000
@@ -54,7 +54,7 @@
 def _set_kernel_initrd_size():
     size = estimate_kernel_size_in_boot()
     if size == 0:
-        logging.warn("estimate_kernel_size_in_boot() returned '0'?")
+        logging.warning("estimate_kernel_size_in_boot() returned '0'?")
         size = 28*1024*1024
     # add small safety buffer
     size += 1*1024*1024
@@ -613,7 +613,7 @@
             # check if we got a new kernel (if we are not inside a
             # chroot)
             if inside_chroot():
-                logging.warn("skipping kernel checks because we run inside a chroot")
+                logging.warning("skipping kernel checks because we run inside a chroot")
             else:
                 self.checkForKernel()
 
@@ -849,7 +849,7 @@
             except SystemError as e:
                 # warn here, but don't fail, its possible that meta-packages
                 # conflict (like ubuntu-desktop vs xubuntu-desktop) LP: #775411
-                logging.warn("Can't mark '%s' for upgrade (%s)" % (key, e))
+                logging.warning("Can't mark '%s' for upgrade (%s)" % (key, e))
 
         # check if we have a meta-pkg, if not, try to guess which one to pick
         if not metaPkgInstalled():
@@ -1082,7 +1082,7 @@
                 st = os.statvfs(d)
                 free = st.f_bavail * st.f_frsize
             else:
-                logging.warn("directory '%s' does not exists" % d)
+                logging.warning("directory '%s' does not exists" % d)
                 free = 0
             if fs_id in mnt_map:
                 logging.debug("Dir %s mounted on %s" %

=== modified file 'DistUpgrade/DistUpgradeConfigParser.py'
--- DistUpgrade/DistUpgradeConfigParser.py	2014-06-26 06:30:22 +0000
+++ DistUpgrade/DistUpgradeConfigParser.py	2015-01-20 22:44:23 +0000
@@ -86,7 +86,8 @@
         p = os.path.join(self.datadir, filename)
         if not os.path.exists(p):
             logging.error("getListFromFile: no '%s' found" % p)
-        items = [x.strip() for x in open(p)]
+        with open(p) as f:
+            items = [x.strip() for x in f]
         return [s for s in items if not s.startswith("#") and not s == ""]
 
 

=== modified file 'DistUpgrade/DistUpgradeController.py'
--- DistUpgrade/DistUpgradeController.py	2014-11-10 13:14:32 +0000
+++ DistUpgrade/DistUpgradeController.py	2015-01-20 22:44:23 +0000
@@ -363,7 +363,8 @@
         logging.debug("_pythonSymlinkCheck run")
         if os.path.exists('/usr/share/python/debian_defaults'):
             config = SafeConfigParser()
-            config.readfp(open('/usr/share/python/debian_defaults'))
+            with open('/usr/share/python/debian_defaults') as f:
+                config.readfp(f)
             try:
                 expected_default = config.get('DEFAULT', 'default-version')
             except NoOptionError:
@@ -552,7 +553,8 @@
                         s += "deb http://archive.ubuntu.com/ubuntu %s main restricted" % self.toDist
                         s += "deb http://archive.ubuntu.com/ubuntu %s-updates main restricted" % self.toDist
                         s += "deb http://security.ubuntu.com/ubuntu %s-security main restricted" % self.toDist
-                        open("/etc/apt/sources.list","w").write(s)
+                        with open("/etc/apt/sources.list","w") as f:
+                            f.write(s)
                     break
 
         # this must map, i.e. second in "from" must be the second in "to"
@@ -565,6 +567,20 @@
                                    for x in pockets]
         self.sources_disabled = False
 
+        # Special quirk to remove extras.ubuntu.com
+        new_list = []
+        for entry in self.sources.list[:]:
+            if "/extras.ubuntu.com" in entry.uri:
+                continue
+            if entry.line.startswith(
+                    "## This software is not part of Ubuntu, but is offered by third-party"):
+                continue
+            if entry.line.startswith(
+                    "## developers who want to ship their latest software."):
+                continue
+            new_list.append(entry)
+        self.sources.list = new_list
+
         # look over the stuff we have
         foundToDist = False
         # collect information on what components (main,universe) are enabled for what distro (sub)version
@@ -657,8 +673,7 @@
                     "/security.ubuntu.com" in entry.uri or
                     "%s-security" % self.fromDist in entry.dist or
                     "%s-backports" % self.fromDist in entry.dist or
-                    "/archive.canonical.com" in entry.uri or
-                    "/extras.ubuntu.com" in entry.uri):
+                    "/archive.canonical.com" in entry.uri):
                     validTo = False
                 if entry.dist in toDists:
                     # so the self.sources.list is already set to the new
@@ -1127,7 +1142,8 @@
                 # the previous release, no packages have been installed
                 # yet (LP: #328655, #356781)
                 if os.path.exists("/var/run/ubuntu-release-upgrader-apt-exception"):
-                    e = open("/var/run/ubuntu-release-upgrader-apt-exception").read()
+                    with open("/var/run/ubuntu-release-upgrader-apt-exception") as f:
+                        e = f.read()
                     logging.error("found exception: '%s'" % e)
                     # if its a ordering bug we can cleanly revert but we need to write
                     # a marker for the parent process to know its this kind of error
@@ -1473,14 +1489,13 @@
         # go over the sources.list and try to find a valid mirror
         # that we can use to add the backports dir
         logging.debug("writing prerequists sources.list at: '%s' " % out)
-        outfile = open(out, "w")
         mirrorlines = self._getPreReqMirrorLines(dumb)
-        for line in open(template):
-            template = Template(line)
-            outline = template.safe_substitute(mirror=mirrorlines)
-            outfile.write(outline)
-            logging.debug("adding '%s' prerequists" % outline)
-        outfile.close()
+        with open(out, "w") as outfile, open(template) as infile:
+            for line in infile:
+                template = Template(line)
+                outline = template.safe_substitute(mirror=mirrorlines)
+                outfile.write(outline)
+                logging.debug("adding '%s' prerequists" % outline)
         return True
 
     def getRequiredBackports(self):

=== modified file 'DistUpgrade/DistUpgradeFetcherCore.py'
--- DistUpgrade/DistUpgradeFetcherCore.py	2014-05-02 13:07:42 +0000
+++ DistUpgrade/DistUpgradeFetcherCore.py	2015-01-20 22:44:23 +0000
@@ -246,13 +246,13 @@
             af2
             result = fetcher.run()
             if result != fetcher.RESULT_CONTINUE:
-                logging.warn("fetch result != continue (%s)" % result)
+                logging.warning("fetch result != continue (%s)" % result)
                 return False
             # check that both files are really there and non-null
             for f in [os.path.basename(self.new_dist.upgradeToolSig),
                       os.path.basename(self.new_dist.upgradeTool)]:
                 if not (os.path.exists(f) and os.path.getsize(f) > 0):
-                    logging.warn("file '%s' missing" % f)
+                    logging.warning("file '%s' missing" % f)
                     return False
             return True
         return False

=== modified file 'DistUpgrade/DistUpgradeMain.py'
--- DistUpgrade/DistUpgradeMain.py	2014-05-02 13:07:42 +0000
+++ DistUpgrade/DistUpgradeMain.py	2015-01-20 22:44:23 +0000
@@ -133,7 +133,8 @@
     try:
         s=subprocess.Popen(["lspci","-nn"], stdout=subprocess.PIPE,
                            universal_newlines=True).communicate()[0]
-        open(os.path.join(logdir, "lspci.txt"), "w").write(s)
+        with open(os.path.join(logdir, "lspci.txt"), "w") as f:
+            f.write(s)
     except OSError as e:
         logging.debug("lspci failed: %s" % e)
     

=== modified file 'DistUpgrade/DistUpgradePatcher.py'
--- DistUpgrade/DistUpgradePatcher.py	2014-06-26 06:30:22 +0000
+++ DistUpgrade/DistUpgradePatcher.py	2015-01-20 22:44:23 +0000
@@ -39,12 +39,15 @@
      STATE_EXPECT_DATA) = range(2)
 
     # this is inefficient for big files
-    orig_lines = open(orig, encoding="UTF-8").readlines()
+    with open(orig, encoding="UTF-8") as f:
+        orig_lines = f.readlines()
     start = end = 0
 
     # we start in wait-for-commend state
     state = STATE_EXPECT_COMMAND
-    for line in open(edpatch, encoding="UTF-8"):
+    with open(edpatch, encoding="UTF-8") as f:
+        lines = f.readlines()
+    for line in lines:
         if state == STATE_EXPECT_COMMAND:
             # in commands get rid of whitespace, 
             line = line.strip()
@@ -102,5 +105,6 @@
         md5.update(result.encode("UTF-8"))
         if md5.hexdigest() != result_md5sum:
             raise PatchError("the md5sum after patching is not correct")
-    open(orig, "w", encoding="UTF-8").write(result)
+    with open(orig, "w", encoding="UTF-8") as f:
+        f.write(result)
     return True

=== modified file 'DistUpgrade/DistUpgradeQuirks.py'
--- DistUpgrade/DistUpgradeQuirks.py	2014-12-12 08:10:50 +0000
+++ DistUpgrade/DistUpgradeQuirks.py	2015-01-20 22:44:23 +0000
@@ -263,7 +263,8 @@
         if not os.path.exists(cpuinfo_path):
             logging.error("cannot open %s ?!?" % cpuinfo_path)
             return True
-        cpuinfo = open(cpuinfo_path).read()
+        with open(cpuinfo_path) as f:
+            cpuinfo = f.read()
         # check family
         if re.search("^cpu family\s*:\s*[345]\s*", cpuinfo, re.MULTILINE):
             logging.debug("found cpu family [345], no i686+")
@@ -304,7 +305,7 @@
         try:
             os.kill(1, 0)
         except:
-            logging.warn("no init found")
+            logging.warning("no init found")
             res = self._view.askYesNoQuestion(
                 _("No init available"),
                 _("Your system appears to be a virtualised environment "
@@ -325,8 +326,9 @@
         if not os.path.exists("/proc/cpuinfo"):
             logging.error("cannot open /proc/cpuinfo ?!?")
             return False
-        cpuinfo = open("/proc/cpuinfo")
-        if re.search("^Processor\s*:\s*ARMv[45]", cpuinfo.read(),
+        with open("/proc/cpuinfo") as f:
+            cpuinfo = f.read()
+        if re.search("^Processor\s*:\s*ARMv[45]", cpuinfo,
                      re.MULTILINE):
             return False
         return True
@@ -401,7 +403,8 @@
         # upgrade from Precise will fail if PAE is not in cpu flags
         logging.debug("_checkPae")
         pae = 0
-        cpuinfo = open('/proc/cpuinfo').read()
+        with open('/proc/cpuinfo') as f:
+            cpuinfo = f.read()
         if re.search("^flags\s+:.* pae ", cpuinfo, re.MULTILINE):
             pae = 1
         if not pae:
@@ -420,7 +423,9 @@
         XORG = "/etc/X11/xorg.conf"
         if not os.path.exists(XORG):
             return False
-        for line in open(XORG):
+        with open(XORG) as f:
+            lines = f.readlines()
+        for line in lines:
             s = line.split("#")[0].strip()
             # check for fglrx driver entry
             if (s.lower().startswith("driver") and
@@ -464,7 +469,8 @@
                 logging.debug("already at target hash, skipping '%s'" % path)
                 continue
             elif md5.hexdigest() != md5sum:
-                logging.warn("unexpected target md5sum, skipping: '%s'" % path)
+                logging.warning("unexpected target md5sum, skipping: '%s'"
+                                % path)
                 continue
             # patchable, do it
             from .DistUpgradePatcher import patch
@@ -488,7 +494,7 @@
         # get pkg
         if (pkgname not in self.controller.cache or
                 not self.controller.cache[pkgname].candidate):
-            logging.warn("can not find '%s' in cache")
+            logging.warning("can not find '%s' in cache")
             return False
         pkg = self.controller.cache[pkgname]
         for (module, pciid_list) in \
@@ -570,7 +576,8 @@
                 os.makedirs("/etc/dpkg/dpkg.cfg.d/")
             except OSError:
                 pass
-            open(cfg, "w").write("foreign-architecture %s\n" % foreign_arch)
+            with open(cfg, "w") as f:
+                f.write("foreign-architecture %s\n" % foreign_arch)
 
     def ensure_recommends_are_installed_on_desktops(self):
         """ ensure that on a desktop install recommends are installed
@@ -580,5 +587,5 @@
             if not apt.apt_pkg.config.find_b("Apt::Install-Recommends"):
                 msg = "Apt::Install-Recommends was disabled,"
                 msg += " enabling it just for the upgrade"
-                logging.warn(msg)
+                logging.warning(msg)
                 apt.apt_pkg.config.set("Apt::Install-Recommends", "1")

=== modified file 'DistUpgrade/DistUpgradeView.py'
--- DistUpgrade/DistUpgradeView.py	2014-05-02 13:07:42 +0000
+++ DistUpgrade/DistUpgradeView.py	2015-01-20 22:44:23 +0000
@@ -139,7 +139,7 @@
     # FIXME: workaround issue in libapt/python-apt that does not 
     #        raise a exception if *all* files fails to download
     if status == apt_pkg.STAT_FAILED:
-      logging.warn("update_status: dlFailed on '%s' " % uri)
+      logging.warning("update_status: dlFailed on '%s' " % uri)
       if uri.endswith("Release.gpg") or uri.endswith("Release"):
         # only care about failures from network, not gpg, bzip, those
         # are different issues
@@ -226,7 +226,8 @@
       except Exception as e:
         print("Exception during pm.DoInstall(): ", e)
         logging.exception("Exception during pm.DoInstall()")
-        open("/var/run/ubuntu-release-upgrader-apt-exception","w").write(str(e))
+        with open("/var/run/ubuntu-release-upgrader-apt-exception","w") as f:
+            f.write(str(e))
         os._exit(pm.ResultFailed)
       os._exit(res)
     self.child_pid = pid

=== modified file 'DistUpgrade/DistUpgradeViewKDE.py'
--- DistUpgrade/DistUpgradeViewKDE.py	2014-10-20 10:24:33 +0000
+++ DistUpgrade/DistUpgradeViewKDE.py	2015-01-20 22:44:23 +0000
@@ -864,7 +864,9 @@
           time.sleep(0.01)
 
   if sys.argv[1] == "--show-in-terminal":
-      for c in open(sys.argv[2]).read():
+      with open(sys.argv[2]) as f:
+          chars = f.read()
+      for c in chars:
           view.terminal_text.insertWithTermCodes( c )
           #print(c, ord(c))
           QApplication.processEvents()

=== modified file 'DistUpgrade/DistUpgradeViewNonInteractive.py'
--- DistUpgrade/DistUpgradeViewNonInteractive.py	2014-05-02 13:07:42 +0000
+++ DistUpgrade/DistUpgradeViewNonInteractive.py	2015-01-20 22:44:23 +0000
@@ -133,7 +133,8 @@
         if not os.path.exists(maintainer_script):
             logging.error("can not find failed maintainer script '%s' " % maintainer_script)
             return
-        interp = open(maintainer_script).readline()[2:].strip().split()[0]
+        with open(maintainer_script) as f:
+            interp = f.readline()[2:].strip().split()[0]
         if ("bash" in interp) or ("/bin/sh" in interp):
             debug_opts = ["-ex"]
         elif ("perl" in interp):
@@ -143,7 +144,9 @@
             logging.warning("unknown interpreter: '%s'" % interp)
 
         # check if debconf is used and fiddle a bit more if it is
-        if ". /usr/share/debconf/confmodule" in open(maintainer_script).read():
+        with open(maintainer_script) as f:
+            maintainer_script_text = f.read()
+        if ". /usr/share/debconf/confmodule" in maintainer_script_text:
             environ["DEBCONF_DEBUG"] = "developer"
             environ["DEBIAN_HAS_FRONTEND"] = "1"
             interp = "/usr/share/debconf/frontend"

=== modified file 'check-new-release-gtk'
--- check-new-release-gtk	2014-08-05 06:33:47 +0000
+++ check-new-release-gtk	2015-01-20 22:44:23 +0000
@@ -101,7 +101,7 @@
     # go into nag mode
     if (ignore_dist == new_dist.name and
         meta_release.no_longer_supported is None):
-        logging.warn("found new dist '%s' but it is on the ignore list" % new_dist.name)
+        logging.warning("found new dist '%s' but it is on the ignore list" % new_dist.name)
         sys.exit()
 
     # show alert on unsupported distros
@@ -157,7 +157,7 @@
 
   def timeout(self, user_data):
     if self.new_dist is None:
-      logging.warn("timeout reached, exiting")
+      logging.warning("timeout reached, exiting")
       Gtk.main_quit()
 
 if __name__ == "__main__":

=== modified file 'data/mirrors.cfg'
--- data/mirrors.cfg	2013-04-30 18:17:22 +0000
+++ data/mirrors.cfg	2015-01-20 22:44:23 +0000
@@ -11,8 +11,6 @@
 ftp://ports.ubuntu.com/ubuntu-ports/
 http://old-releases.ubuntu.com/
 ftp://old-releases.ubuntu.com/
-http://extras.ubuntu.com/ubuntu
-ftp://extras.ubuntu.com/ubuntu
 
 #commercial (both urls are valid)
 http://archive.canonical.com

=== modified file 'debian/changelog'
--- debian/changelog	2014-12-12 08:15:22 +0000
+++ debian/changelog	2015-01-20 22:44:23 +0000
@@ -1,3 +1,10 @@
+ubuntu-release-upgrader (1:15.04.4) UNRELEASED; urgency=medium
+
+  * Remove extras.ubuntu.com (LP: #1409555)
+  * Fix resource warnings & logging.warn deprecation warning.
+
+ -- Dimitri John Ledkov <[e-mail]>  Mon, 12 Jan 2015 22:52:47 +0000
+
 ubuntu-release-upgrader (1:15.04.3) vivid; urgency=low
 
   * add compatbility for vte 2.90 (and we need to keep that until

=== added file 'tests/data-sources-list-test/sources.list.extras'
--- tests/data-sources-list-test/sources.list.extras	1970-01-01 00:00:00 +0000
+++ tests/data-sources-list-test/sources.list.extras	2015-01-20 22:44:23 +0000
@@ -0,0 +1,8 @@
+deb http://archive.ubuntu.com/ubuntu feisty main restricted
+deb http://archive.ubuntu.com/ubuntu feisty-updates main restricted
+deb http://security.ubuntu.com/ubuntu/ feisty-security main restricted
+
+## This software is not part of Ubuntu, but is offered by third-party
+## developers who want to ship their latest software.
+deb http://extras.ubuntu.com/ubuntu/ feisty main
+deb-src http://extras.ubuntu.com/ubuntu/ feisty main

=== modified file 'tests/test_cdrom.py'
--- tests/test_cdrom.py	2013-08-05 17:22:35 +0000
+++ tests/test_cdrom.py	2015-01-20 22:44:23 +0000
@@ -26,9 +26,9 @@
 
     def testWriteDatabase(self):
         expect = \
-            "CD::36e3f69081b7d10081d167b137886a71-2 " \
+            "CD::0380987599d9f666b749fbfe29d5b440-2 " \
             "\"Ubuntu 8.10 _Intrepid Ibex_ - Beta amd64 (20080930.4)\";\n" \
-            "CD::36e3f69081b7d10081d167b137886a71-2::Label " \
+            "CD::0380987599d9f666b749fbfe29d5b440-2::Label " \
             "\"Ubuntu 8.10 _Intrepid Ibex_ - Beta amd64 (20080930.4)\";\n"
         p = CURDIR + "/test-data-cdrom/"
         database = CURDIR + "/test-data-cdrom/cdrom.list"
@@ -39,7 +39,8 @@
             os.unlink(database)
         cdrom = AptCdrom(None, p)
         cdrom._writeDatabase()
-        self.assertEqual(expect, open(database).read())
+        with open(database) as f:
+            self.assertEqual(expect, f.read())
 
     def testScanCD(self):
         p = CURDIR + "/test-data-cdrom"
@@ -134,16 +135,17 @@
     def test_comment_out(self):
         tmpdir = tempfile.mkdtemp()
         sourceslist = os.path.join(tmpdir, "sources.list")
-        open(sourceslist, "w")
         apt_pkg.config.set("dir::etc::sourcelist", sourceslist)
         apt_pkg.config.set("dir::state::lists", tmpdir)
         view = Mock()
         cdrom = AptCdrom(view, CURDIR + "/test-data-cdrom")
         cdrom.add()
         cdrom.comment_out_cdrom_entry()
-        for line in open(sourceslist):
+        with open(sourceslist) as f:
+            sourceslines = f.readlines()
+        for line in sourceslines:
             self.assertTrue(line.startswith("#"))
-        self.assertEqual(len(open(sourceslist).readlines()), 2)
+        self.assertEqual(len(sourceslines), 2)
 
 
 if __name__ == "__main__":

=== modified file 'tests/test_prerequists.py'
--- tests/test_prerequists.py	2013-08-05 17:22:35 +0000
+++ tests/test_prerequists.py	2015-01-20 22:44:23 +0000
@@ -89,7 +89,8 @@
         # unset sourceparts
         apt_pkg.config.set("Dir::Etc::sourceparts", tmpdir)
         # write empty status file
-        open(tmpdir + "/status", "w")
+        with open(tmpdir + "/status", "w") as f:
+            f
         os.makedirs(tmpdir + "/lists/partial")
         apt_pkg.config.set("Dir::State", tmpdir)
         apt_pkg.config.set("Dir::State::status", tmpdir + "/status")
@@ -117,7 +118,8 @@
         template = os.path.join(self.testdir, "prerequists-sources.list.in")
         out = os.path.join(tmpdir, "prerequists-sources.list")
         # write empty status file
-        open(tmpdir + "/status", "w")
+        with open(tmpdir + "/status", "w") as f:
+            f
         os.makedirs(tmpdir + "/lists/partial")
         apt_pkg.config.set("Dir::State", tmpdir)
         apt_pkg.config.set("Dir::State::status", tmpdir + "/status")
@@ -143,7 +145,8 @@
             "prerequists-sources.list.in.no_archive_falllback")
         out = os.path.join(tmpdir, "prerequists-sources.list")
         # write empty status file
-        open(tmpdir + "/status", "w")
+        with open(tmpdir + "/status", "w") as f:
+            f
         os.makedirs(tmpdir + "/lists/partial")
         apt_pkg.config.set("Dir::State", tmpdir)
         apt_pkg.config.set("Dir::State::status", tmpdir + "/status")
@@ -168,7 +171,8 @@
                                 "prerequists-sources.list.in.broken")
         out = os.path.join(tmpdir, "prerequists-sources.list")
         # write empty status file
-        open(tmpdir + "/status", "w")
+        with open(tmpdir + "/status", "w") as f:
+            f
         os.makedirs(tmpdir + "/lists/partial")
         apt_pkg.config.set("Dir::State", tmpdir)
         apt_pkg.config.set("Dir::State::status", tmpdir + "/status")
@@ -181,13 +185,14 @@
         self.assertTrue(exp)
 
     def _verifySources(self, filename, expected):
-        sources_list = open(filename).read()
+        with open(filename) as f:
+            sources_list = f.read()
         for l in expected.split("\n"):
             if l:
                 self.assertTrue(l in sources_list,
                                 "expected entry '%s' in '%s' missing, "
                                 "got:\n%s" % (l, filename,
-                                              open(filename).read()))
+                                              sources_list))
 
 if __name__ == "__main__":
     unittest.main()

=== modified file 'tests/test_quirks.py'
--- tests/test_quirks.py	2014-12-12 08:10:50 +0000
+++ tests/test_quirks.py	2015-01-20 22:44:23 +0000
@@ -40,8 +40,10 @@
         """ helper for test_patch to verify that we get the expected result """
         # simple case is foo
         patchdir = CURDIR + "/patchdir/"
-        self.assertFalse("Hello" in open(patchdir + "foo").read())
-        self.assertTrue("Hello" in open(patchdir + "foo_orig").read())
+        with open(patchdir + "foo") as f:
+            self.assertFalse("Hello" in f.read())
+        with open(patchdir + "foo_orig") as f:
+            self.assertTrue("Hello" in f.read())
         md5 = hashlib.md5()
         with open(patchdir + "foo", "rb") as patch:
             md5.update(patch.read())
@@ -62,8 +64,9 @@
             md5.update(patch.read())
         self.assertEqual(md5.hexdigest(), "cddc4be46bedd91db15ddb9f7ddfa804")
         # test that incorrect md5sum after patching rejects the patch
-        self.assertEqual(open(patchdir + "fail").read(),
-                         open(patchdir + "fail_orig").read())
+        with open(patchdir + "fail") as f1, open(patchdir + "fail_orig") as f2:
+            self.assertEqual(f1.read(),
+                             f2.read())
 
     def test_patch(self):
         q = DistUpgradeQuirks(MockController(), MockConfig)

=== modified file 'tests/test_sources_list.py'
--- tests/test_sources_list.py	2014-05-02 12:37:57 +0000
+++ tests/test_sources_list.py	2015-01-20 22:44:23 +0000
@@ -137,6 +137,29 @@
 deb http://archive.canonical.com/ubuntu gutsy partner
 """)
 
+    def test_extras_removal(self):
+        """
+        test removal of extras.ubuntu.com archives
+        """
+        original = os.path.join(self.testdir,
+                                 "sources.list.extras")
+        shutil.copy(original,
+                    os.path.join(self.testdir, "sources.list"))
+        apt_pkg.config.set("Dir::Etc::sourceparts",
+                           os.path.join(self.testdir, "sources.list.d"))
+        v = DistUpgradeViewNonInteractive()
+        d = DistUpgradeController(v, datadir=self.testdir)
+        d.openCache(lock=False)
+        res = d.updateSourcesList()
+        self.assertTrue(res)
+
+        sources_file = apt_pkg.config.find_file("Dir::Etc::sourcelist")
+        self.assertEqual(open(sources_file).read(),"""deb http://archive.ubuntu.com/ubuntu gutsy main restricted
+deb http://archive.ubuntu.com/ubuntu gutsy-updates main restricted
+deb http://security.ubuntu.com/ubuntu/ gutsy-security main restricted
+
+""")
+
     def test_powerpc_transition(self):
         """
         test transition of powerpc to ports.ubuntu.com
@@ -431,7 +454,8 @@
 
     def _verifySources(self, expected):
         sources_file = apt_pkg.config.find_file("Dir::Etc::sourcelist")
-        sources_list = open(sources_file).read()
+        with open(sources_file) as f:
+            sources_list = f.read()
         for l in expected.split("\n"):
             self.assertTrue(
                 l in sources_list.split("\n"),

=== modified file 'tests/test_xorg_fix_intrepid.py'
--- tests/test_xorg_fix_intrepid.py	2014-05-03 06:37:43 +0000
+++ tests/test_xorg_fix_intrepid.py	2015-01-20 22:44:23 +0000
@@ -24,18 +24,23 @@
     def test_simple(self):
         shutil.copy(self.ORIG, self.NEW)
         replace_driver_from_xorg("fglrx", "ati", self.NEW)
-        self.assertEqual(open(self.NEW).read(), open(self.ORIG).read())
+        with open(self.NEW) as n, open(self.ORIG) as o:
+            self.assertEqual(n.read(), o.read())
 
     def test_remove(self):
         shutil.copy(self.FGLRX, self.NEW)
-        self.assertTrue("fglrx" in open(self.NEW).read())
+        with open(self.NEW) as f:
+            self.assertTrue("fglrx" in f.read())
         replace_driver_from_xorg("fglrx", "ati", self.NEW)
-        self.assertFalse("fglrx" in open(self.NEW).read())
+        with open(self.NEW) as f:
+            self.assertFalse("fglrx" in f.read())
 
     def test_omment(self):
         shutil.copy(self.FGLRX, self.NEW)
         comment_out_driver_from_xorg("fglrx", self.NEW)
-        for line in open(self.NEW):
+        with open(self.NEW) as n:
+            lines = n.readlines()
+        for line in lines:
             if re.match('^#.*Driver.*fglrx', line):
                 import logging
                 logging.info("commented out line found")

=== modified file 'utils/update_mirrors.py'
--- utils/update_mirrors.py	2014-06-26 06:24:06 +0000
+++ utils/update_mirrors.py	2015-01-20 22:44:23 +0000
@@ -5,18 +5,19 @@
 
 # read what we have
 current_mirrors = set()
-for line in open(sys.argv[1], "r"):
-    current_mirrors.add(line.strip())
+with open(sys.argv[1], "r") as f:
+    for line in f:
+        current_mirrors.add(line.strip())
 
     
-outfile = open(sys.argv[1], "a")
 d = feedparser.parse("https://launchpad.net/ubuntu/+archivemirrors-rss")
 
 #import pprint
 #pp  = pprint.PrettyPrinter(indent=4)
 #pp.pprint(d)
 
-for entry in d.entries:
-    for link in entry.links:
-        if link.href not in current_mirrors:
-            outfile.write(link.href + "\n")
+with open(sys.argv[1], "a") as outfile:
+    for entry in d.entries:
+        for link in entry.links:
+            if link.href not in current_mirrors:
+                outfile.write(link.href + "\n")