Merge ~smoser/curtin:cleanup/fix-tip-pycodestyle-invalid-escape-sequences into curtin:master

[Scott Moser]

pycodestyle: Fix invalid escape sequences in string literals.

A bit of information from python doc that I got by never having known:
  String literals may optionally be prefixed with a letter `r' or `R';
  such strings are called raw strings and use different rules for
  backslash escape sequences.
  ...
  Unless an `r' or `R' prefix is present, escape sequences in strings are
  interpreted according to rules similar to those used by Standard C.

So basically, any use of \ not followed by one of [\'"abfnrtv]
or \ooo (octal) \xhh (hex) or a newline is invalid. This is most
commonly seen for us in regex. To solve, you either:
 a.) use a raw string r'...'
 b.) correctly escape the \ that was not intended to be interpreted.

Python has deprecated these invalid string literals now
  https://bugs.python.org/issue27364
and pycodestyle is identifying them with a W605 warning.
  https://github.com/PyCQA/pycodestyle/pull/676

[Ryan Harper (raharper)]

IIRC, if we want to use \ escapes, then the string must be a raw string. There are few cases where you drop the r'' and others were you add it.

At least for regex, I think we can consistently use r'', and for other causes (we pass strings to a shell) we've escaped things like $; those, IIRC, don't need escaping unless we're running them through shell.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:e23ce89821977fa13f3f4ffdcb686660752c9449
https://jenkins.ubuntu.com/server/job/curtin-ci/893/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/893
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/893
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/893
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/893

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/893/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c09d36659333650c5f526365f2a77d0421664ecd
https://jenkins.ubuntu.com/server/job/curtin-ci/895/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/895
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/895
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/895
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/895

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/895/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

LGTM. +1 with whichever direction you take on that last regex change.

review: Approve

f9f2277... by Scott Moser on 2018-04-13

take ryan's suggestion for ua_val replacement

[Scott Moser (smoser)]

i added ryan's last suggestion.
and i'm going to merge to fix tip-pycodestyle.

review: Approve

[Scott Moser (smoser)]

Thank you for your merge proposal.

Your branch has been set to 'Work in progress'.
Please set the branch back to 'Needs Review' after resolving the issues below.

Thanks again,
Your friendly neighborhood curtin robot.

Please fix the following issues:
------------------------------
Commit message lints:
 - Line #4 has 1 too many characters. Line starts with: " such strings are called"...
------------------------------

For more information, see commit message guidelines at
https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#do-these-things-for-each-feature-or-bug

review: Needs Fixing

[Scott Moser (smoser)]

adjusted message.

review: Approve

[Scott Moser (smoser)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=2ee52452

diff --git a/curtin/block/mdadm.py b/curtin/block/mdadm.py
index b0f5591..2e73e71 100644
--- a/curtin/block/mdadm.py
+++ b/curtin/block/mdadm.py
@@ -483,7 +483,7 @@ def __mdadm_detail_to_dict(input):
     '''
     data = {}
 
-    device = re.findall('^(\/dev\/[a-zA-Z0-9-\._]+)', input)
+    device = re.findall(r'^(\/dev\/[a-zA-Z0-9-\._]+)', input)
     if len(device) == 1:
         data.update({'device': device[0]})
     else:
@@ -491,9 +491,8 @@ def __mdadm_detail_to_dict(input):
 
     #  FIXME: probably could do a better regex to match the LHS which
     #         has one, two or three words
-    for f in re.findall('(\w+|\w+\ \w+|\w+\ \w+\ \w+)' +
-                        '\ \:\ ([a-zA-Z0-9\-\.,: \(\)=\']+)',
-                        input, re.MULTILINE):
+    rem = r'(\w+|\w+\ \w+|\w+\ \w+\ \w+)\ \:\ ([a-zA-Z0-9\-\.,: \(\)=\']+)'
+    for f in re.findall(rem, input, re.MULTILINE):
         key = f[0].replace(' ', '_').lower()
         val = f[1]
         if key in data:
diff --git a/curtin/commands/curthooks.py b/curtin/commands/curthooks.py
index 9e51a65..422fb63 100644
--- a/curtin/commands/curthooks.py
+++ b/curtin/commands/curthooks.py
@@ -336,7 +336,7 @@ def setup_grub(cfg, target):
                 export LANG=C;
                 for d in "$@"; do
                     sgdisk "$d" --print |
-                        awk "\$6 == prep { print d \$1 }" "d=$d" prep=4100
+                        awk '$6 == prep { print d $1 }' "d=$d" prep=4100
                 done
                 """)
             try:
diff --git a/tests/vmtests/__init__.py b/tests/vmtests/__init__.py
index 64fc867..b5aa538 100644
--- a/tests/vmtests/__init__.py
+++ b/tests/vmtests/__init__.py
@@ -1529,7 +1529,7 @@ def check_install_log(install_log):
     # regexps expected in curtin output
     install_pass = INSTALL_PASS_MSG
     install_fail = "({})".format("|".join([
-                   'Installation\ failed',
+                   'Installation failed',
                    'ImportError: No module named.*',
                    'Unexpected error while running command',
                    'E: Unable to locate package.*',
diff --git a/tests/vmtests/test_pollinate_useragent.py b/tests/vmtests/test_pollinate_useragent.py
index c076fbc..abd6daf 100644
--- a/tests/vmtests/test_pollinate_useragent.py
+++ b/tests/vmtests/test_pollinate_useragent.py
@@ -24,7 +24,7 @@ class TestPollinateUserAgent(VMBaseClass):
         self.output_files_exist(["pollinate_print_user_agent"])
         agent_values = self.load_collect_file("pollinate_print_user_agent")
         if len(agent_values) == 0:
-            pollver = re.search('pollinate\s(?P<version>\S+)',
+            pollver = re.search(r'pollinate\s(?P<version>\S+)',
                                 self.load_collect_file("debian-packages.txt"))
             msg = ("pollinate client '%s' does not support "
                    "--print-user-agent'" % pollver.groupdict()['version'])
@@ -45,7 +45,7 @@ class TestPollinateUserAgent(VMBaseClass):
             """
             ua_val = line.split()[0]
             # escape + and . that are likely in maas/curtin version strings
-            regex = r'%s' % ua_val.replace('+', '\+').replace('.', '\.')
+            regex = '%s' % ua_val.replace('+', r'\+').replace('.', r'\.')
             hit = re.search(regex, agent_values)
             self.assertIsNotNone(hit)
             self.assertEqual(ua_val, hit.group())