Merge lp:~parthm/bzr/300062-2.2-bad-pattern-error-part-2 into lp:bzr/2.2

[Parth Malwankar]

'bzr ignore' fails on bad pattern. bad pattern error messages now shows the faulting pattern.

This fix is towards bug #300062
It does the following:
1. `bzr ignore PATTERNS` now fails with an error if an invalid pattern is provided.
2. For pattern errors, the faulting pattern is displayed to the user.

E.g.
[a1234]% ~/src/bzr.dev/300062-2.2-bad-pattern-error-part-2/bzr st
bzr: ERROR: Invalid pattern(s) found. File ~/.bazaar/ignore or .bzrignore contains error(s).
  RE:[
[a1234]% ~/src/bzr.dev/300062-2.2-bad-pattern-error-part-2/bzr ignore "RE:*.cpp"
bzr: error: Invalid ignore pattern(s) found.
  RE:*.cpp
bzr: ERROR: Invalid pattern(s) found.
[a1234]%

This patch is the same as the https://code.launchpad.net/~parthm/bzr/300062-bad-pattern-error-part-2/+merge/29467 but is targeted to lp:bzr/2.2
If also contains update for e.message -> e.msg fix that came up since the previous patch was created.

[John A Meinel (jameinel)]

79 - self._add_patterns(ext_patterns,_sub_extension,
80 - prefix=r'(?:.*/)?(?!.*/)(?:.*\.)')
81 - self._add_patterns(base_patterns,_sub_basename,
82 - prefix=r'(?:.*/)?(?!.*/)')
83 - self._add_patterns(path_patterns,_sub_fullpath)
84 + pattern_lists[Globster.identify(pat)].append(pat)
85 + for pattern_type, patterns in pattern_lists.iteritems():
86 + self._add_patterns(patterns,
87 + Globster.translators[pattern_type],
88 + Globster.prefixes[pattern_type])

^- I think factoring the patterns into a dict indirected code is interesting. I'm a bit concerned about:
for pattern_type, patterns in pattern_lists.iteritems():

I'm pretty sure we explicitly intended the order of 'check extensions' then 'check basename' then 'check fullpath'. So that we can do checks on just the little bit of the content before we have to check against the whole content. I don't have any explicit proof of this, but turning the current deterministic ordering with a random one doesn't seem ideal.

otherwise the change seems fine to me. I may be over-concerned, but I do believe that 'is_ignored()' is a fair amount of overhead in the 'initial add of 30k files', and I'd like us to at least be aware of it. I do wonder about changing the internals so that instead of using '?:.*/)?(?!.*/)) to filter out everything but the basename for some patterns, if we wouldn't be better off knowing that this is a basename-only pattern and just supply that to the match.

I'm certainly willing to discuss it, but the easiest way to move forward is to just make the order of _add_patterns deterministic by iterating over a fixed ordering.

review: Needs Fixing

[Parth Malwankar (parthm)]

> ^- I think factoring the patterns into a dict indirected code is interesting.
> I'm a bit concerned about:
> for pattern_type, patterns in pattern_lists.iteritems():
>
> I'm pretty sure we explicitly intended the order of 'check extensions' then
> 'check basename' then 'check fullpath'. So that we can do checks on just the
> little bit of the content before we have to check against the whole content. I
> don't have any explicit proof of this, but turning the current deterministic
> ordering with a random one doesn't seem ideal.
>

Makes sense. I have updated the patch to make _add_patterns order deterministic.

>
> otherwise the change seems fine to me. I may be over-concerned, but I do
> believe that 'is_ignored()' is a fair amount of overhead in the 'initial add
> of 30k files', and I'd like us to at least be aware of it. I do wonder about
> changing the internals so that instead of using '?:.*/)?(?!.*/)) to filter out
> everything but the basename for some patterns, if we wouldn't be better off
> knowing that this is a basename-only pattern and just supply that to the
> match.
>

Filed bug #607258 to track this.
Thanks for the review.

>
> I'm certainly willing to discuss it, but the easiest way to move forward is to
> just make the order of _add_patterns deterministic by iterating over a fixed
> ordering.

[Vincent Ladeuil (vila)]

43 + TYPE_FULLPATH = 1
44 + TYPE_BASENAME = 2
45 + TYPE_EXTENSION = 3

This looks a bit C-ish, why don't you just use regular keys ('fullpath', 'basename' and 'extension') ?

From a design point of view it looks like you're defining several set of objects associated with the same key which sounds like you want a dict or even a registry...

Also, while this is certainly out of scope for this proposal, keep in mind that some plugins want to define their own '.xxx-ignore' files and our current API is already a pain to use, it would be nice if we can help them by providing the right set of function/classes to process a set of paths against a given set of patterns coming from a file (or several in bzr case...).

[Parth Malwankar (parthm)]

> 43 + TYPE_FULLPATH = 1
> 44 + TYPE_BASENAME = 2
> 45 + TYPE_EXTENSION = 3
>
> This looks a bit C-ish, why don't you just use regular keys ('fullpath',
> 'basename' and 'extension') ?
>
> From a design point of view it looks like you're defining several set of
> objects associated with the same key which sounds like you want a dict or even
> a registry...
>

I have updated the patch to use a dict of dict with strings. Its much cleaner now.

> Also, while this is certainly out of scope for this proposal, keep in mind
> that some plugins want to define their own '.xxx-ignore' files and our current
> API is already a pain to use, it would be nice if we can help them by
> providing the right set of function/classes to process a set of paths against
> a given set of patterns coming from a file (or several in bzr case...).

Makes sense. I think it may be possible to build an API on top of the
current implementation to make it easy. I will look into bzr-upload as
suggested by you on IRC.

Thanks for the review.

[Vincent Ladeuil (vila)]

Yup, far cleaner.

I think you've also addressed jam's concerns so good to land for me.

review: Approve

[Parth Malwankar (parthm)]

sent to pqm by email

=== modified file 'NEWS'
--- NEWS	2010-07-20 15:17:58 +0000
+++ NEWS	2010-07-21 10:19:44 +0000
@@ -31,6 +31,11 @@
 Bug Fixes
 *********
 
+* ``bzr ignore PATTERNS`` exits with error if a bad pattern is supplied.
+  ``InvalidPattern`` exception error message now shows faulting
+  regular expression.
+  (Parth Malwankar #300062)
+
 * Configuration files in ``${BZR_HOME}`` are now written in an atomic
   way which should help avoid problems with concurrent writers.
   (Vincent Ladeuil, #525571)

=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py	2010-07-18 14:22:34 +0000
+++ bzrlib/builtins.py	2010-07-21 10:19:44 +0000
@@ -2712,6 +2712,14 @@
                 "NAME_PATTERN or --default-rules.")
         name_pattern_list = [globbing.normalize_pattern(p)
                              for p in name_pattern_list]
+        bad_patterns = ''
+        for p in name_pattern_list:
+            if not globbing.Globster.is_pattern_valid(p):
+                bad_patterns += ('\n  %s' % p)
+        if bad_patterns:
+            msg = ('Invalid ignore pattern(s) found. %s' % bad_patterns)
+            ui.ui_factory.show_error(msg)
+            raise errors.InvalidPattern('')
         for name_pattern in name_pattern_list:
             if (name_pattern[0] == '/' or
                 (len(name_pattern) > 1 and name_pattern[1] == ':')):

=== modified file 'bzrlib/globbing.py'
--- bzrlib/globbing.py	2010-07-09 16:16:11 +0000
+++ bzrlib/globbing.py	2010-07-21 10:19:44 +0000
@@ -179,24 +179,41 @@
     so are matched first, then the basename patterns, then the fullpath
     patterns.
     """
+    # We want to _add_patterns in a specific order (as per type_list below)
+    # starting with the shortest and going to the longest.
+    # As some Python version don't support ordered dicts the list below is
+    # used to select inputs for _add_pattern in a specific order.
+    pattern_types = [ "extension", "basename", "fullpath" ]
+
+    pattern_info = {
+        "extension" : {
+            "translator" : _sub_extension,
+            "prefix" : r'(?:.*/)?(?!.*/)(?:.*\.)'
+        },
+        "basename" : {
+            "translator" : _sub_basename,
+            "prefix" : r'(?:.*/)?(?!.*/)'
+        },
+        "fullpath" : {
+            "translator" : _sub_fullpath,
+            "prefix" : r''
+        },
+    }
+
     def __init__(self, patterns):
         self._regex_patterns = []
-        path_patterns = []
-        base_patterns = []
-        ext_patterns = []
+        pattern_lists = {
+            "extension" : [],
+            "basename" : [],
+            "fullpath" : [],
+        }
         for pat in patterns:
             pat = normalize_pattern(pat)
-            if pat.startswith(u'RE:') or u'/' in pat:
-                path_patterns.append(pat)
-            elif pat.startswith(u'*.'):
-                ext_patterns.append(pat)
-            else:
-                base_patterns.append(pat)
-        self._add_patterns(ext_patterns,_sub_extension,
-            prefix=r'(?:.*/)?(?!.*/)(?:.*\.)')
-        self._add_patterns(base_patterns,_sub_basename,
-            prefix=r'(?:.*/)?(?!.*/)')
-        self._add_patterns(path_patterns,_sub_fullpath)
+            pattern_lists[Globster.identify(pat)].append(pat)
+        pi = Globster.pattern_info
+        for t in Globster.pattern_types:
+            self._add_patterns(pattern_lists[t], pi[t]["translator"],
+                pi[t]["prefix"])
 
     def _add_patterns(self, patterns, translator, prefix=''):
         while patterns:
@@ -221,10 +238,50 @@
             # the combined pattern we sent to regex. Instead we indicate to
             # the user that an ignore file needs fixing.
             mutter('Invalid pattern found in regex: %s.', e.msg)
-            e.msg = "File ~/.bazaar/ignore or .bzrignore contains errors."
+            e.msg = "File ~/.bazaar/ignore or .bzrignore contains error(s)."
+            bad_patterns = ''
+            for _, patterns in self._regex_patterns:
+                for p in patterns:
+                    if not Globster.is_pattern_valid(p):
+                        bad_patterns += ('\n  %s' % p)
+            e.msg += bad_patterns
             raise e
         return None
 
+    @staticmethod
+    def identify(pattern):
+        """Returns pattern category.
+
+        :param pattern: normalized pattern.
+        Identify if a pattern is fullpath, basename or extension
+        and returns the appropriate type.
+        """
+        if pattern.startswith(u'RE:') or u'/' in pattern:
+            return "fullpath"
+        elif pattern.startswith(u'*.'):
+            return "extension"
+        else:
+            return "basename"
+
+    @staticmethod
+    def is_pattern_valid(pattern):
+        """Returns True if pattern is valid.
+
+        :param pattern: Normalized pattern.
+        is_pattern_valid() assumes pattern to be normalized.
+        see: globbing.normalize_pattern
+        """
+        result = True
+        translator = Globster.pattern_info[Globster.identify(pattern)]["translator"]
+        tpattern = '(%s)' % translator(pattern)
+        try:
+            re_obj = re.compile(tpattern, re.UNICODE)
+            re_obj.search("") # force compile
+        except errors.InvalidPattern, e:
+            result = False
+        return result
+
+
 class ExceptionGlobster(object):
     """A Globster that supports exception patterns.
     
@@ -272,14 +329,9 @@
         self._regex_patterns = []
         for pat in patterns:
             pat = normalize_pattern(pat)
-            if pat.startswith(u'RE:') or u'/' in pat:
-                self._add_patterns([pat], _sub_fullpath)
-            elif pat.startswith(u'*.'):
-                self._add_patterns([pat], _sub_extension,
-                    prefix=r'(?:.*/)?(?!.*/)(?:.*\.)')
-            else:
-                self._add_patterns([pat], _sub_basename,
-                    prefix=r'(?:.*/)?(?!.*/)')
+            t = Globster.identify(pat)
+            self._add_patterns([pat], Globster.pattern_info[t]["translator"],
+                Globster.pattern_info[t]["prefix"])
 
 
 _slashes = re.compile(r'[\\/]+')

=== modified file 'bzrlib/tests/blackbox/test_ignore.py'
--- bzrlib/tests/blackbox/test_ignore.py	2010-06-11 07:32:12 +0000
+++ bzrlib/tests/blackbox/test_ignore.py	2010-07-21 10:19:44 +0000
@@ -162,3 +162,19 @@
         tree = self.make_branch_and_tree('a')
         self.run_bzr(['ignore', '--directory=a', 'README'])
         self.check_file_contents('a/.bzrignore', 'README\n')
+
+    def test_ignored_invalid_pattern(self):
+        """Ensure graceful handling for invalid ignore pattern.
+
+        Test case for #300062.
+        Invalid pattern should show clear error message.
+        Invalid pattern should not be added to .bzrignore file.
+        """
+        tree = self.make_branch_and_tree('.')
+        out, err = self.run_bzr(['ignore', 'RE:*.cpp', 'foo', 'RE:['], 3)
+        self.assertEqual(out, '')
+        self.assertContainsRe(err,
+            'Invalid ignore pattern.*RE:\*\.cpp.*RE:\[', re.DOTALL)
+        self.assertNotContainsRe(err, 'foo', re.DOTALL)
+        self.assertFalse(os.path.isfile('.bzrignore'))
+

=== modified file 'bzrlib/tests/per_workingtree/test_is_ignored.py'
--- bzrlib/tests/per_workingtree/test_is_ignored.py	2010-02-17 17:11:16 +0000
+++ bzrlib/tests/per_workingtree/test_is_ignored.py	2010-07-21 10:19:44 +0000
@@ -22,6 +22,16 @@
 
 class TestIsIgnored(TestCaseWithWorkingTree):
 
+    def _set_user_ignore_content(self, ignores):
+        """Create user ignore file and set its content to ignores."""
+        config.ensure_config_dir_exists()
+        user_ignore_file = config.user_ignore_config_filename()
+        f = open(user_ignore_file, 'wb')
+        try:
+            f.write(ignores)
+        finally:
+            f.close()
+
     def test_is_ignored(self):
         tree = self.make_branch_and_tree('.')
         # this will break if a tree changes the ignored format. That is fine
@@ -46,6 +56,11 @@
                            '#comment\n'
                            ' xx \n' # whitespace
             )])
+        # We set user ignore file to contain '' to avoid patterns from
+        # user ignore being used instead of bzrignore. For .e.g. If we
+        # don't do this 'foo.~1~' will match '*~' default user ignore
+        # pattern instead of '*.~*' from bzr ignore as we expect below.
+        self._set_user_ignore_content('')
         # is_ignored returns the matching ignore regex when a path is ignored.
         # we check some expected matches for each rule, and one or more
         # relevant not-matches that look plausible as cases for bugs.
@@ -119,19 +134,16 @@
 
         config.ensure_config_dir_exists()
         user_ignore_file = config.user_ignore_config_filename()
-        f = open(user_ignore_file, 'wb')
-        try:
-            f.write('*.py[co]\n'
-                    './.shelf\n'
-                    '# comment line\n'
-                    '\n' #Blank line
-                    '\r\n' #Blank dos line
-                    ' * \n' #Trailing and suffix spaces
-                    'crlf\r\n' # dos style line
-                    '*\xc3\xa5*\n' # u'\xe5'.encode('utf8')
-                    )
-        finally:
-            f.close()
+        self._set_user_ignore_content(
+            '*.py[co]\n'
+            './.shelf\n'
+            '# comment line\n'
+            '\n' #Blank line
+            '\r\n' #Blank dos line
+            ' * \n' #Trailing and suffix spaces
+            'crlf\r\n' # dos style line
+            '*\xc3\xa5*\n' # u'\xe5'.encode('utf8')
+            )
 
         # Rooted
         self.assertEqual('./.shelf', tree.is_ignored('.shelf'))

=== modified file 'bzrlib/tests/test_globbing.py'
--- bzrlib/tests/test_globbing.py	2010-07-09 16:16:11 +0000
+++ bzrlib/tests/test_globbing.py	2010-07-21 10:19:44 +0000
@@ -15,6 +15,8 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 
+import re
+
 from bzrlib import errors
 from bzrlib.globbing import (
     Globster,
@@ -311,10 +313,11 @@
 
     def test_bad_pattern(self):
         """Ensure that globster handles bad patterns cleanly."""
-        patterns = [u'RE:[']
+        patterns = [u'RE:[', u'/home/foo', u'RE:*.cpp']
         g = Globster(patterns)
-        e = self.assertRaises(errors.InvalidPattern, g.match, 'foo')
-        self.assertContainsRe(e.msg, "File.*ignore.*contains errors")
+        e = self.assertRaises(errors.InvalidPattern, g.match, 'filename')
+        self.assertContainsRe(e.msg,
+            "File.*ignore.*contains error.*RE:\[.*RE:\*\.cpp", flags=re.DOTALL)
 
 
 class TestExceptionGlobster(TestCase):