bug search fails to find results when punctuation is adjacent to regular text in the document (e.g. '"from"', '<div>')

I tried the search with 'sqlobject' and 'sqlobject.select' on the https://launchpad.net/products/launchpad/+bugs and got some results, but bug 3096 didn't show up.

Weird.

Probably related: searching for "2.6.20-12" does not produce relevant results, I'd expect to find Bug #94083 and #93648, which have that exact string in the title, but they don't appear.

I still believe this is high given its complete breakage of basic queries.

Escalated by Kate

I've assigned this to myself for the purposes of investigation; since Yellow Squad is but a couple of weeks (excepting the Christmas break and the Thunderepic in January) from feature rotation I might not be able to fix this in the time available, but I might at least be able to shed some more light on it.

Another example: In the Mixxx project, searching for '1.10 crash' or even '1.10 crashes' does not include a bug titled 'mixxx.exe 1.10 Beta1 immediately crashes at startup' in its results, I'm guessing due to the presence of the period within the version number.

_build_search_text_clause() in bugtasksearch.py generates these search
clauses:

    SQL("BugTaskFlat.fti @@ ftq(?)", params=(searchtext,))

So, no tokenisation in Python.

I added two bugs with some of "bad search terms" mentioned above to my
local launchpad_dev DB:

select bug.description, bugtaskflat.fti
    from bugtaskflat, bug where bug.id=bugtaskflat.bug and bug.id>=16;

row 16:
  description: from "from" foo "bar" <div> community-contributions.py SQLObject.select 2.6.20-12 1.10 crash
  fti: '-12':8 '1.10':9 '2.6.20':7 'bar':4 'community-contributions.py':5 'crash':10 'foo':3 'sqlobject.select':6

row 17:
  description: from "from" foo "bar" div community-contributions.py SQLObject.select 2.6.20-12 1.10 crash
  fti: '-12':10 '1.10':11 '2.6.20':9 'bar':5 'community-contributions.py':7 'crash':12 'div':6 'foo':4 'sqlobject.select':8 'xxx':1B

The only difference between these rows is '<div>' vs. 'div'

Neither '<div>' nor 'div' appear in the first FTI: it seems that the FTI
tokenizer simply drops anything between '<' and '>'.

search queries:

select bug from bugtaskflat where fti @@ ftq('sqlobject.select');
-> no result.

select ftq('sqlobject.select');
                    ftq
--------------------------------------------
 'sqlobject' & 'select' | 'sqlobjectselect'
(1 row)

So, ftq('sqlobject.select') generates a reasonable expression -- but the
full text index stores 'sqlobject.select' instead of two words 'sqlobject'
and 'select'.

The query below works though:

select bug from bugtaskflat where fti @@ 'sqlobject.select';

 bug
-----
  16
  17

A search for "community-contributions.py" has the same problem: The index
stores the complete word, but:

select ftq('community-contributions.py');
                              ftq
---------------------------------------------------------------
 'communiti' & 'contribut' & 'py' | 'communitycontributionspi'
(1 row)

"From" is probably in the set of stop words. I am not sure if it makes
sense to remove "from" from this set...

select bug from bugtaskflat where fti @@ ftq('2.6.20-12');
 bug
-----
(0 rows)

select ftq('2.6.20-12'); ftq
-------------------------------------------------
 ( '2' & '6' | '26' ) & ( '20' & '12' | '2012' )
(1 row)

while the fTI stores these numbers:

  fti: '-12':8 '1.10':9 '2.6.20':7 '

There at least two issues:

1. Treatment of '<' and '>'.

Postgres' text search machinery seems to simply ignore HTML/XML nodes,
at least in the config we use:

select to_tsvector('<div>');
 to_tsvector
-------------

(1 row)

select to_tsquery('<div>');
NOTICE: text-search query contains only stop words or doesn't contain lexemes, ignored
 to_tsquery
------------

(1 row)

But the function ftq() (part of our schema definition) strips '<' and '>'
before it generates the query object:

select ftq('<div>');
  ftq
-------
 'div'

2. Treatment of punctuation. From the definition of ftq():

        punctuation = r"[^\w\s\-\&\|\!\(\)']"
        query = re.sub(r"(?u)(\w)%s+(\w)" % (punctuation,), r"\1-\2", query)

This replaces 'object.attr' with 'object-attr' (as well as 'object<attr'
-> 'object-attr'), and this is later replaced with
'((object&attr)|(objectattr)':

        def hyphen_repl(match):
            bits = match.group(0).split("-")
            return "((%s)|%s)" % ("&".join(bits), "".join(bits))
        query = re.sub(r"(?u)\b\w+-[\w\-]+\b", hyphen_repl, query)

but the FTI stores the original 'object.attr'. It might make sense
to search for 'object' & 'attr' or for 'objectattr' but we should
additionally keep the original 'object.attr' in the query.

Note that this is some of our oldest code, based on tsearch2 with
PostgreSQL 8.0. Since then, tsearch has been moved into PostgreSQL
core, improved and documented (Chapter 12 of the PostgreSQL 9.1
manual, Full Text Search). Many of the issues may be fixable with
properly configuration (stop word lists etc.), and new facilities may
make it possible to simplify this old code (pluggable parsers etc.)

--
Stuart Bishop <email address hidden>

see also bug 1015511 and bug 1015519

Fixed in stable r15464 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15464>.

Tagged as qa-ok. The issues described here and in bug 1015511 and bug 1015519 are not fixed though.

Thank you so much, for fixing such a longstanding bug. (Excuse me you consider this confirmation just bug spam.) It affected me recently, when LP did not find existing reports with this subject:

  package manpages 3.35-0.1ubuntu1 failed to install/upgrade: trying to overwrite '/usr/share/man/man1/getent.1.gz', which is also in package libc-bin 2.15-0ubuntu15

(That was bug 1017289.) I had to browse the list of recent bugs for that package, to find the report, and mark duplicates. I tested an upload of the same crashfile, today, and LP directed me to the existing bug. To me this is the difference between the bug reporter feeling lost in a maze of twisty little passages, or, feeling well-guided, and that's the difference between participating or giving up.

Seeing a bug that's a million numbers old, getting fixed, is a morale booster in itself, too. A million thanks. :)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29.06.2012 21:31, Edward Donovan wrote:
> Thank you so much, for fixing such a longstanding bug. (Excuse me
> you consider this confirmation just bug spam.) It affected me
> recently,

Bug reports are for conversations about bugs, and conversations have
also some social aspects. So your comment is definitely not spam.

> when LP did not find existing reports with this subject:
>
> package manpages 3.35-0.1ubuntu1 failed to install/upgrade: trying
> to overwrite '/usr/share/man/man1/getent.1.gz', which is also in
> package libc-bin 2.15-0ubuntu15
>
> (That was bug 1017289.) I had to browse the list of recent bugs
> for that package, to find the report, and mark duplicates. I
> tested an upload of the same crashfile, today, and LP directed me
> to the existing bug. To me this is the difference between the bug
> reporter feeling lost in a maze of twisty little passages, or,
> feeling well-guided, and that's the difference between
> participating or giving up.
>
> Seeing a bug that's a million numbers old, getting fixed, is a
> morale booster in itself, too. A million thanks. :)

welcome :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFP8VLyekBPhm8NrtARAkQSAKCOXZ495PKgFRLebAfhSKEivrQcWACgh0di
rwj1AgIRUb+cp3wcqQDbJic=
=omP4
-----END PGP SIGNATURE-----