Merge lp:~boiko/history-service/store_normalized_phone_numbers into lp:history-service

Proposed by Gustavo Pichorim Boiko
Status: Merged
Approved by: Tiago Salem Herrmann
Approved revision: 214
Merged at revision: 211
Proposed branch: lp:~boiko/history-service/store_normalized_phone_numbers
Merge into: lp:history-service
Prerequisite: lp:~boiko/history-service/custom_functions_in_schema_generation
Diff against target: 422 lines (+137/-64)
12 files modified
historyprivate/utils.cpp (+15/-0)
historyprivate/utils_p.h (+1/-0)
plugins/sqlite/CMakeLists.txt (+2/-1)
plugins/sqlite/schema/CMakeLists.txt (+1/-1)
plugins/sqlite/schema/v12.sql (+2/-0)
plugins/sqlite/sqlitedatabase.cpp (+21/-1)
plugins/sqlite/sqlitehistoryplugin.cpp (+69/-46)
plugins/sqlite/sqlitehistoryplugin.h (+1/-1)
src/phoneutils.cpp (+18/-9)
src/phoneutils_p.h (+2/-1)
tests/libhistoryservice/ManagerTest.cpp (+4/-3)
tests/libhistoryservice/PhoneUtilsTest.cpp (+1/-1)
To merge this branch: bzr merge lp:~boiko/history-service/store_normalized_phone_numbers
Reviewer Review Type Date Requested Status
Tiago Salem Herrmann (community) Approve
PS Jenkins bot continuous-integration Approve
Review via email: mp+269986@code.launchpad.net

This proposal supersedes a proposal from 2015-08-27.

Commit message

Optimize the thread matching by avoiding some phone number formatting and validating.

Description of the change

Optimize the thread matching by avoiding some phone number formatting and validating.

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote : Posted in a previous version of this proposal
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote : Posted in a previous version of this proposal
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote : Posted in a previous version of this proposal
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote : Posted in a previous version of this proposal
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote : Posted in a previous version of this proposal
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
210. By Gustavo Pichorim Boiko

Verify that the thread returned is not null on tests.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
211. By Gustavo Pichorim Boiko

Add missing copyright header.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
212. By Gustavo Pichorim Boiko

Try a phone number that matches brazilian format.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
213. By Gustavo Pichorim Boiko

Merge latest changes from parent branch

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
214. By Gustavo Pichorim Boiko

Change the way we store the normalized ID when creating new threads.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
Tiago Salem Herrmann (tiagosh) wrote :

looks good. thanks.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'historyprivate/utils.cpp'
2--- historyprivate/utils.cpp 2015-04-09 18:53:44 +0000
3+++ historyprivate/utils.cpp 2015-10-02 18:06:56 +0000
4@@ -21,6 +21,7 @@
5
6 #include "utils_p.h"
7 #include "phoneutils_p.h"
8+#include <QDebug>
9 #include <QStringList>
10 #include <QMap>
11
12@@ -70,4 +71,18 @@
13 return id1 == id2;
14 }
15
16+QString Utils::normalizeId(const QString &accountId, const QString &id)
17+{
18+ QString normalizedId = id;
19+ // for now we only normalize phone number IDs
20+ if (matchFlagsForAccount(accountId) & History::MatchPhoneNumber) {
21+ normalizedId = PhoneUtils::normalizePhoneNumber(id);
22+ }
23+ if (normalizedId.isEmpty()) {
24+ qWarning() << "Normalized phone number is empty:" << accountId << id;
25+ normalizedId = id;
26+ }
27+ return normalizedId;
28+}
29+
30 }
31
32=== modified file 'historyprivate/utils_p.h'
33--- historyprivate/utils_p.h 2015-04-01 20:05:39 +0000
34+++ historyprivate/utils_p.h 2015-10-02 18:06:56 +0000
35@@ -32,6 +32,7 @@
36 static MatchFlags matchFlagsForAccount(const QString &accountId);
37 static QString protocolFromAccountId(const QString &accountId);
38 static bool compareIds(const QString &accountId, const QString &id1, const QString & id2);
39+ static QString normalizeId(const QString &accountId, const QString &id);
40
41 private:
42 Utils();
43
44=== modified file 'plugins/sqlite/CMakeLists.txt'
45--- plugins/sqlite/CMakeLists.txt 2015-04-07 14:40:58 +0000
46+++ plugins/sqlite/CMakeLists.txt 2015-10-02 18:06:56 +0000
47@@ -16,6 +16,7 @@
48
49 include_directories(
50 ${CMAKE_SOURCE_DIR}/src
51+ ${CMAKE_SOURCE_DIR}/historyprivate
52 ${SQLITE3_INCLUDE_DIRS}
53 )
54
55@@ -36,7 +37,7 @@
56 add_custom_target(qrc_update DEPENDS ${QRC_FILE} schema_update)
57
58 add_dependencies(sqlitehistoryplugin schema_update qrc_update)
59-target_link_libraries(sqlitehistoryplugin historyservice ${SQLITE3_LIBRARIES})
60+target_link_libraries(sqlitehistoryplugin historyservice historyprivate ${SQLITE3_LIBRARIES})
61 install(TARGETS sqlitehistoryplugin DESTINATION ${HISTORY_PLUGIN_PATH})
62
63 add_subdirectory(schema)
64
65=== modified file 'plugins/sqlite/schema/CMakeLists.txt'
66--- plugins/sqlite/schema/CMakeLists.txt 2015-10-02 18:06:56 +0000
67+++ plugins/sqlite/schema/CMakeLists.txt 2015-10-02 18:06:56 +0000
68@@ -14,6 +14,6 @@
69 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/..)
70 add_executable(generate_schema generate_schema.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../sqlitedatabase.cpp)
71 qt5_use_modules(generate_schema Core DBus Sql)
72-target_link_libraries(generate_schema historyservice ${SQLITE3_LIBRARIES})
73+target_link_libraries(generate_schema historyservice historyprivate ${SQLITE3_LIBRARIES})
74
75 add_custom_target(schema_update DEPENDS ${SCHEMA_FILE} ${VERSION_FILE})
76
77=== added file 'plugins/sqlite/schema/v12.sql'
78--- plugins/sqlite/schema/v12.sql 1970-01-01 00:00:00 +0000
79+++ plugins/sqlite/schema/v12.sql 2015-10-02 18:06:56 +0000
80@@ -0,0 +1,2 @@
81+ALTER TABLE thread_participants ADD COLUMN normalizedId varchar(255);
82+UPDATE thread_participants SET normalizedId = normalizeId(accountId, participantId);
83
84=== modified file 'plugins/sqlite/sqlitedatabase.cpp'
85--- plugins/sqlite/sqlitedatabase.cpp 2015-10-02 18:06:56 +0000
86+++ plugins/sqlite/sqlitedatabase.cpp 2015-10-02 18:06:56 +0000
87@@ -23,6 +23,7 @@
88 #include "sqlite3.h"
89 #include "sqlitedatabase.h"
90 #include "types.h"
91+#include "utils_p.h"
92 #include <QStandardPaths>
93 #include <QSqlDriver>
94 #include <QSqlQuery>
95@@ -43,6 +44,21 @@
96 sqlite3_result_int(context, (int)PhoneUtils::comparePhoneNumbers(arg1, arg2));
97 }
98
99+void compareNormalizedPhoneNumbers(sqlite3_context *context, int argc, sqlite3_value **argv)
100+{
101+ QString arg1((const char*)sqlite3_value_text(argv[0]));
102+ QString arg2((const char*)sqlite3_value_text(argv[1]));
103+ sqlite3_result_int(context, (int)PhoneUtils::compareNormalizedPhoneNumbers(arg1, arg2));
104+}
105+
106+void normalizeId(sqlite3_context *context, int argc, sqlite3_value **argv)
107+{
108+ QString accountId((const char*)sqlite3_value_text(argv[0]));
109+ QString id((const char*)sqlite3_value_text(argv[1]));
110+ QString normalizedId = History::Utils::normalizeId(accountId, id);
111+ sqlite3_result_text(context, strdup(normalizedId.toUtf8().data()), -1, &free);
112+}
113+
114 SQLiteDatabase::SQLiteDatabase(QObject *parent) :
115 QObject(parent), mSchemaVersion(0)
116 {
117@@ -175,9 +191,13 @@
118 return false;
119 }
120
121- // create the comparePhoneNumbers custom sqlite function
122+ // create the comparePhoneNumbers custom sqlite functions
123 sqlite3 *handle = database().driver()->handle().value<sqlite3*>();
124 sqlite3_create_function(handle, "comparePhoneNumbers", 2, SQLITE_ANY, NULL, &comparePhoneNumbers, NULL, NULL);
125+ sqlite3_create_function(handle, "compareNormalizedPhoneNumbers", 2, SQLITE_ANY, NULL, &compareNormalizedPhoneNumbers, NULL, NULL);
126+
127+ // and also create the normalizeId function
128+ sqlite3_create_function(handle, "normalizeId", 2, SQLITE_ANY, NULL, &normalizeId, NULL, NULL);
129
130 parseVersionInfo();
131
132
133=== modified file 'plugins/sqlite/sqlitehistoryplugin.cpp'
134--- plugins/sqlite/sqlitehistoryplugin.cpp 2015-01-28 23:15:24 +0000
135+++ plugins/sqlite/sqlitehistoryplugin.cpp 2015-10-02 18:06:56 +0000
136@@ -1,5 +1,5 @@
137 /*
138- * Copyright (C) 2013 Canonical, Ltd.
139+ * Copyright (C) 2013-2015 Canonical, Ltd.
140 *
141 * Authors:
142 * Gustavo Pichorim Boiko <gustavo.boiko@canonical.com>
143@@ -26,6 +26,7 @@
144 #include "sqlitehistorythreadview.h"
145 #include "intersectionfilter.h"
146 #include "unionfilter.h"
147+#include "utils_p.h"
148 #include <QDateTime>
149 #include <QDebug>
150 #include <QStringList>
151@@ -63,6 +64,7 @@
152 return QVariantMap();
153 }
154
155+ bool phoneCompare = (matchFlags & History::MatchPhoneNumber);
156 QSqlQuery query(SQLiteDatabase::instance()->database());
157
158 // select all the threads the first participant is listed in, and from that list
159@@ -71,13 +73,15 @@
160 QString queryString("SELECT threadId FROM thread_participants WHERE %1 AND type=:type AND accountId=:accountId");
161
162 // FIXME: for now we just compare differently when using MatchPhoneNumber
163- if (matchFlags & History::MatchPhoneNumber) {
164- queryString = queryString.arg("comparePhoneNumbers(participantId, :participantId)");
165+ QString firstParticipant = participants.first();
166+ if (phoneCompare) {
167+ queryString = queryString.arg("compareNormalizedPhoneNumbers(normalizedId, :participantId)");
168+ firstParticipant = PhoneUtils::normalizePhoneNumber(firstParticipant);
169 } else {
170 queryString = queryString.arg("participantId=:participantId");
171 }
172 query.prepare(queryString);
173- query.bindValue(":participantId", participants[0]);
174+ query.bindValue(":participantId", firstParticipant);
175 query.bindValue(":type", type);
176 query.bindValue(":accountId", accountId);
177 if (!query.exec()) {
178@@ -91,54 +95,72 @@
179 }
180
181 QString existingThread;
182- // now for each threadId, check if all the other participants are listed
183- Q_FOREACH(const QString &threadId, threadIds) {
184- query.prepare("SELECT participantId FROM thread_participants WHERE "
185- "threadId=:threadId AND type=:type AND accountId=:accountId");
186- query.bindValue(":threadId", threadId);
187- query.bindValue(":type", type);
188- query.bindValue(":accountId", accountId);
189- if (!query.exec()) {
190- qCritical() << "Error:" << query.lastError() << query.lastQuery();
191- return QVariantMap();
192- }
193-
194- QStringList threadParticipants;
195- while (query.next()) {
196- threadParticipants << query.value(0).toString();
197- }
198-
199- if (threadParticipants.count() != participants.count()) {
200- continue;
201- }
202-
203- // and now compare the lists
204- bool found = true;
205- Q_FOREACH(const QString &participant, participants) {
206- if (matchFlags & History::MatchPhoneNumber) {
207- // we need to iterate the list and call the phone number comparing function for
208- // each participant from the given thread
209- bool inList = false;
210- Q_FOREACH(const QString &threadParticipant, threadParticipants) {
211- if (PhoneUtils::comparePhoneNumbers(threadParticipant, participant)) {
212- inList = true;
213+ if (participants.count() == 1 && !threadIds.isEmpty()) {
214+ existingThread = threadIds.first();
215+ } else {
216+ QStringList normalizedParticipants;
217+ if (phoneCompare) {
218+ Q_FOREACH(const QString &participant, participants) {
219+ normalizedParticipants << PhoneUtils::normalizePhoneNumber(participant);
220+ }
221+ } else {
222+ normalizedParticipants = participants;
223+ }
224+
225+ // now for each threadId, check if all the other participants are listed
226+ Q_FOREACH(const QString &threadId, threadIds) {
227+ queryString = "SELECT %1 FROM thread_participants WHERE "
228+ "threadId=:threadId AND type=:type AND accountId=:accountId";
229+ query.prepare(queryString.arg(phoneCompare ? "normalizedId" : "participantId"));
230+ query.bindValue(":threadId", threadId);
231+ query.bindValue(":type", type);
232+ query.bindValue(":accountId", accountId);
233+ if (!query.exec()) {
234+ qCritical() << "Error:" << query.lastError() << query.lastQuery();
235+ return QVariantMap();
236+ }
237+
238+ QStringList threadParticipants;
239+ while (query.next()) {
240+ threadParticipants << query.value(0).toString();
241+ }
242+
243+ // we can't use query.size() as it always return -1
244+ if (threadParticipants.count() != participants.count()) {
245+ continue;
246+ }
247+
248+ // and now compare the lists
249+ bool found = true;
250+ Q_FOREACH(const QString &participant, normalizedParticipants) {
251+ if (phoneCompare) {
252+ // we need to iterate the list and call the phone number comparing function for
253+ // each participant from the given thread
254+ bool inList = false;
255+ QStringList::iterator it = threadParticipants.begin();
256+ while (it != threadParticipants.end()) {
257+ if (PhoneUtils::compareNormalizedPhoneNumbers(*it, participant)) {
258+ inList = true;
259+ threadParticipants.erase(it);
260+ break;
261+ }
262+ ++it;
263+ }
264+ if (!inList) {
265+ found = false;
266 break;
267 }
268- }
269- if (!inList) {
270+ } else if (!threadParticipants.contains(participant)) {
271 found = false;
272 break;
273 }
274- } else if (!threadParticipants.contains(participant)) {
275- found = false;
276+ }
277+
278+ if (found) {
279+ existingThread = threadId;
280 break;
281 }
282 }
283-
284- if (found) {
285- existingThread = threadId;
286- break;
287- }
288 }
289
290 return getSingleThread(type, accountId, existingThread);
291@@ -235,12 +257,13 @@
292
293 // and insert the participants
294 Q_FOREACH(const QString &participant, participants) {
295- query.prepare("INSERT INTO thread_participants (accountId, threadId, type, participantId)"
296- "VALUES (:accountId, :threadId, :type, :participantId)");
297+ query.prepare("INSERT INTO thread_participants (accountId, threadId, type, participantId, normalizedId)"
298+ "VALUES (:accountId, :threadId, :type, :participantId, :normalizedId)");
299 query.bindValue(":accountId", accountId);
300 query.bindValue(":threadId", threadId);
301 query.bindValue(":type", type);
302 query.bindValue(":participantId", participant);
303+ query.bindValue(":normalizedId", History::Utils::normalizeId(accountId, participant));
304 if (!query.exec()) {
305 qCritical() << "Error:" << query.lastError() << query.lastQuery();
306 return QVariantMap();
307
308=== modified file 'plugins/sqlite/sqlitehistoryplugin.h'
309--- plugins/sqlite/sqlitehistoryplugin.h 2015-01-28 23:08:01 +0000
310+++ plugins/sqlite/sqlitehistoryplugin.h 2015-10-02 18:06:56 +0000
311@@ -1,5 +1,5 @@
312 /*
313- * Copyright (C) 2013 Canonical, Ltd.
314+ * Copyright (C) 2013-2015 Canonical, Ltd.
315 *
316 * Authors:
317 * Gustavo Pichorim Boiko <gustavo.boiko@canonical.com>
318
319=== modified file 'src/phoneutils.cpp'
320--- src/phoneutils.cpp 2015-07-31 21:37:05 +0000
321+++ src/phoneutils.cpp 2015-10-02 18:06:56 +0000
322@@ -59,29 +59,38 @@
323
324 bool PhoneUtils::comparePhoneNumbers(const QString &phoneNumberA, const QString &phoneNumberB)
325 {
326- static i18n::phonenumbers::PhoneNumberUtil *phonenumberUtil = i18n::phonenumbers::PhoneNumberUtil::GetInstance();
327+ // just do a simple string comparison first
328+ if (phoneNumberA == phoneNumberB) {
329+ return true;
330+ }
331+
332 QString normalizedPhoneNumberA = normalizePhoneNumber(phoneNumberA);
333 QString normalizedPhoneNumberB = normalizePhoneNumber(phoneNumberB);
334
335- // just do a simple string comparison if we are dealing with non phone numbers
336- if (!isPhoneNumber(phoneNumberA) || !isPhoneNumber(phoneNumberB)) {
337- return phoneNumberA == phoneNumberB;
338+ return compareNormalizedPhoneNumbers(normalizedPhoneNumberA, normalizedPhoneNumberB);
339+}
340+
341+bool PhoneUtils::compareNormalizedPhoneNumbers(const QString &numberA, const QString &numberB)
342+{
343+ if (numberA == numberB) {
344+ return true;
345 }
346
347- if (normalizedPhoneNumberA.size() < 7 || normalizedPhoneNumberB.size() < 7) {
348- return normalizedPhoneNumberA == normalizedPhoneNumberB;
349+ static i18n::phonenumbers::PhoneNumberUtil *phonenumberUtil = i18n::phonenumbers::PhoneNumberUtil::GetInstance();
350+
351+ if (numberA.size() < 7 || numberB.size() < 7) {
352+ return false;
353 }
354
355 i18n::phonenumbers::PhoneNumberUtil::MatchType match = phonenumberUtil->
356- IsNumberMatchWithTwoStrings(phoneNumberA.toStdString(),
357- phoneNumberB.toStdString());
358+ IsNumberMatchWithTwoStrings(numberA.toStdString(),
359+ numberB.toStdString());
360 return (match > i18n::phonenumbers::PhoneNumberUtil::NO_MATCH);
361 }
362
363 bool PhoneUtils::isPhoneNumber(const QString &phoneNumber)
364 {
365 static i18n::phonenumbers::PhoneNumberUtil *phonenumberUtil = i18n::phonenumbers::PhoneNumberUtil::GetInstance();
366- std::string formattedNumber;
367 i18n::phonenumbers::PhoneNumber number;
368 i18n::phonenumbers::PhoneNumberUtil::ErrorType error;
369 error = phonenumberUtil->Parse(phoneNumber.toStdString(), region().toStdString(), &number);
370
371=== modified file 'src/phoneutils_p.h'
372--- src/phoneutils_p.h 2015-07-17 19:03:41 +0000
373+++ src/phoneutils_p.h 2015-10-02 18:06:56 +0000
374@@ -31,7 +31,8 @@
375 Q_OBJECT
376 public:
377 explicit PhoneUtils(QObject *parent = 0);
378- Q_INVOKABLE static bool comparePhoneNumbers(const QString &number1, const QString &number2);
379+ Q_INVOKABLE static bool comparePhoneNumbers(const QString &numberA, const QString &numberB);
380+ Q_INVOKABLE static bool compareNormalizedPhoneNumbers(const QString &numberA, const QString &numberB);
381 Q_INVOKABLE static bool isPhoneNumber(const QString &identifier);
382 Q_INVOKABLE static QString normalizePhoneNumber(const QString &identifier);
383 private:
384
385=== modified file 'tests/libhistoryservice/ManagerTest.cpp'
386--- tests/libhistoryservice/ManagerTest.cpp 2013-12-09 21:18:14 +0000
387+++ tests/libhistoryservice/ManagerTest.cpp 2015-10-02 18:06:56 +0000
388@@ -76,9 +76,9 @@
389 << (QStringList() << "oneParticipant");
390 QTest::newRow("voice thread using phone match") << "anotherAccountId"
391 << History::EventTypeVoice
392- << (QStringList() << "+1234567890")
393+ << (QStringList() << "+554198765432")
394 << History::MatchFlags(History::MatchPhoneNumber)
395- << (QStringList() << "4567890");
396+ << (QStringList() << "98765432");
397 }
398
399 void ManagerTest::testThreadForParticipants()
400@@ -100,7 +100,8 @@
401
402 // now try to get the thread again to see if it is returned correctly
403 History::Thread sameThread = mManager->threadForParticipants(accountId, type, participantsToMatch, matchFlags, false);
404- QVERIFY(sameThread == thread);
405+ QVERIFY(!sameThread.isNull());
406+ QCOMPARE(sameThread, thread);
407 }
408
409 void ManagerTest::testQueryEvents()
410
411=== modified file 'tests/libhistoryservice/PhoneUtilsTest.cpp'
412--- tests/libhistoryservice/PhoneUtilsTest.cpp 2015-07-15 13:35:23 +0000
413+++ tests/libhistoryservice/PhoneUtilsTest.cpp 2015-10-02 18:06:56 +0000
414@@ -64,7 +64,7 @@
415 QTest::newRow("string equal") << "12345678" << "12345678" << true;
416 QTest::newRow("number with dash") << "1234-5678" << "12345678" << true;
417 QTest::newRow("number with area code") << "12312345678" << "12345678" << true;
418- QTest::newRow("number with extension") << "12345678#123" << "12345678" << true;
419+ QTest::newRow("number with extension") << "12345678#123" << "12345678" << false;
420 QTest::newRow("both numbers with extension") << "(123)12345678#1" << "12345678#1" << true;
421 QTest::newRow("numbers with different extension") << "1234567#1" << "1234567#2" << false;
422 QTest::newRow("short/emergency numbers") << "190" << "190" << true;

Subscribers

People subscribed via source and target branches