Merge lp:~mterry/unity8/cancel-pam-harder into lp:unity8

Proposed by Michael Terry
Status: Merged
Approved by: Albert Astals Cid
Approved revision: 1644
Merged at revision: 1780
Proposed branch: lp:~mterry/unity8/cancel-pam-harder
Merge into: lp:unity8
Diff against target: 225 lines (+117/-22)
4 files modified
plugins/LightDM/liblightdm/GreeterPrivate.cpp (+29/-5)
plugins/LightDM/liblightdm/GreeterPrivate.h (+1/-1)
tests/plugins/LightDM/CMakeLists.txt (+35/-16)
tests/plugins/LightDM/pam.cpp (+52/-0)
To merge this branch: bzr merge lp:~mterry/unity8/cancel-pam-harder
Reviewer Review Type Date Requested Status
Albert Astals Cid (community) Approve
PS Jenkins bot (community) continuous-integration Needs Fixing
Michał Sawicz Needs Fixing
Review via email: mp+251174@code.launchpad.net

Commit message

Fix a possible crash in our PAM threading code. (LP: #1425362)

So it was possible for a PAM subthread (which does the actual authenticating) to get mismatched state with the main thread. When this happened (i.e. when the subthread tried to use a pam_handle object that the main thread had already closed), a crash occurs. Like LP: #1425362.

So I've done a few cleanups here:
- Use a cancellable QConcurrent method (::mapped instead of ::run). Being able to cancel the thread is a stronger hammer than just hoping we answer all questions from the thread and it will close itself.
- We now block when the subthread sends a prompt request to the main thread. This makes handling those signals more predictable when we're trying to close the thread.
- We will process events while waiting for the subthread to close, so that we can actually process the above signals in the main thread.

I've added a dumb little test that tries to authenticate/cancel/authenticate 100 times in quick succession. Without this branch, we'd crash reliably anywhere from attempt 2 to 7. With this branch, we're fine all 100.

Description of the change

Fix a possible crash in our PAM threading code. (LP: #1425362)

So it was possible for a PAM subthread (which does the actual authenticating) to get mismatched state with the main thread. When this happened (i.e. when the subthread tried to use a pam_handle object that the main thread had already closed), a crash occurs. Like LP: #1425362.

So I've done a few cleanups here:
- Use a cancellable QConcurrent method (::mapped instead of ::run). Being able to cancel the thread is a stronger hammer than just hoping we answer all questions from the thread and it will close itself.
- We now block when the subthread sends a prompt request to the main thread. This makes handling those signals more predictable when we're trying to close the thread.
- We will process events while waiting for the subthread to close, so that we can actually process the above signals in the main thread.

I've added a dumb little test that tries to authenticate/cancel/authenticate 100 times in quick succession. Without this branch, we'd crash reliably anywhere from attempt 2 to 7. With this branch, we're fine all 100.

Users didn't notice this bug, because the timing is tough to reproduce naturally (I would have thought, but then bug 1425362 noticed in a VM... Maybe the slowness of a VM helps surface this).

== Checklist ==

 * Are there any related MPs required for this MP to build/function as expected? Please list.
 No

 * Did you perform an exploratory manual test run of your code change and any related functionality?
 Yes... I added a test anyway, that stresses this code

 * Did you make sure that your branch does not contain spurious tags?
 Yes

 * If you changed the packaging (debian), did you subscribe the ubuntu-unity team to this MP?
 NA

 * If you changed the UI, has there been a design review?
 NA

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Michael Terry (mterry) wrote :

Marco Trevisan (the original bug filer) notes that this branch fixes his crash.

Revision history for this message
Albert Astals Cid (aacid) wrote :

Tests never seem to finish now

 6/22 Test #6: testGreeterPam ...................***Timeout 1500.00 sec

review: Needs Fixing
Revision history for this message
Albert Astals Cid (aacid) wrote :

The test passes for me locally but fails in CI, i wonder if either it needs some extra dependency we don't have on CI or if it needs X/xvfb

Revision history for this message
Michał Sawicz (saviq) wrote :

Interestingly, this is what fails for me under sbuild:

22/23 Test #23: wizardsystemtest .................***Failed 15.22 sec
********* Start testing of SystemTest *********
Config: Using QtTest library 5.4.0, Qt 5.4.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 4.9.2)
PASS : SystemTest::initTestCase()
PASS : SystemTest::testEnable()
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
PASS : SystemTest::testDisable()
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
FAIL! : SystemTest::testNoticeChanges() Compared values are not the same
   Actual (((spy.count()))): 3
   Expected (4) : 4
   Loc: [/«BUILDDIR»/unity8-8.02+15.04.20150302/tests/plugins/Wizard/tst_system.cpp(120)]
PASS : SystemTest::cleanupTestCase()
Totals: 4 passed, 1 failed, 0 skipped, 0 blacklisted
********* Finished testing of SystemTest *********

review: Needs Fixing
Revision history for this message
Michael Terry (mterry) wrote :

Huh. During build it passes in jenkins (since make test is run during build). It's just during the qmluitest run that it fails.

Revision history for this message
Michael Terry (mterry) wrote :

<Saviq> mterry, huh, the failure I reported actually is there in trunk for me...

So that leaves just the jenkins failure. I'm trying to see how the qmluitests environment is different from the build one.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Michael Terry (mterry) wrote :

OK, addressed the build failure in jenkins. The remaining qmluitest failures are issues in trunk.

Revision history for this message
Francis Ginther (fginther) wrote :

The above mako testing failed due to an unbootable image making it into the pipeline. The ci job has been restarted.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Albert Astals Cid (aacid) wrote :

Small fix:
 * You can change the two futureWatcher.future().something() to futureWatcher.something()

About QCoreApplication::processEvents it's a bit on the nasty side, is it possible to put the code in a slot attached to the futurewatcher finished signal?

Also does respond(QString()) need to happen repeteadly inside the loop or just once?

review: Needs Information
Revision history for this message
Michael Terry (mterry) wrote :

> You can change the two futureWatcher.future().something()
> to futureWatcher.something()

Done.

> About QCoreApplication::processEvents it's a bit on the nasty
> side, is it possible to put the code in a slot attached to
> the futurewatcher finished signal?

This would mean holding on to a map of QFuture objects to pam_handle objects so we'd know which ones need to have pam_end called on them when the thread is done. And keeping a set of waiting-QFuture-responses for each thread.

I just figured threading code and PAM code are each complicated enough that I didn't want to try to be fancy. So I favored the direct route.

But I agree that a manual processEvents isn't super sexy. I can change it to the above if you'd like.

> Also does respond(QString()) need to happen
> repeteadly inside the loop or just once?

PAM can offer multiple prompts in a row. The child-thread will block on each response one at a time in a small loop. QFuture.cancel() will not interrupt a waiting child-thread.

But if you can promise that it *will* interrupt the thread the second it stops waiting, before it can get to waiting on a second prompt, then we only need respond() once. And we could even get rid of the loop and just call processEvents once.

I was not so confident in how cancel() worked. But I haven't dealt with Qt threads often.

Revision history for this message
Albert Astals Cid (aacid) wrote :

As far as i know, no, cancel won't terminate the thread, actually i think would not cancel authenticateWithPam in the middle of an execution, it's just that the future has not started executing and is cancelled before doing so.

As discussed on irc, please "add code to verify that all respond() have been properly dispatched"

review: Needs Fixing
Revision history for this message
Albert Astals Cid (aacid) wrote :

add code to the test :D

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Albert Astals Cid (aacid) wrote :

 * Did you perform an exploratory manual test run of the code change and any related functionality?
Yes

 * Did CI run pass?
No, but the added one passes

 * Did you make sure that the branch does not contain spurious tags?
Yes

review: Approve
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Michał Sawicz (saviq) wrote :

This prevents me from unlocking unity8 on my desktop. The password entry field goes wiggle-wiggle-wiggle every 5s or so without any interaction.

The unity8.log prints a seemingly related:
> This backend doesn't support multiple users

review: Needs Fixing
Revision history for this message
Michael Terry (mterry) wrote :

OK, try this again. I was able to reproduce your problem on the desktop, Saviq. Fixed it.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Albert Astals Cid (aacid) :
review: Abstain
Revision history for this message
Albert Astals Cid (aacid) wrote :

I retested the saviq was having and i can't reproduce anymore.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'plugins/LightDM/liblightdm/GreeterPrivate.cpp'
2--- plugins/LightDM/liblightdm/GreeterPrivate.cpp 2015-01-20 11:50:19 +0000
3+++ plugins/LightDM/liblightdm/GreeterPrivate.cpp 2015-04-23 00:56:41 +0000
4@@ -56,7 +56,8 @@
5 this, SLOT(handleMessage(pam_handle *, QString, QLightDM::Greeter::MessageType)));
6 // This next connect is how we pass ResponseFutures between threads
7 connect(this, SIGNAL(showPrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)),
8- this, SLOT(handlePrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)));
9+ this, SLOT(handlePrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)),
10+ Qt::BlockingQueuedConnection);
11 }
12
13 ~GreeterImpl()
14@@ -68,6 +69,13 @@
15 {
16 // Clear out any existing PAM interactions first
17 cancelPam();
18+ if (pamHandle != nullptr) {
19+ // While we were cancelling pam above, we processed Qt events.
20+ // Which may have allowed someone to call start() on us again.
21+ // In which case, we'll bail on our current start() call.
22+ // This isn't racy because it's all in the same thread.
23+ return;
24+ }
25
26 AppData *appData = new AppData();
27 appData->impl = this;
28@@ -79,7 +87,7 @@
29
30 if (pam_start("lightdm", username.toUtf8(), &conversation, &pamHandle) == PAM_SUCCESS) {
31 appData->handle = pamHandle;
32- futureWatcher.setFuture(QtConcurrent::run(authenticateWithPam, pamHandle));
33+ futureWatcher.setFuture(QtConcurrent::mapped(QList<pam_handle*>() << pamHandle, authenticateWithPam));
34 } else {
35 delete appData;
36 greeterPrivate->authenticated = false;
37@@ -88,7 +96,7 @@
38 }
39 }
40
41- static int authenticateWithPam(pam_handle* pamHandle)
42+ static int authenticateWithPam(pam_handle* const& pamHandle)
43 {
44 int pamStatus = pam_authenticate(pamHandle, 0);
45 if (pamStatus == PAM_SUCCESS) {
46@@ -242,11 +250,22 @@
47 private:
48 void cancelPam()
49 {
50- // Unfortunately we can't simply cancel our QFuture because QtConcurrent::run doesn't support cancel
51 if (pamHandle != nullptr) {
52+ QFuture<int> pamFuture = futureWatcher.future();
53 pam_handle *handle = pamHandle;
54 pamHandle = nullptr; // to disable normal finishPam() handling
55- while (respond(QString())); // clear our local queue of QFutures
56+ pamFuture.cancel();
57+
58+ // Note the empty loop, we just want to clear the futures queue.
59+ // Any further prompts from the pam thread will be immediately
60+ // responded to/dismissed in handlePrompt().
61+ while (respond(QString()));
62+
63+ // Now let signal/slot handling happen so the thread can finish
64+ while (!pamFuture.isFinished()) {
65+ QCoreApplication::processEvents();
66+ }
67+
68 pam_end(handle, PAM_CONV_ERR);
69 }
70 }
71@@ -266,6 +285,11 @@
72 {
73 }
74
75+GreeterPrivate::~GreeterPrivate()
76+{
77+ delete m_impl;
78+}
79+
80 void GreeterPrivate::handleAuthenticate()
81 {
82 m_impl->start(authenticationUser);
83
84=== modified file 'plugins/LightDM/liblightdm/GreeterPrivate.h'
85--- plugins/LightDM/liblightdm/GreeterPrivate.h 2015-01-20 11:50:19 +0000
86+++ plugins/LightDM/liblightdm/GreeterPrivate.h 2015-04-23 00:56:41 +0000
87@@ -30,7 +30,7 @@
88 {
89 public:
90 explicit GreeterPrivate(Greeter* parent=0);
91- virtual ~GreeterPrivate() = default;
92+ virtual ~GreeterPrivate();
93
94 // These variables may not be used by all subclasses, that's no problem
95 bool authenticated;
96
97=== modified file 'tests/plugins/LightDM/CMakeLists.txt'
98--- tests/plugins/LightDM/CMakeLists.txt 2015-03-02 14:31:08 +0000
99+++ tests/plugins/LightDM/CMakeLists.txt 2015-04-23 00:56:41 +0000
100@@ -1,8 +1,38 @@
101+add_definitions(
102+ -DCURRENT_SOURCE_DIR="${CMAKE_CURRENT_SOURCE_DIR}"
103+ )
104+include_directories(
105+ ${CMAKE_CURRENT_BINARY_DIR}
106+ )
107+
108 add_executable(GreeterDBusTestExec
109 dbus.cpp
110 ${CMAKE_SOURCE_DIR}/plugins/LightDM/Greeter.cpp
111 )
112 qt5_use_modules(GreeterDBusTestExec Core DBus Quick Test)
113+target_link_libraries(GreeterDBusTestExec
114+ -L${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm
115+ -llightdm-qt5-2
116+ )
117+target_include_directories(GreeterDBusTestExec PUBLIC
118+ ${CMAKE_SOURCE_DIR}/plugins/LightDM
119+ ${CMAKE_SOURCE_DIR}/tests/mocks/LightDM
120+ )
121+add_binary_qml_test(GreeterDBus "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "QML2_IMPORT_PATH=${CMAKE_BINARY_DIR}/tests/mocks")
122+
123+add_executable(GreeterPamTestExec
124+ pam.cpp
125+ ${CMAKE_SOURCE_DIR}/plugins/LightDM/liblightdm/GreeterPrivate.cpp
126+ )
127+qt5_use_modules(GreeterPamTestExec Concurrent Core Test)
128+target_link_libraries(GreeterPamTestExec
129+ integratedLightDM
130+ )
131+target_include_directories(GreeterPamTestExec PUBLIC
132+ ${CMAKE_SOURCE_DIR}/plugins/LightDM/liblightdm
133+ )
134+add_qmltest_target(testGreeterPam "${CMAKE_CURRENT_BINARY_DIR}/GreeterPamTestExec" TRUE FALSE)
135+add_dependencies(testGreeterPam GreeterPamTestExec)
136
137 add_executable(GreeterUsersModelTestExec
138 usersmodel.cpp
139@@ -10,24 +40,13 @@
140 ${CMAKE_SOURCE_DIR}/plugins/Utils/unitysortfilterproxymodelqml.cpp
141 )
142 qt5_use_modules(GreeterUsersModelTestExec Core Test)
143-
144-include_directories(
145- ${CMAKE_CURRENT_BINARY_DIR}
146+target_link_libraries(GreeterUsersModelTestExec
147+ -L${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm
148+ -llightdm-qt5-2
149+ )
150+target_include_directories(GreeterUsersModelTestExec PUBLIC
151 ${CMAKE_SOURCE_DIR}/plugins/LightDM
152 ${CMAKE_SOURCE_DIR}/plugins/Utils
153 ${CMAKE_SOURCE_DIR}/tests/mocks/LightDM
154 )
155-
156-target_link_libraries(GreeterDBusTestExec
157- MockLightDM
158- )
159-
160-target_link_libraries(GreeterUsersModelTestExec
161- MockLightDM
162- )
163-
164-add_definitions(-DCURRENT_SOURCE_DIR="${CMAKE_CURRENT_SOURCE_DIR}")
165-
166-add_binary_qml_test(GreeterDBus "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "QML2_IMPORT_PATH=${CMAKE_BINARY_DIR}/tests/mocks")
167-
168 add_binary_qml_test(GreeterUsersModel "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "LIBLIGHTDM_MOCK_MODE=full")
169
170=== added file 'tests/plugins/LightDM/pam.cpp'
171--- tests/plugins/LightDM/pam.cpp 1970-01-01 00:00:00 +0000
172+++ tests/plugins/LightDM/pam.cpp 2015-04-23 00:56:41 +0000
173@@ -0,0 +1,52 @@
174+/*
175+ * Copyright (C) 2015 Canonical, Ltd.
176+ *
177+ * This program is free software; you can redistribute it and/or modify
178+ * it under the terms of the GNU General Public License as published by
179+ * the Free Software Foundation; version 3.
180+ *
181+ * This program is distributed in the hope that it will be useful,
182+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
183+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
184+ * GNU General Public License for more details.
185+ *
186+ * You should have received a copy of the GNU General Public License
187+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
188+ */
189+
190+#include "GreeterPrivate.h"
191+
192+#include <QtTest>
193+
194+class GreeterPamTest : public QObject
195+{
196+ Q_OBJECT
197+
198+private Q_SLOTS:
199+
200+ void init()
201+ {
202+ m_greeterpriv = new QLightDM::GreeterPrivate();
203+ }
204+
205+ void cleanup()
206+ {
207+ delete m_greeterpriv;
208+ QTRY_COMPARE(QThreadPool::globalInstance()->activeThreadCount(), 0);
209+ }
210+
211+ void testRapidFireAuthentication()
212+ {
213+ m_greeterpriv->authenticationUser = qgetenv("USER");
214+ for (int i = 0; i < 100; i++) {
215+ m_greeterpriv->handleAuthenticate();
216+ }
217+ }
218+
219+private:
220+ QLightDM::GreeterPrivate *m_greeterpriv;
221+};
222+
223+QTEST_GUILESS_MAIN(GreeterPamTest)
224+
225+#include "pam.moc"

Subscribers

People subscribed via source and target branches