Merge lp:~mterry/unity8/cancel-pam-harder into lp:unity8

Proposed by Michael Terry
Status: Merged
Approved by: Albert Astals Cid
Approved revision: 1644
Merged at revision: 1780
Proposed branch: lp:~mterry/unity8/cancel-pam-harder
Merge into: lp:unity8
Diff against target: 225 lines (+117/-22)
4 files modified
plugins/LightDM/liblightdm/GreeterPrivate.cpp (+29/-5)
plugins/LightDM/liblightdm/GreeterPrivate.h (+1/-1)
tests/plugins/LightDM/CMakeLists.txt (+35/-16)
tests/plugins/LightDM/pam.cpp (+52/-0)
To merge this branch: bzr merge lp:~mterry/unity8/cancel-pam-harder
Reviewer Review Type Date Requested Status
Albert Astals Cid (community) Approve
PS Jenkins bot (community) continuous-integration Needs Fixing
Michał Sawicz Needs Fixing
Review via email: mp+251174@code.launchpad.net

Commit message

Fix a possible crash in our PAM threading code. (LP: #1425362)

So it was possible for a PAM subthread (which does the actual authenticating) to get mismatched state with the main thread. When this happened (i.e. when the subthread tried to use a pam_handle object that the main thread had already closed), a crash occurs. Like LP: #1425362.

So I've done a few cleanups here:
- Use a cancellable QConcurrent method (::mapped instead of ::run). Being able to cancel the thread is a stronger hammer than just hoping we answer all questions from the thread and it will close itself.
- We now block when the subthread sends a prompt request to the main thread. This makes handling those signals more predictable when we're trying to close the thread.
- We will process events while waiting for the subthread to close, so that we can actually process the above signals in the main thread.

I've added a dumb little test that tries to authenticate/cancel/authenticate 100 times in quick succession. Without this branch, we'd crash reliably anywhere from attempt 2 to 7. With this branch, we're fine all 100.

Description of the change

Fix a possible crash in our PAM threading code. (LP: #1425362)

So it was possible for a PAM subthread (which does the actual authenticating) to get mismatched state with the main thread. When this happened (i.e. when the subthread tried to use a pam_handle object that the main thread had already closed), a crash occurs. Like LP: #1425362.

So I've done a few cleanups here:
- Use a cancellable QConcurrent method (::mapped instead of ::run). Being able to cancel the thread is a stronger hammer than just hoping we answer all questions from the thread and it will close itself.
- We now block when the subthread sends a prompt request to the main thread. This makes handling those signals more predictable when we're trying to close the thread.
- We will process events while waiting for the subthread to close, so that we can actually process the above signals in the main thread.

I've added a dumb little test that tries to authenticate/cancel/authenticate 100 times in quick succession. Without this branch, we'd crash reliably anywhere from attempt 2 to 7. With this branch, we're fine all 100.

Users didn't notice this bug, because the timing is tough to reproduce naturally (I would have thought, but then bug 1425362 noticed in a VM... Maybe the slowness of a VM helps surface this).

== Checklist ==

 * Are there any related MPs required for this MP to build/function as expected? Please list.
 No

 * Did you perform an exploratory manual test run of your code change and any related functionality?
 Yes... I added a test anyway, that stresses this code

 * Did you make sure that your branch does not contain spurious tags?
 Yes

 * If you changed the packaging (debian), did you subscribe the ubuntu-unity team to this MP?
 NA

 * If you changed the UI, has there been a design review?
 NA

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Michael Terry (mterry) wrote :

Marco Trevisan (the original bug filer) notes that this branch fixes his crash.

Revision history for this message
Albert Astals Cid (aacid) wrote :

Tests never seem to finish now

 6/22 Test #6: testGreeterPam ...................***Timeout 1500.00 sec

review: Needs Fixing
Revision history for this message
Albert Astals Cid (aacid) wrote :

The test passes for me locally but fails in CI, i wonder if either it needs some extra dependency we don't have on CI or if it needs X/xvfb

Revision history for this message
Michał Sawicz (saviq) wrote :

Interestingly, this is what fails for me under sbuild:

22/23 Test #23: wizardsystemtest .................***Failed 15.22 sec
********* Start testing of SystemTest *********
Config: Using QtTest library 5.4.0, Qt 5.4.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 4.9.2)
PASS : SystemTest::initTestCase()
PASS : SystemTest::testEnable()
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
PASS : SystemTest::testDisable()
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
FAIL! : SystemTest::testNoticeChanges() Compared values are not the same
   Actual (((spy.count()))): 3
   Expected (4) : 4
   Loc: [/«BUILDDIR»/unity8-8.02+15.04.20150302/tests/plugins/Wizard/tst_system.cpp(120)]
PASS : SystemTest::cleanupTestCase()
Totals: 4 passed, 1 failed, 0 skipped, 0 blacklisted
********* Finished testing of SystemTest *********

review: Needs Fixing
Revision history for this message
Michael Terry (mterry) wrote :

Huh. During build it passes in jenkins (since make test is run during build). It's just during the qmluitest run that it fails.

Revision history for this message
Michael Terry (mterry) wrote :

<Saviq> mterry, huh, the failure I reported actually is there in trunk for me...

So that leaves just the jenkins failure. I'm trying to see how the qmluitests environment is different from the build one.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Michael Terry (mterry) wrote :

OK, addressed the build failure in jenkins. The remaining qmluitest failures are issues in trunk.

Revision history for this message
Francis Ginther (fginther) wrote :

The above mako testing failed due to an unbootable image making it into the pipeline. The ci job has been restarted.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Albert Astals Cid (aacid) wrote :

Small fix:
 * You can change the two futureWatcher.future().something() to futureWatcher.something()

About QCoreApplication::processEvents it's a bit on the nasty side, is it possible to put the code in a slot attached to the futurewatcher finished signal?

Also does respond(QString()) need to happen repeteadly inside the loop or just once?

review: Needs Information
Revision history for this message
Michael Terry (mterry) wrote :

> You can change the two futureWatcher.future().something()
> to futureWatcher.something()

Done.

> About QCoreApplication::processEvents it's a bit on the nasty
> side, is it possible to put the code in a slot attached to
> the futurewatcher finished signal?

This would mean holding on to a map of QFuture objects to pam_handle objects so we'd know which ones need to have pam_end called on them when the thread is done. And keeping a set of waiting-QFuture-responses for each thread.

I just figured threading code and PAM code are each complicated enough that I didn't want to try to be fancy. So I favored the direct route.

But I agree that a manual processEvents isn't super sexy. I can change it to the above if you'd like.

> Also does respond(QString()) need to happen
> repeteadly inside the loop or just once?

PAM can offer multiple prompts in a row. The child-thread will block on each response one at a time in a small loop. QFuture.cancel() will not interrupt a waiting child-thread.

But if you can promise that it *will* interrupt the thread the second it stops waiting, before it can get to waiting on a second prompt, then we only need respond() once. And we could even get rid of the loop and just call processEvents once.

I was not so confident in how cancel() worked. But I haven't dealt with Qt threads often.

Revision history for this message
Albert Astals Cid (aacid) wrote :

As far as i know, no, cancel won't terminate the thread, actually i think would not cancel authenticateWithPam in the middle of an execution, it's just that the future has not started executing and is cancelled before doing so.

As discussed on irc, please "add code to verify that all respond() have been properly dispatched"

review: Needs Fixing
Revision history for this message
Albert Astals Cid (aacid) wrote :

add code to the test :D

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Albert Astals Cid (aacid) wrote :

 * Did you perform an exploratory manual test run of the code change and any related functionality?
Yes

 * Did CI run pass?
No, but the added one passes

 * Did you make sure that the branch does not contain spurious tags?
Yes

review: Approve
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Michał Sawicz (saviq) wrote :

This prevents me from unlocking unity8 on my desktop. The password entry field goes wiggle-wiggle-wiggle every 5s or so without any interaction.

The unity8.log prints a seemingly related:
> This backend doesn't support multiple users

review: Needs Fixing
Revision history for this message
Michael Terry (mterry) wrote :

OK, try this again. I was able to reproduce your problem on the desktop, Saviq. Fixed it.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Albert Astals Cid (aacid) :
review: Abstain
Revision history for this message
Albert Astals Cid (aacid) wrote :

I retested the saviq was having and i can't reproduce anymore.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'plugins/LightDM/liblightdm/GreeterPrivate.cpp'
--- plugins/LightDM/liblightdm/GreeterPrivate.cpp 2015-01-20 11:50:19 +0000
+++ plugins/LightDM/liblightdm/GreeterPrivate.cpp 2015-04-23 00:56:41 +0000
@@ -56,7 +56,8 @@
56 this, SLOT(handleMessage(pam_handle *, QString, QLightDM::Greeter::MessageType)));56 this, SLOT(handleMessage(pam_handle *, QString, QLightDM::Greeter::MessageType)));
57 // This next connect is how we pass ResponseFutures between threads57 // This next connect is how we pass ResponseFutures between threads
58 connect(this, SIGNAL(showPrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)),58 connect(this, SIGNAL(showPrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)),
59 this, SLOT(handlePrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)));59 this, SLOT(handlePrompt(pam_handle *, QString, QLightDM::Greeter::PromptType, QLightDM::GreeterImpl::ResponseFuture)),
60 Qt::BlockingQueuedConnection);
60 }61 }
6162
62 ~GreeterImpl()63 ~GreeterImpl()
@@ -68,6 +69,13 @@
68 {69 {
69 // Clear out any existing PAM interactions first70 // Clear out any existing PAM interactions first
70 cancelPam();71 cancelPam();
72 if (pamHandle != nullptr) {
73 // While we were cancelling pam above, we processed Qt events.
74 // Which may have allowed someone to call start() on us again.
75 // In which case, we'll bail on our current start() call.
76 // This isn't racy because it's all in the same thread.
77 return;
78 }
7179
72 AppData *appData = new AppData();80 AppData *appData = new AppData();
73 appData->impl = this;81 appData->impl = this;
@@ -79,7 +87,7 @@
7987
80 if (pam_start("lightdm", username.toUtf8(), &conversation, &pamHandle) == PAM_SUCCESS) {88 if (pam_start("lightdm", username.toUtf8(), &conversation, &pamHandle) == PAM_SUCCESS) {
81 appData->handle = pamHandle;89 appData->handle = pamHandle;
82 futureWatcher.setFuture(QtConcurrent::run(authenticateWithPam, pamHandle));90 futureWatcher.setFuture(QtConcurrent::mapped(QList<pam_handle*>() << pamHandle, authenticateWithPam));
83 } else {91 } else {
84 delete appData;92 delete appData;
85 greeterPrivate->authenticated = false;93 greeterPrivate->authenticated = false;
@@ -88,7 +96,7 @@
88 }96 }
89 }97 }
9098
91 static int authenticateWithPam(pam_handle* pamHandle)99 static int authenticateWithPam(pam_handle* const& pamHandle)
92 {100 {
93 int pamStatus = pam_authenticate(pamHandle, 0);101 int pamStatus = pam_authenticate(pamHandle, 0);
94 if (pamStatus == PAM_SUCCESS) {102 if (pamStatus == PAM_SUCCESS) {
@@ -242,11 +250,22 @@
242private:250private:
243 void cancelPam()251 void cancelPam()
244 {252 {
245 // Unfortunately we can't simply cancel our QFuture because QtConcurrent::run doesn't support cancel
246 if (pamHandle != nullptr) {253 if (pamHandle != nullptr) {
254 QFuture<int> pamFuture = futureWatcher.future();
247 pam_handle *handle = pamHandle;255 pam_handle *handle = pamHandle;
248 pamHandle = nullptr; // to disable normal finishPam() handling256 pamHandle = nullptr; // to disable normal finishPam() handling
249 while (respond(QString())); // clear our local queue of QFutures257 pamFuture.cancel();
258
259 // Note the empty loop, we just want to clear the futures queue.
260 // Any further prompts from the pam thread will be immediately
261 // responded to/dismissed in handlePrompt().
262 while (respond(QString()));
263
264 // Now let signal/slot handling happen so the thread can finish
265 while (!pamFuture.isFinished()) {
266 QCoreApplication::processEvents();
267 }
268
250 pam_end(handle, PAM_CONV_ERR);269 pam_end(handle, PAM_CONV_ERR);
251 }270 }
252 }271 }
@@ -266,6 +285,11 @@
266{285{
267}286}
268287
288GreeterPrivate::~GreeterPrivate()
289{
290 delete m_impl;
291}
292
269void GreeterPrivate::handleAuthenticate()293void GreeterPrivate::handleAuthenticate()
270{294{
271 m_impl->start(authenticationUser);295 m_impl->start(authenticationUser);
272296
=== modified file 'plugins/LightDM/liblightdm/GreeterPrivate.h'
--- plugins/LightDM/liblightdm/GreeterPrivate.h 2015-01-20 11:50:19 +0000
+++ plugins/LightDM/liblightdm/GreeterPrivate.h 2015-04-23 00:56:41 +0000
@@ -30,7 +30,7 @@
30{30{
31public:31public:
32 explicit GreeterPrivate(Greeter* parent=0);32 explicit GreeterPrivate(Greeter* parent=0);
33 virtual ~GreeterPrivate() = default;33 virtual ~GreeterPrivate();
3434
35 // These variables may not be used by all subclasses, that's no problem35 // These variables may not be used by all subclasses, that's no problem
36 bool authenticated;36 bool authenticated;
3737
=== modified file 'tests/plugins/LightDM/CMakeLists.txt'
--- tests/plugins/LightDM/CMakeLists.txt 2015-03-02 14:31:08 +0000
+++ tests/plugins/LightDM/CMakeLists.txt 2015-04-23 00:56:41 +0000
@@ -1,8 +1,38 @@
1add_definitions(
2 -DCURRENT_SOURCE_DIR="${CMAKE_CURRENT_SOURCE_DIR}"
3 )
4include_directories(
5 ${CMAKE_CURRENT_BINARY_DIR}
6 )
7
1add_executable(GreeterDBusTestExec8add_executable(GreeterDBusTestExec
2 dbus.cpp9 dbus.cpp
3 ${CMAKE_SOURCE_DIR}/plugins/LightDM/Greeter.cpp10 ${CMAKE_SOURCE_DIR}/plugins/LightDM/Greeter.cpp
4 )11 )
5qt5_use_modules(GreeterDBusTestExec Core DBus Quick Test)12qt5_use_modules(GreeterDBusTestExec Core DBus Quick Test)
13target_link_libraries(GreeterDBusTestExec
14 -L${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm
15 -llightdm-qt5-2
16 )
17target_include_directories(GreeterDBusTestExec PUBLIC
18 ${CMAKE_SOURCE_DIR}/plugins/LightDM
19 ${CMAKE_SOURCE_DIR}/tests/mocks/LightDM
20 )
21add_binary_qml_test(GreeterDBus "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "QML2_IMPORT_PATH=${CMAKE_BINARY_DIR}/tests/mocks")
22
23add_executable(GreeterPamTestExec
24 pam.cpp
25 ${CMAKE_SOURCE_DIR}/plugins/LightDM/liblightdm/GreeterPrivate.cpp
26 )
27qt5_use_modules(GreeterPamTestExec Concurrent Core Test)
28target_link_libraries(GreeterPamTestExec
29 integratedLightDM
30 )
31target_include_directories(GreeterPamTestExec PUBLIC
32 ${CMAKE_SOURCE_DIR}/plugins/LightDM/liblightdm
33 )
34add_qmltest_target(testGreeterPam "${CMAKE_CURRENT_BINARY_DIR}/GreeterPamTestExec" TRUE FALSE)
35add_dependencies(testGreeterPam GreeterPamTestExec)
636
7add_executable(GreeterUsersModelTestExec37add_executable(GreeterUsersModelTestExec
8 usersmodel.cpp38 usersmodel.cpp
@@ -10,24 +40,13 @@
10 ${CMAKE_SOURCE_DIR}/plugins/Utils/unitysortfilterproxymodelqml.cpp40 ${CMAKE_SOURCE_DIR}/plugins/Utils/unitysortfilterproxymodelqml.cpp
11 )41 )
12qt5_use_modules(GreeterUsersModelTestExec Core Test)42qt5_use_modules(GreeterUsersModelTestExec Core Test)
1343target_link_libraries(GreeterUsersModelTestExec
14include_directories(44 -L${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm
15 ${CMAKE_CURRENT_BINARY_DIR}45 -llightdm-qt5-2
46 )
47target_include_directories(GreeterUsersModelTestExec PUBLIC
16 ${CMAKE_SOURCE_DIR}/plugins/LightDM48 ${CMAKE_SOURCE_DIR}/plugins/LightDM
17 ${CMAKE_SOURCE_DIR}/plugins/Utils49 ${CMAKE_SOURCE_DIR}/plugins/Utils
18 ${CMAKE_SOURCE_DIR}/tests/mocks/LightDM50 ${CMAKE_SOURCE_DIR}/tests/mocks/LightDM
19 )51 )
20
21target_link_libraries(GreeterDBusTestExec
22 MockLightDM
23 )
24
25target_link_libraries(GreeterUsersModelTestExec
26 MockLightDM
27 )
28
29add_definitions(-DCURRENT_SOURCE_DIR="${CMAKE_CURRENT_SOURCE_DIR}")
30
31add_binary_qml_test(GreeterDBus "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "QML2_IMPORT_PATH=${CMAKE_BINARY_DIR}/tests/mocks")
32
33add_binary_qml_test(GreeterUsersModel "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "LIBLIGHTDM_MOCK_MODE=full")52add_binary_qml_test(GreeterUsersModel "${CMAKE_BINARY_DIR}/tests/mocks/LightDM/liblightdm" MockLightDM "LIBLIGHTDM_MOCK_MODE=full")
3453
=== added file 'tests/plugins/LightDM/pam.cpp'
--- tests/plugins/LightDM/pam.cpp 1970-01-01 00:00:00 +0000
+++ tests/plugins/LightDM/pam.cpp 2015-04-23 00:56:41 +0000
@@ -0,0 +1,52 @@
1/*
2 * Copyright (C) 2015 Canonical, Ltd.
3 *
4 * This program is free software; you can redistribute it and/or modify
5 * it under the terms of the GNU General Public License as published by
6 * the Free Software Foundation; version 3.
7 *
8 * This program is distributed in the hope that it will be useful,
9 * but WITHOUT ANY WARRANTY; without even the implied warranty of
10 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11 * GNU General Public License for more details.
12 *
13 * You should have received a copy of the GNU General Public License
14 * along with this program. If not, see <http://www.gnu.org/licenses/>.
15 */
16
17#include "GreeterPrivate.h"
18
19#include <QtTest>
20
21class GreeterPamTest : public QObject
22{
23 Q_OBJECT
24
25private Q_SLOTS:
26
27 void init()
28 {
29 m_greeterpriv = new QLightDM::GreeterPrivate();
30 }
31
32 void cleanup()
33 {
34 delete m_greeterpriv;
35 QTRY_COMPARE(QThreadPool::globalInstance()->activeThreadCount(), 0);
36 }
37
38 void testRapidFireAuthentication()
39 {
40 m_greeterpriv->authenticationUser = qgetenv("USER");
41 for (int i = 0; i < 100; i++) {
42 m_greeterpriv->handleAuthenticate();
43 }
44 }
45
46private:
47 QLightDM::GreeterPrivate *m_greeterpriv;
48};
49
50QTEST_GUILESS_MAIN(GreeterPamTest)
51
52#include "pam.moc"

Subscribers

People subscribed via source and target branches