Merge lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime into lp:unity-scopes-api/devel

[Marcus Tomlinson]

Updated RegistryObject to monitor and maintain all scope process states, as to properly launch and shutdown processes accordingly.

Updated RegistryObject to monitor and maintain all scope process states using process-cpp.

On locate(), RegistryObject checks the process' current running state and executes the appropriate action: stopping -> wait, running -> return, stopped -> start.

A background "death observer" thread (provided by process-cpp) is instantiated on entry to scoperegistry and provided to RegistryObject in order to catch child death events asynchronously.

The inherited sig mask from scoperegistry (sigterm, sigchld, etc.) is cleared on entry to scoperunner in order to provide scopes with a clean runtime environment.

The correct tear-down of scope processes is handled on destruction of RegistryObject.

Slow reacting processes are handled with appropriate timeouts.

Tests added to verify new RegistryObject functionality.

[Marcus Tomlinson (marcustomlinson)]

The changes to process-cpp required for this branch have now landed. Ready for review.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:265
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/256/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/257/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/256/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/256/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/256/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

266. By Marcus Tomlinson on 2014-03-12

Fixed RegistryI_test

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:266
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/257/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/258/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/257/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/257/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/257/rebuild

review: Needs Fixing (continuous-integration)

[Paweł Stołowski (stolowski)]

611 + // broadcast message to all scope_processes_

Why do we do that? Can you add a comment in the code explaining it?

review: Needs Information

[Paweł Stołowski (stolowski)]

661 +void RegistryObject::ScopeProcess::exec()
662 +{
..
665 + if (state() == ScopeProcess::Running)
..
670 + else if (state() == ScopeProcess::Stopping)
..
681 + update_state(Starting);

Shouldn't we be locking state for the entire scope of this method, rather then only when reading / updating the state? I think we have a potential race in exec() if there are two concurrent calls to the registry for same scope id - between the time we examine state and update it, both callers can find out the scope is not yet running (nor stopping) and will be trying to start scope proces)?

720 + // 4. add the scope process to the death observer
721 + core::posix::ChildProcess::DeathObserver::instance().add(process_);
722 +}

Shouldn't this happen early (before we start the process)? What if it dies shortly after we check that the state is Running in line #706, but before we register the observer?

review: Needs Information

[Paweł Stołowski (stolowski)]

711 + if (!wait_for_state(ScopeProcess::Running, 1000))
712 + {
713 + kill();
714 + throw unity::ResourceException("RegistryObject::ScopeProcess: exec() aborted. Scope: \""
715 + + exec_data_.scope_id + "\" took too long to start.");
716 + }

Please consider catching exception from kill() here.

[Paweł Stołowski (stolowski)]

694 + process_ = core::posix::exec(program, argv, env, core::posix::StandardStream::empty);

I think we should pass stdout|stderr, or otherwise we'll loose all error messages from scopes, no?

review: Needs Information

[Michi Henning (michihenning)]

Currently, locate() locks on entry. If we have several requests to locate() concurrently, such as when an aggregator wants to talk to several children, start-up of the children will be serialised, as far as I can see: the second scope won't be exec'd until after the first scope has done its exec() and is ready. I think we need to be able to fire execs concurrently, otherwise things will take too long.

On the binding side, I'm planning to wait at most 1.5 seconds for the exec to complete before giving up. So, the sum of start-up times of any scopes being activated together must not exceed that limit.

[Marcus Tomlinson (marcustomlinson)]

> 720 + // 4. add the scope process to the death observer
> 721 +
> core::posix::ChildProcess::DeathObserver::instance().add(process_);
> 722 +}
>
> Shouldn't this happen early (before we start the process)? What if it dies
> shortly after we check that the state is Running in line #706, but before we
> register the observer?

There is never a time where you can add a ChildProcess to the death observer early enough. See, a ChildProcess can only be constructed after it's process is started, which is already too late as the process may have crashed immediately (hence the death observer misses that signal).

This was sorted out in process-cpp so that when an already dead process is added it will fire the on_process_death signal, which in turn, will then execute the correct ChildProcess tear down routine.

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

267. By Marcus Tomlinson on 2014-03-13

Merged devel

268. By Marcus Tomlinson on 2014-03-13

Beefed up the comments in RegistryObject::on_process_death

269. By Marcus Tomlinson on 2014-03-13

Fixed warning in documentation.

270. By Marcus Tomlinson on 2014-03-13

Reduced mutexes in RegistryObject::ScopeProcess to a single process_mutex_ to better safety-proof and simplify concurrent request handling.

271. By Marcus Tomlinson on 2014-03-13

Give stdout|stderr handles to child processes in order to pipe their outputs to the registry's output stream.

272. By Marcus Tomlinson on 2014-03-13

Reduced scope of the mutex lock in RegistryObject::locate() as to avoid a pile-up on multiple locate() requests for different scopes.

273. By Marcus Tomlinson on 2014-03-13

Catch exceptions from kill() calls in ScopeProcess::exec().

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:271
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/265/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/266/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/265/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/265/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/265/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

274. By Marcus Tomlinson on 2014-03-13

If a kill hard fails, detach from the process and move on.

275. By Marcus Tomlinson on 2014-03-13

DeathObserver::quit() throws, so try...catch added to the RegistryObject destructor.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:273
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/266/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/267/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/266/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/266/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/266/rebuild

review: Needs Fixing (continuous-integration)

[Marcus Tomlinson (marcustomlinson)]

Alright, I think I've addressed all review comments so far.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:275
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/267/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/268/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/267/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/267/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/267/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

276. By Marcus Tomlinson on 2014-03-13

Updated symbols file.

[Paweł Stołowski (stolowski)]

786 + ///! TODO: This should not be here. A ready signal from the scope should trigger "running".

This would be nice to have; any reason no to fix it right away?

review: Needs Information

[Michi Henning (michihenning)]

> This would be nice to have; any reason no to fix it right away?

Mea culpa. The communication from the scope to the registry doesn't exist yet :-(

[Marcus Tomlinson (marcustomlinson)]

> 786 + ///! TODO: This should not be here. A ready signal from the scope
> should trigger "running".
>
> This would be nice to have; any reason no to fix it right away?

This will be the next branch I'm working on. It's not a trivial change so I think it's better we separate them.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:276
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/270/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/271
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/270
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/270/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/270

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/270/rebuild

review: Approve (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

277. By Marcus Tomlinson on 2014-03-13

in_lock_* methods renamed to *_unlocked

[Michal Hruby (mhr3)]

So, I tried to run this against the tests in lp:unity-scopes-shell (because yey for no end-to-end registry tests in -api), and it was behaving strangely - running make test sometimes finished in 1.0seconds, other times in 1.8 seconds, and sometimes after 30seconds.

Possibly (just) a cleanup issue as when looking at the processes when the test was taking 30seconds to finish, the scope process was a zombie after about a second of running the tests.

review: Needs Fixing

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:277
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/274/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/275
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/274
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/274/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/274

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/274/rebuild

review: Approve (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

278. By Marcus Tomlinson on 2014-03-14

Detach from death_observer_thread_ if quit throws.

[Marcus Tomlinson (marcustomlinson)]

> So, I tried to run this against the tests in lp:unity-scopes-shell (because
> yey for no end-to-end registry tests in -api), and it was behaving strangely -
> running make test sometimes finished in 1.0seconds, other times in 1.8
> seconds, and sometimes after 30seconds.
>
> Possibly (just) a cleanup issue as when looking at the processes when the test
> was taking 30seconds to finish, the scope process was a zombie after about a
> second of running the tests.

Ok so firstly, the 1 to 2 second delay I think is a scoperunner issue. Sometimes a scoperunner, after being "kill" signalled, hangs around like a zombie. I send a kill signal and wait (currently for 1s) for the process to die. If it doesn't die within that period, I print the following message, and move on (there is little more I can do at this point):

"Scope: "x" is taking longer than expected to terminate (This process is likely to close upon termination of the parent application)."

I think the 30s shutdown (which I believe is actually just a deadlock timing out in your test), is due to the death observer thread missing the quit() request in the RegistryObject destructor. I'm busy fixing that in process-cpp now.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:278
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/275/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/276
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/275
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/275/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/275

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/275/rebuild

review: Approve (continuous-integration)

[Paweł Stołowski (stolowski)]

Besides the deadlock issue (a problem with process-cpp), my only request for now as discussed on the standup would be to have a stress test that starts a bunch of scopes and tries to start same scope multiple times (simulate a case where e.g. a few clients try to hit same scope at the same time, making sure we have just one such scope process running).

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

279. By Marcus Tomlinson on 2014-03-17

Added test for RegistryObject::locate

280. By Marcus Tomlinson on 2014-03-17

Added checks for process count during locate test.

[Marcus Tomlinson (marcustomlinson)]

> Besides the deadlock issue (a problem with process-cpp), my only request for
> now as discussed on the standup would be to have a stress test that starts a
> bunch of scopes and tries to start same scope multiple times (simulate a case
> where e.g. a few clients try to hit same scope at the same time, making sure
> we have just one such scope process running).

Alright, the process-cpp issue has been MR-ed and will soon be landing.
I also added the requested tests.

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

281. By Marcus Tomlinson on 2014-03-17

Updated symbols

282. By Marcus Tomlinson on 2014-03-17

Merged devel

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:280
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/281/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/282/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/281/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/281/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/281/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

283. By Marcus Tomlinson on 2014-03-17

Some formatting fixes

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:282
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/283/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/284
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/283
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/283/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/283

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/283/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:283
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/284/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/285
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/284
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/284/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/284

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/284/rebuild

review: Approve (continuous-integration)

[Paweł Stołowski (stolowski)]

Thanks for the test, great stuff!!

One more question re the test - could add one more case to it: after all 6 scopes get started, kill/stop one of them with SIGQUIT/SIGINT etc., then ensure the process is gone (5 scopes active), and then locate it again to verify it's started again?

review: Needs Information

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

284. By Marcus Tomlinson on 2014-03-17

Added checks for scope process deaths to locate test.

[Marcus Tomlinson (marcustomlinson)]

> Thanks for the test, great stuff!!
>
> One more question re the test - could add one more case to it: after all 6
> scopes get started, kill/stop one of them with SIGQUIT/SIGINT etc., then
> ensure the process is gone (5 scopes active), and then locate it again to
> verify it's started again?

Alright, done. Looking into this actually revealed a mistake in the registry where removing a scope didn't kill it's process. Fixed that too.

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

285. By Marcus Tomlinson on 2014-03-17

Updated kill() logic to first try terminate the scope gracefully, then kill only if it takes too long to close.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:284
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/285/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/286/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/285/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/285/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/285/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

286. By Marcus Tomlinson on 2014-03-17

Wait for scope to properly die after calling kill in locate test.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:285
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/286/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/287/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/286/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/286/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/286/rebuild

review: Needs Fixing (continuous-integration)

[Paweł Stołowski (stolowski)]

This looks good to me! Thanks!

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:286
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/287/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/288/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/287/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/287/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/287/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

287. By Marcus Tomlinson on 2014-03-17

Wait for the SIGCHLD signal to reach the registry

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:287
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/288/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/289
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/288
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/288/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/288

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/288/rebuild

review: Approve (continuous-integration)

[Paweł Stołowski (stolowski)]

Hmm, I'm having issues when running scoperegistry + demo scopes from this branch. Stopping the registry gives errors on some scope, and then trying to run registry again gives errors about some addresses still in use.

pawel@trusty ~/src/ubuntu/unity-scopes-api/scope-cfg-runner/build/demo $ ../scoperegistry/scoperegistry Runtime.ini
scoperegistry: unity::ResourceException: cannot open scope installation directory "/custom/usr/lib/x86_64-linux-gnu/unity/scopes": No such file or directory
scoperegistry: could not open OEM installation directory, ignoring OEM scopes
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-A" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-B" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-C" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-D" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-N" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-S" started
^CRegistryObject::ScopeProcess::in_lock_kill(): Failed to kill scope: "scope-S"
RegistryObject::ScopeProcess::~ScopeProcess(): unity::ResourceException: Scope: "scope-S" is taking longer than expected to terminate (This process is likely to close upon termination of the parent application).
pawel@trusty ~/src/ubuntu/unity-scopes-api/scope-cfg-runner/build/demo $ ../scoperegistry/scoperegistry Runtime.ini
scoperegistry: unity::ResourceException: cannot open scope installation directory "/custom/usr/lib/x86_64-linux-gnu/unity/scopes": No such file or directory
scoperegistry: could not open OEM installation directory, ignoring OEM scopes
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-A" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-B" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-C" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-D" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-N" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-S" started
~ObjectAdapter(): exception from shutdown(): unity::scopes::MiddlewareException: Object adapter in Failed state (adapter: scope-B)
~ObjectAdapter(): exception from wait_for_shutdown(): unity::scopes::MiddlewareException: Object adapter in Failed state (adapter: scope-B)
unexpected exception in add_scope_object(): unity::scopes::MiddlewareException: ObjectAdapter::run_workers(): broker thread failure (adapter: scope-B):
    unity::scopes::MiddlewareException: ObjectAdapter: broker thread failure (adapter: scope-B): address in use: ipc:///tmp/scope-B
~ObjectAdapter(): exception from shutdown(): unity::scopes::MiddlewareException: Object adapter in Failed state (adapter: scope-A)
~ObjectAdapter(): exception from wait_for_shutdown(): unity::scopes::MiddlewareException: Object adapter in Failed state (adapter: scope-A)
unexpected exception in add_scope_object(): unity::scopes::MiddlewareException: ObjectAdapter::run_workers(): broker thread failure (adapt...

review: Needs Fixing

[Michal Hruby (mhr3)]

948 + while (state_ != state && time_left > 0)
949 + {
950 + auto start = std::chrono::high_resolution_clock::now();
951 + state_change_cond_.wait_for(lock, std::chrono::milliseconds(time_left));
952 +
953 + // update time left
954 + time_left -= std::chrono::duration_cast<std::chrono::milliseconds>(
955 + std::chrono::high_resolution_clock::now() - start).count();
956 + }

Why so complicated? condvar.wait_for(lock, timeout, [](){return state_ == state;}); Would do, no?

Why don't the clear_handle_unlocked and update_state_unlocked don't have the unique_lock as a param? Would make the semantics safer.

review: Needs Fixing

[Michal Hruby (mhr3)]

942 +bool RegistryObject::ScopeProcess::wait_for_state_unlocked(std::unique_lock<std::mutex>& lock,

Also, this is a bit confusing, has the _unlocked suffix, yet it drops the mutex, when waiting for the condvar.

[Marcus Tomlinson (marcustomlinson)]

> 948 + while (state_ != state && time_left > 0)
> 949 + {
> 950 + auto start = std::chrono::high_resolution_clock::now();
> Why so complicated? condvar.wait_for(lock, timeout, [](){return state_ ==
> state;}); Would do, no?

No. [](){return state_ == state;} is just a predicate to check before locking, which is handled by the if() statement anyway. This wait_for can be woken by any state change, not necessarily the one we want, that's why I need to return to wait_for in a loop until we do get the state we want

> Why don't the clear_handle_unlocked and update_state_unlocked don't have the
> unique_lock as a param? Would make the semantics safer.

Those methods take in the lock because they need to use it in the wait_for, the others don't use it they just need it locked before they're called.

[Marcus Tomlinson (marcustomlinson)]

> 942 +bool RegistryObject::ScopeProcess::wait_for_state_unlocked(std::uniqu
> e_lock<std::mutex>& lock,
>
> Also, this is a bit confusing, has the _unlocked suffix, yet it drops the
> mutex, when waiting for the condvar.

Hense the original "in_lock_*" prefix. These methods simply must be called while process_mutex_ is locked.

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

288. By Marcus Tomlinson on 2014-03-18

Kill scope processes one-by-one in the RegistryObject destructor (Just calling clear on the map is not safe)

289. By Marcus Tomlinson on 2014-03-18

Cleaned up wait_for_state_unlocked() logic.

[Michal Hruby (mhr3)]

> > 942 +bool
> RegistryObject::ScopeProcess::wait_for_state_unlocked(std::uniqu
> > e_lock<std::mutex>& lock,
> >
> > Also, this is a bit confusing, has the _unlocked suffix, yet it drops the
> > mutex, when waiting for the condvar.
>
> Hense the original "in_lock_*" prefix. These methods simply must be called
> while process_mutex_ is locked.

Then I'd suggest using overloaded method instead (just for this): one wait_for_state() without the unique_lock param and one with it (of course one will call the other).

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

290. By Marcus Tomlinson on 2014-03-18

Cleaned up kill_unlocked() method

291. By Marcus Tomlinson on 2014-03-18

wait_for_state_unlock() and kill_unlock() renamed to wait_for_state() and kill() as the lock is implicitly required.

[Marcus Tomlinson (marcustomlinson)]

> 948 + while (state_ != state && time_left > 0)
> 949 + {
> 950 + auto start = std::chrono::high_resolution_clock::now();
> Why so complicated? condvar.wait_for(lock, timeout, [](){return state_ ==
> state;}); Would do, no?

done.

> Then I'd suggest using overloaded method instead (just for this): one
> wait_for_state() without the unique_lock param and one with it (of course one
> will call the other).

done.

[Marcus Tomlinson (marcustomlinson)]

> Hmm, I'm having issues when running scoperegistry + demo scopes from this
> branch. Stopping the registry gives errors on some scope, and then trying to
> run registry again gives errors about some addresses still in use.

I'm really sorry about this pawel, this was due to a late change I made to the RegistryObject destructor. I've fixed this (again).

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

292. By Marcus Tomlinson on 2014-03-18

Updated symbols

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:289
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/289/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/290
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/289/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/289

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/289/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:292
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/290/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/291
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/290
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/290/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/290

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/290/rebuild

review: Approve (continuous-integration)

[Paweł Stołowski (stolowski)]

There is still something weird going on when I stop scoperegistry with Ctrl+C.. No more leftover proccesses, but instead every single demo scope gives:

^CRegistryObject::ScopeProcess::kill(): Scope: "scope-A" is taking longer than expected to terminate gracefully. Killing the process instead.
RegistryObject::ScopeProcess::kill(): Scope: "scope-B" is taking longer than expected to terminate gracefully. Killing the process instead.
RegistryObject::ScopeProcess::kill(): Scope: "scope-C" is taking longer than expected to terminate gracefully. Killing the process instead.
RegistryObject::ScopeProcess::kill(): Scope: "scope-D" is taking longer than expected to terminate gracefully. Killing the process instead.
RegistryObject::ScopeProcess::kill(): Scope: "scope-N" is taking longer than expected to terminate gracefully. Killing the process instead.
RegistryObject::ScopeProcess::kill(): Scope: "scope-S" is taking longer than expected to terminate gracefully. Killing the process instead.

These scope are very simple, so I find it hard to believe... On a side note, all this happens in sequence, so it takes (num_of_scopes * timeout) for shutdown (but I'm not going to block on this problem right now).

Another question: is this still expected that all scopes are automatically started when registry is started? Wasn't this branch supposed to change that so that first call to locate() starts a scope?

BTW, I'm running up-to-date trusty, libprocess-cpp is 0.0.1+14.04.20140317-0ubuntu1.

review: Needs Information

[Michal Hruby (mhr3)]

I can confirm what Pawel is saying, there's still some weirdness going on, namely:

1) scoperunners don't terminate after receiving SIGTERM
2) I do not see the log line "RegistryObject::ScopeProcess::on_process_death(): ..." after sending the scoperunner process SIGKILL.
3) After sending SIGKILL I still see the process in ps list as a zombie (ie waitpid not called)
4) Sending a request after sending SIGKILL to scoperunner doesn't restart the scope (although I guess that's because the registry doesn't realize it's dead (see 2)). Is that supposed to work with this branch?

review: Needs Fixing

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

293. By Marcus Tomlinson on 2014-03-19

Added internal SigTermHandler class to handle SIGTERM signal in scoperegistry, scoperunner and smartscopesproxy.

[Marcus Tomlinson (marcustomlinson)]

> Another question: is this still expected that all scopes are automatically
> started when registry is started? Wasn't this branch supposed to change that
> so that first call to locate() starts a scope?

Yes, unfortunately that's how we still need to do it. This requires rebinding logic to be added to ZmqObjectProxy::invoke_. I will be working on that and the scope-to-reg signals over the next 2 weeks.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:293
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/291/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/292
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/291
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/291/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/291

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/291/rebuild

review: Approve (continuous-integration)

[Marcus Tomlinson (marcustomlinson)]

> I can confirm what Pawel is saying, there's still some weirdness going on,
> namely:
>
> 1) scoperunners don't terminate after receiving SIGTERM

fixed.

> 2) I do not see the log line
> "RegistryObject::ScopeProcess::on_process_death(): ..." after sending the
> scoperunner process SIGKILL.
> 3) After sending SIGKILL I still see the process in ps list as a zombie (ie
> waitpid not called)

There seems to be yet another iffy situation in process-cpp where it just misses SIGCHLD signals :( I've debugged scoperunner on killing it and it exits properly, the ::poll in the death observer just doesn't react to it?? I don't know if you have any ideas?

> 4) Sending a request after sending SIGKILL to scoperunner doesn't restart the
> scope (although I guess that's because the registry doesn't realize it's dead
> (see 2)). Is that supposed to work with this branch?

Re-binding doesn't work yet (see last reply to Pawel)

[Michal Hruby (mhr3)]

8 - (c++)"unity::scopes::Runtime::create(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
10 + (c++)"unity::scopes::Runtime::create(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0replaceme

Something wrong there, pls revert (did gensymbols decide to sort the symbols differently this time?)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

294. By Marcus Tomlinson on 2014-03-19

Reverted Runtime::create replacement in symbols

[Marcus Tomlinson (marcustomlinson)]

> 8 - (c++)"unity::scopes::Runtime::create(std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&)@Base"
> 0.4.0+14.04.20140312.1
> 10 + (c++)"unity::scopes::Runtime::create(std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&)@Base" 0replaceme
>
> Something wrong there, pls revert (did gensymbols decide to sort the symbols
> differently this time?)

K, reverted that one. Yeah, looks like gensymbols reordered one other symbol too, but it's a "0replaceme" one anyways.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:294
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/292/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/293
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/292
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/292/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/292

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/292/rebuild

review: Approve (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

295. By Marcus Tomlinson on 2014-03-20

Updated RegistryObject::ScopeProcess::state() to const

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:295
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/297/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/298/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/297/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/297/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/297/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

296. By Marcus Tomlinson on 2014-03-20

Updated symbols

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:296
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/298/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/299
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/298
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/298/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/298

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/298/rebuild

review: Approve (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

297. By Marcus Tomlinson on 2014-03-24

Merged lp:~thomas-voss/unity-scopes-api/socpe_process_lifetime

298. By Marcus Tomlinson on 2014-03-24

Merged devel and fixed conflicts

299. By Marcus Tomlinson on 2014-03-24

Fixed resolve mistake

[Marcus Tomlinson (marcustomlinson)]

Alright, this is ready for review again.
Note: lp:~thomas-voss/process-cpp/fix-death-observer is required in conjunction with this branch.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:297
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/305/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/306/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/305/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/305/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/305/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

300. By Marcus Tomlinson on 2014-03-24

Fixed bzr bd issues

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:299
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/306/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/307/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/306/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/306/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/306/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:300
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/307/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/308/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/307/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/307/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/307/rebuild

review: Needs Fixing (continuous-integration)

[Michal Hruby (mhr3)]

1932 + unity::scopes::internal::SigTermHandler::*;

Please remove.

Otherwise it's looking fine, but needs to wait for process-cpp branches to land first. Is there going to be version bump in process-cpp? Build deps should be updated here.

review: Needs Fixing

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

301. By Marcus Tomlinson on 2014-03-25

Added comment to explain tear down sequence.

302. By Marcus Tomlinson on 2014-03-25

Removed SigTermHandler from unity-scopes.map

303. By Marcus Tomlinson on 2014-03-25

Updated build deps for process-cpp 1.0.0

[Marcus Tomlinson (marcustomlinson)]

> 1932 + unity::scopes::internal::SigTermHandler::*;
>
> Please remove.

ah yes, nice catch. done.

> Otherwise it's looking fine, but needs to wait for process-cpp branches to
> land first. Is there going to be version bump in process-cpp? Build deps
> should be updated here.

build deps updated.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:302
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/309/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/310/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/309/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/309/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/309/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:303
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/310/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/311/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/310/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/310/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/310/rebuild

review: Needs Fixing (continuous-integration)

[Michal Hruby (mhr3)]

Looks good to me now, still need to wait for process-cpp though.

review: Approve

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

304. By Marcus Tomlinson on 2014-03-26

Merged devel

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:304
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/313/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/314/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/313/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/313/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/313/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

305. By Marcus Tomlinson on 2014-03-26

Merged trunk and resolved conflicts

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:305
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/314/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/315/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/314/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/314/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/314/rebuild

review: Needs Fixing (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

306. By Marcus Tomlinson on 2014-03-28

Clear the sig mask (inherited from scoperegistry) on entry.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:306
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/316/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/317
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/316
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/316/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/316

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/316/rebuild

review: Approve (continuous-integration)

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

307. By Marcus Tomlinson on 2014-03-28

Merged trunk

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:307
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/317/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/318
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/317
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/317/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/317

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/317/rebuild

review: Approve (continuous-integration)

[Paweł Stołowski (stolowski)]

Hmm, still no luck with it; I'm testing it with demo scopes and kill scope-A manually with kill <pid of scope>:

pawel@trusty ~/src/ubuntu/unity-scopes-api/scope_process_lifetime/build/demo $ ../scoperegistry/scoperegistry Runtime.ini
scoperegistry: unity::ResourceException: cannot open scope installation directory "/custom/usr/lib/x86_64-linux-gnu/unity/scopes": No such file or directory
scoperegistry: could not open OEM installation directory, ignoring OEM scopes
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-A" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-B" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-C" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-D" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-N" started
RegistryObject::ScopeProcess::exec(): Process for scope: "scope-S" started
RegistryObject::ScopeProcess::on_process_death(): Process for scope: "scope-A" terminated

The death of scope-A is correctly detected, but I tried several times to query it after killing it, and getting this:

pawel@trusty ~/src/ubuntu/unity-scopes-api/scope_process_lifetime/build/demo $ ./client scope-A q
Scope metadata:
        scope_id: scope-A
        display_name: scope-A.DisplayName
        description: scope-A.Description
        art: scope-A.Art
        icon: scope-A.Icon
        search_hint: scope-A.SearchHint
        hot_key: scope-A.HotKey
query complete, status: error: unity::scopes::TimeoutException: Request timed out after 300 milliseconds
unity::scopes::TimeoutException: Request timed out after 300 milliseconds

And the scope is not started.

NB, I'm running it with the fix_env_var_split branc of libprocess-cpp.

review: Needs Fixing

[Marcus Tomlinson (marcustomlinson)]

> Hmm, still no luck with it; I'm testing it with demo scopes and kill scope-A
> manually with kill <pid of scope>:
>
> pawel@trusty ~/src/ubuntu/unity-scopes-api/scope_process_lifetime/build/demo $
> ../scoperegistry/scoperegistry Runtime.ini
> scoperegistry: unity::ResourceException: cannot open scope installation
> directory "/custom/usr/lib/x86_64-linux-gnu/unity/scopes": No such file or
> directory
> scoperegistry: could not open OEM installation directory, ignoring OEM scopes
> RegistryObject::ScopeProcess::exec(): Process for scope: "scope-A" started
> RegistryObject::ScopeProcess::exec(): Process for scope: "scope-B" started
> RegistryObject::ScopeProcess::exec(): Process for scope: "scope-C" started
> RegistryObject::ScopeProcess::exec(): Process for scope: "scope-D" started
> RegistryObject::ScopeProcess::exec(): Process for scope: "scope-N" started
> RegistryObject::ScopeProcess::exec(): Process for scope: "scope-S" started
> RegistryObject::ScopeProcess::on_process_death(): Process for scope: "scope-A"
> terminated
>
> The death of scope-A is correctly detected, but I tried several times to query
> it after killing it, and getting this:
>
> pawel@trusty ~/src/ubuntu/unity-scopes-api/scope_process_lifetime/build/demo $
> ./client scope-A q
> Scope metadata:
> scope_id: scope-A
> display_name: scope-A.DisplayName
> description: scope-A.Description
> art: scope-A.Art
> icon: scope-A.Icon
> search_hint: scope-A.SearchHint
> hot_key: scope-A.HotKey
> query complete, status: error: unity::scopes::TimeoutException: Request timed
> out after 300 milliseconds
> unity::scopes::TimeoutException: Request timed out after 300 milliseconds
>
> And the scope is not started.
>
> NB, I'm running it with the fix_env_var_split branc of libprocess-cpp.

As discussed on IRC, this branch essentially just replicates what we already had using process-cpp. This implementation is really just a more correct / cleaner version of the previous one. We now have timeouts on process operations, terminate using sigterm instead of jumping straight to sigkill, keep track of all process states, and one could argue that we now have a better separation of concerns (between process-cpp and unity-scopes-api).

Other items on the todo list to come in other branches are:
1. re-binding logic on failure to execute a query (call locate() to ensure a scope is running).
2. introduce scope-to-registry signals to inform registry of changes in the scope's running state.
3. address idle timeouts of hanging / non-responding scopes.

[lp:~marcustomlinson/unity-scopes-api/scope_process_lifetime]

308. By Marcus Tomlinson on 2014-03-28

Don't terminate scoperunner on sigint, as this disables one from debugging their scope.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:308
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/318/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-amd64-ci/319
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/318
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-armhf-ci/318/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-trusty-i386-ci/318

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/318/rebuild

review: Approve (continuous-integration)

[Paweł Stołowski (stolowski)]

OK, I run this on the phone and didn't spot any regressions. Let's get it in.

review: Approve

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2014-03-12 16:41:30 +0000
+++ CMakeLists.txt	2014-03-28 15:41:24 +0000
@@ -56,7 +56,7 @@
 include(FindPkgConfig)
 find_package(Boost COMPONENTS regex serialization REQUIRED)
 pkg_check_modules(UNITY_API libunity-api>=0.1.1 REQUIRED)
-pkg_check_modules(PROCESS_CPP process-cpp REQUIRED)
+pkg_check_modules(PROCESS_CPP process-cpp>=1.0.0 REQUIRED)
 pkg_check_modules(LTTNG_UST lttng-ust REQUIRED)
 pkg_check_modules(LIBURCU_BP liburcu-bp REQUIRED)
 find_program(LTTNG_EXECUTABLE lttng)

=== modified file 'debian/changelog'
--- debian/changelog	2014-03-20 12:27:33 +0000
+++ debian/changelog	2014-03-28 15:41:24 +0000
@@ -1,3 +1,9 @@
+unity-scopes-api (0.4.1+14.04.20140326.1-0ubuntu1) trusty; urgency=low
+
+  * New rebuild forced
+
+ -- Ubuntu daily release <[e-mail]>  Wed, 26 Mar 2014 17:57:37 +0000
+
 unity-scopes-api (0.4.1-0ubuntu1) UNRELEASED; urgency=medium
 
   [ Pawel Stolowski ]
@@ -5,12 +11,22 @@
 
  -- Pawel Stolowski <[e-mail]>  Wed, 12 Mar 2014 17:03:16 +0100
 
+unity-scopes-api (0.4.0+14.04.20140324-0ubuntu1) trusty; urgency=low
+
+  [ Michal Hruby ]
+  * Backport invalidation raciness fix.
+
+ -- Ubuntu daily release <[e-mail]>  Mon, 24 Mar 2014 13:06:04 +0000
+
 unity-scopes-api (0.4.0+14.04.20140319-0ubuntu1) trusty; urgency=low
 
   [ Michal Hruby ]
   * Backport the invalidation fix.
 
- -- Ubuntu daily release <[e-mail]>  Wed, 19 Mar 2014 16:39:49 +0000
+  [ Ubuntu daily release ]
+  * debian/*symbols: auto-update new symbols to released version
+
+ -- Ubuntu daily release <[e-mail]>  Mon, 24 Mar 2014 13:06:02 +0000
 
 unity-scopes-api (0.4.0+14.04.20140312.2-0ubuntu1) trusty; urgency=low
 

=== modified file 'debian/control'
--- debian/control	2014-03-03 11:50:58 +0000
+++ debian/control	2014-03-28 15:41:24 +0000
@@ -9,7 +9,7 @@
                pkg-config,
                python3,
                capnproto,
-               libprocess-cpp-dev,
+               libprocess-cpp-dev (>= 1.0.0),
                libunity-api-dev (>= 7.80.5~),
                libboost-regex-dev,
                libboost-serialization-dev,

=== modified file 'debian/libunity-scopes0.symbols'
--- debian/libunity-scopes0.symbols	2014-03-21 15:18:32 +0000
+++ debian/libunity-scopes0.symbols	2014-03-28 15:41:24 +0000
@@ -279,7 +279,7 @@
  (c++)"unity::scopes::Result::operator=(unity::scopes::Result&&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::Result::operator=(unity::scopes::Result const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::Result::operator[](std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::Runtime::create_scope_runtime(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0replaceme
+ (c++)"unity::scopes::Runtime::create_scope_runtime(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140324
  (c++)"unity::scopes::Runtime::create(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::Runtime::destroy()@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::Runtime::run_scope(unity::scopes::ScopeBase*)@Base" 0.4.0+14.04.20140312.1
@@ -389,11 +389,9 @@
  (c++)"unity::scopes::internal::smartscopes::SSScopeObject::activate(unity::scopes::Result const&, unity::scopes::ActionMetadata const&, std::shared_ptr<unity::scopes::internal::MWReply> const&, unity::scopes::internal::InvokeInfo const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSScopeObject::SSScopeObject(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<unity::scopes::internal::MiddlewareBase>, std::shared_ptr<unity::scopes::internal::smartscopes::SSRegistryObject>)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSScopeObject::~SSScopeObject()@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::get_metadata(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::refresh_thread()@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::get_remote_scopes()@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::add(unity::scopes::internal::smartscopes::RemoteScope const&, unity::scopes::ScopeMetadata const&, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unity::scopes::ScopeMetadata, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unity::scopes::ScopeMetadata> > >&, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&)@Base" 0replaceme
- (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::list()@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::add(unity::scopes::internal::smartscopes::RemoteScope const&, unity::scopes::ScopeMetadata const&, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unity::scopes::ScopeMetadata, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unity::scopes::ScopeMetadata> > >&, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&)@Base" 0.4.0+14.04.20140324
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::locate(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::SSRegistryObject(std::shared_ptr<unity::scopes::internal::MiddlewareBase>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::~SSRegistryObject()@Base" 0.4.0+14.04.20140312.1
@@ -402,17 +400,25 @@
  (c++)"unity::scopes::internal::RegistryConfig::REGISTRY_CONFIG_GROUP@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryConfig::RegistryConfig(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryConfig::~RegistryConfig()@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::spawn_scope(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::get_metadata(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::kill_process(int)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::add_local_scope(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unity::scopes::ScopeMetadata const&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::on_process_death(int)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::clear_handle_unlocked()@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::update_state_unlocked(unity::scopes::internal::RegistryObject::ScopeProcess::ProcessState)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::exec(core::posix::ChildProcess::DeathObserver&)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::kill(std::unique_lock<std::mutex>&)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::kill()@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::ScopeProcess(unity::scopes::internal::RegistryObject::ScopeExecData)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::ScopeProcess(unity::scopes::internal::RegistryObject::ScopeProcess const&)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::ScopeProcess(unity::scopes::internal::RegistryObject::ScopeExecData)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::ScopeProcess(unity::scopes::internal::RegistryObject::ScopeProcess const&)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::~ScopeProcess()@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeExecData::~ScopeExecData()@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::add_local_scope(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unity::scopes::ScopeMetadata const&, unity::scopes::internal::RegistryObject::ScopeExecData const&)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::is_scope_running(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::on_process_death(core::posix::Process const&)@Base" 0replaceme
  (c++)"unity::scopes::internal::RegistryObject::remove_local_scope(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryObject::set_remote_registry(std::shared_ptr<unity::scopes::internal::MWRegistry> const&)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::list()@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryObject::locate(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::is_dead(int)@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::shutdown()@Base" 0.4.0+14.04.20140312.1
- (c++)"unity::scopes::internal::RegistryObject::RegistryObject()@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::RegistryObject::RegistryObject(core::posix::ChildProcess::DeathObserver&)@Base" 0replaceme
  (c++)"unity::scopes::internal::RegistryObject::~RegistryObject()@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::MiddlewareFactory::MiddlewareData::~MiddlewareData()@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::MiddlewareFactory::to_kind(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@Base" 0.4.0+14.04.20140312.1
@@ -580,6 +586,7 @@
  (c++)"unity::scopes::Category::icon() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::Category::title() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::Category::serialize() const@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::RuntimeImpl::configfile() const@Base" 0replaceme
  (c++)"unity::scopes::internal::RuntimeImpl::reply_reaper() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RuntimeImpl::proxy_to_string(std::shared_ptr<unity::scopes::Object> const&) const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RuntimeImpl::string_to_proxy(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0.4.0+14.04.20140312.1
@@ -604,7 +611,9 @@
  (c++)"unity::scopes::internal::ScopeLoader::name() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::ScopeLoader::libpath() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::get_base_url(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::get_metadata(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0replaceme
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::get_ssclient() const@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::list() const@Base" 0replaceme
  (c++)"unity::scopes::internal::smartscopes::SSRegistryObject::has_scope(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::MiddlewareBase::runtime() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryConfig::endpointdir() const@Base" 0.4.0+14.04.20140312.1
@@ -617,6 +626,11 @@
  (c++)"unity::scopes::internal::RegistryConfig::mw_kind() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryConfig::endpoint() const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::RegistryConfig::identity() const@Base" 0.4.0+14.04.20140312.1
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::wait_for_state(unity::scopes::internal::RegistryObject::ScopeProcess::ProcessState, int) const@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::wait_for_state(std::unique_lock<std::mutex>&, unity::scopes::internal::RegistryObject::ScopeProcess::ProcessState, int) const@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::ScopeProcess::state() const@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::get_metadata(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0replaceme
+ (c++)"unity::scopes::internal::RegistryObject::list() const@Base" 0replaceme
  (c++)"unity::scopes::internal::MiddlewareFactory::find_unlocked(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::MiddlewareFactory::find(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0.4.0+14.04.20140312.1
  (c++)"unity::scopes::internal::MiddlewareFactory::find(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@Base" 0.4.0+14.04.20140312.1

=== modified file 'demo/scopes/scope-A/scope-A.ini.in'
--- demo/scopes/scope-A/scope-A.ini.in	2014-02-27 15:50:32 +0000
+++ demo/scopes/scope-A/scope-A.ini.in	2014-03-28 15:41:24 +0000
@@ -2,7 +2,7 @@
 DisplayName = scope-A.DisplayName
 Description = scope-A.Description
 Art = scope-A.Art
-Author = Canonical
+Author = Canonical Ltd.
 Icon = scope-A.Icon
 SearchHint = scope-A.SearchHint
 HotKey = scope-A.HotKey

=== modified file 'demo/scopes/scope-B/scope-B.ini.in'
--- demo/scopes/scope-B/scope-B.ini.in	2014-02-27 15:50:32 +0000
+++ demo/scopes/scope-B/scope-B.ini.in	2014-03-28 15:41:24 +0000
@@ -1,4 +1,4 @@
 [ScopeConfig]
 DisplayName = scope-B.DisplayName
 Description = scope-B.Description
-Author = Canonical
+Author = Canonical Ltd.

=== modified file 'demo/scopes/scope-C/scope-C.ini.in'
--- demo/scopes/scope-C/scope-C.ini.in	2014-02-27 15:50:32 +0000
+++ demo/scopes/scope-C/scope-C.ini.in	2014-03-28 15:41:24 +0000
@@ -1,4 +1,4 @@
 [ScopeConfig]
 DisplayName = scope-C.DisplayName
 Description = scope-C.Description
-Author = Canonical
+Author = Canonical Ltd.

=== modified file 'demo/scopes/scope-D/scope-D.ini.in'
--- demo/scopes/scope-D/scope-D.ini.in	2014-02-27 15:50:32 +0000
+++ demo/scopes/scope-D/scope-D.ini.in	2014-03-28 15:41:24 +0000
@@ -1,4 +1,4 @@
 [ScopeConfig]
 DisplayName = scope-D.DisplayName
 Description = scope-D.Description
-Author = Canonical
+Author = Canonical Ltd.

=== modified file 'demo/scopes/scope-N/scope-N.ini.in'
--- demo/scopes/scope-N/scope-N.ini.in	2014-02-27 15:50:32 +0000
+++ demo/scopes/scope-N/scope-N.ini.in	2014-03-28 15:41:24 +0000
@@ -1,4 +1,4 @@
 [ScopeConfig]
 DisplayName = scope-N.DisplayName
 Description = scope-N.Description
-Author = Canonical
+Author = Canonical Ltd.

=== modified file 'demo/scopes/scope-S/scope-S.ini.in'
--- demo/scopes/scope-S/scope-S.ini.in	2014-02-27 15:50:32 +0000
+++ demo/scopes/scope-S/scope-S.ini.in	2014-03-28 15:41:24 +0000
@@ -1,4 +1,4 @@
 [ScopeConfig]
 DisplayName = scope-S.DisplayName
 Description = scope-S.Description
-Author = Canonical
+Author = Canonical Ltd.

=== modified file 'doc/tutorial.dox'
--- doc/tutorial.dox	2014-03-20 17:26:28 +0000
+++ doc/tutorial.dox	2014-03-28 15:41:24 +0000
@@ -33,7 +33,7 @@
 be written with the intention of running remotely on the Smart Scopes Server.</b>
 
 (For more information on how to deploy your scope to the Smart Scopes Server, or how to implement a native remote scope using
-the SSS REST API see: <link_not_yet_available>)
+the SSS REST API see: <i>link_not_yet_available</i>)
 
 \section develop Developing scopes
 

=== modified file 'include/unity/scopes/internal/RegistryObject.h'
--- include/unity/scopes/internal/RegistryObject.h	2014-03-07 04:19:32 +0000
+++ include/unity/scopes/internal/RegistryObject.h	2014-03-28 15:41:24 +0000
@@ -13,16 +13,20 @@
  * You should have received a copy of the GNU Lesser General Public License
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  *
- * Authored by: Michi Henning <[e-mail]>
+ * Authored by: Marcus Tomlinson <[e-mail]>
  */
 
 #ifndef UNITY_SCOPES_INTERNAL_REGISTRYOBJECT_H
 #define UNITY_SCOPES_INTERNAL_REGISTRYOBJECT_H
 
+#include <core/posix/child_process.h>
+
 #include <unity/scopes/internal/MWRegistryProxyFwd.h>
 #include <unity/scopes/internal/RegistryObjectBase.h>
 
+#include <condition_variable>
 #include <mutex>
+#include <thread>
 
 namespace unity
 {
@@ -33,40 +37,84 @@
 namespace internal
 {
 
-class RegistryObjectPrivate;
-
-// Maintains a map of <scope name, scope proxy> pairs.
-
 class RegistryObject : public RegistryObjectBase
 {
 public:
+    struct ScopeExecData
+    {
+        ScopeExecData() = default;
+        ScopeExecData(std::initializer_list<std::string>) = delete;
+        std::string scope_id;
+        std::string scoperunner_path;
+        std::string runtime_config;
+        std::string scope_config;
+    };
+
+public:
     UNITY_DEFINES_PTRS(RegistryObject);
 
-    RegistryObject();
+    RegistryObject(core::posix::ChildProcess::DeathObserver& death_observer);
     virtual ~RegistryObject();
 
     // Remote operation implementations
-    virtual ScopeMetadata get_metadata(std::string const& scope_id) override;
-    virtual MetadataMap list() override;
+    virtual ScopeMetadata get_metadata(std::string const& scope_id) const override;
+    virtual MetadataMap list() const override;
     virtual ScopeProxy locate(std::string const& scope_id) override;
 
     // Local methods
     bool add_local_scope(std::string const& scope_id, ScopeMetadata const& scope,
-                         std::vector<std::string> const& spawn_command);
+                         ScopeExecData const& scope_exec_data);
     bool remove_local_scope(std::string const& scope_id);
     void set_remote_registry(MWRegistryProxy const& remote_registry);
 
-private:
-    void spawn_scope(std::string const& scope_id);
-    void shutdown();
-    static int kill_process(pid_t pid);
-    static bool is_dead(pid_t pid);
-
-private:
+    bool is_scope_running( std::string const& scope_id );
+
+private:
+    void on_process_death(core::posix::Process const& process);
+
+    class ScopeProcess
+    {
+    public:
+        enum ProcessState
+        {
+            Stopped, Starting, Running, Stopping
+        };
+
+        ScopeProcess(ScopeExecData exec_data);
+        ScopeProcess(ScopeProcess const& other);
+        ~ScopeProcess();
+
+        ProcessState state() const;
+        bool wait_for_state(ProcessState state, int timeout_ms) const;
+
+        void exec(core::posix::ChildProcess::DeathObserver& death_observer);
+        void kill();
+
+        bool on_process_death(pid_t pid);
+
+    private:
+        // the following methods must be called with process_mutex_ locked
+        void clear_handle_unlocked();
+        void update_state_unlocked(ProcessState state);
+
+        bool wait_for_state(std::unique_lock<std::mutex>& lock,
+                            ProcessState state, int timeout_ms) const;
+        void kill(std::unique_lock<std::mutex>& lock);
+
+    private:
+        const ScopeExecData exec_data_;
+        ProcessState state_ = Stopped;
+        mutable std::mutex process_mutex_;
+        mutable std::condition_variable state_change_cond_;
+        core::posix::ChildProcess process_ = core::posix::ChildProcess::invalid();
+    };
+
+private:
+    core::posix::ChildProcess::DeathObserver& death_observer_;
+    core::ScopedConnection death_observer_connection_;
     mutable std::mutex mutex_;
     MetadataMap scopes_;
-    std::map<std::string, pid_t> scope_processes_;
-    std::map<std::string, std::vector<std::string>> commands_;
+    std::map<std::string, ScopeProcess> scope_processes_;
     MWRegistryProxy remote_registry_;
 };
 

=== modified file 'include/unity/scopes/internal/RegistryObjectBase.h'
--- include/unity/scopes/internal/RegistryObjectBase.h	2014-03-07 04:19:32 +0000
+++ include/unity/scopes/internal/RegistryObjectBase.h	2014-03-28 15:41:24 +0000
@@ -36,8 +36,8 @@
 public:
     UNITY_DEFINES_PTRS(RegistryObjectBase);
 
-    virtual ScopeMetadata get_metadata(std::string const& scope_id) = 0;
-    virtual MetadataMap list() = 0;
+    virtual ScopeMetadata get_metadata(std::string const& scope_id) const = 0;
+    virtual MetadataMap list() const = 0;
     virtual ScopeProxy locate(std::string const& scope_id) = 0;
 };
 

=== modified file 'include/unity/scopes/internal/RuntimeImpl.h'
--- include/unity/scopes/internal/RuntimeImpl.h	2014-03-07 04:19:32 +0000
+++ include/unity/scopes/internal/RuntimeImpl.h	2014-03-28 15:41:24 +0000
@@ -45,6 +45,7 @@
     void destroy();
 
     std::string scope_id() const;
+    std::string configfile() const;
     MiddlewareFactory const* factory() const;
     RegistryProxy registry() const;
     std::string registry_configfile() const;
@@ -64,6 +65,7 @@
 
     std::atomic_bool destroyed_;
     std::string scope_id_;
+    std::string configfile_;
     MiddlewareFactory::UPtr middleware_factory_;
     MiddlewareBase::SPtr middleware_;
     mutable RegistryProxy registry_;

=== modified file 'include/unity/scopes/internal/smartscopes/SSRegistryObject.h'
--- include/unity/scopes/internal/smartscopes/SSRegistryObject.h	2014-03-20 16:19:04 +0000
+++ include/unity/scopes/internal/smartscopes/SSRegistryObject.h	2014-03-28 15:41:24 +0000
@@ -43,12 +43,12 @@
     UNITY_DEFINES_PTRS(SSRegistryObject);
 
     SSRegistryObject(MiddlewareBase::SPtr middleware, std::string const& ss_scope_endpoint,
-                     uint no_reply_timeout, uint refresh_rate_in_sec, std::string const& sss_url = "",
+                     uint http_reply_timeout, uint refresh_rate_in_sec, std::string const& sss_url = "",
                      bool caching_enabled = true);
     virtual ~SSRegistryObject() noexcept;
 
-    ScopeMetadata get_metadata(std::string const& scope_id) override;
-    MetadataMap list() override;
+    ScopeMetadata get_metadata(std::string const& scope_id) const override;
+    MetadataMap list() const override;
 
     ScopeProxy locate(std::string const& scope_id) override;
 

=== modified file 'scoperegistry/CMakeLists.txt'
--- scoperegistry/CMakeLists.txt	2013-12-17 14:16:19 +0000
+++ scoperegistry/CMakeLists.txt	2014-03-28 15:41:24 +0000
@@ -1,7 +1,6 @@
 set(SRC
         FindFiles.cpp
         scoperegistry.cpp
-        SignalThread.cpp
         ScopeSet.cpp
 )
 

=== removed file 'scoperegistry/SignalThread.cpp'
--- scoperegistry/SignalThread.cpp	2014-02-12 06:10:04 +0000
+++ scoperegistry/SignalThread.cpp	1970-01-01 00:00:00 +0000
@@ -1,121 +0,0 @@
-/*
- * Copyright (C) 2013 Canonical Ltd
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Lesser General Public License version 3 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- *
- * Authored by: Michi Henning <[e-mail]>
- */
-
-#include <SignalThread.h>
-
-#include <unity/UnityExceptions.h>
-
-#include <cassert>
-#include <iostream>
-
-#include <signal.h>
-#include <string.h>
-#include <unistd.h>
-
-using namespace std;
-using namespace unity;
-
-namespace scoperegistry
-{
-
-SignalThread::SignalThread() :
-    done_(false)
-{
-    // Block signals in the caller, so they are delivered to the thread we are about to create.
-    sigemptyset(&sigs_);
-    sigaddset(&sigs_, SIGINT);
-    sigaddset(&sigs_, SIGHUP);
-    sigaddset(&sigs_, SIGTERM);
-    sigaddset(&sigs_, SIGUSR1);
-    int err = pthread_sigmask(SIG_BLOCK, &sigs_, nullptr);
-    if (err != 0)
-    {
-        throw SyscallException("pthread_sigmask failed", err);
-    }
-
-    // Make ourselves a process group leader.
-    setpgid(0, 0);
-
-    // Run a signal handling thread that waits for any of the above signals.
-    lock_guard<mutex> lock(mutex_);
-    handler_thread_ = thread([this]{ this->wait_for_sigs(); });
-}
-
-SignalThread::~SignalThread()
-{
-    stop();
-    handler_thread_.join();
-}
-
-void SignalThread::activate(function<void()> callback)
-{
-    lock_guard<mutex> lock(mutex_);
-    callback_ = callback;
-}
-
-void SignalThread::stop()
-{
-    lock_guard<mutex> lock(mutex_);
-    if (!done_)
-    {
-        done_ = true;
-        kill(getpid(), SIGUSR1);
-    }
-}
-
-// Wait for termination signals. When a termination signal arrives, we
-// invoke the callback (if set). If we receive SIGUSR1 (because
-// we are terminating ourself, we exit the thread.
-
-void SignalThread::wait_for_sigs()
-{
-    int signo;
-    for (;;)
-    {
-        int err = sigwait(&sigs_, &signo);
-        if (err != 0)
-        {
-            cerr << "scoperegistry: sigwait failed: " << strerror(err) << endl;
-            _exit(1);
-        }
-        switch (signo)
-        {
-            case SIGINT:
-            case SIGHUP:
-            case SIGTERM:
-            {
-                lock_guard<mutex> lock(mutex_);
-                if (callback_)
-                {
-                    callback_();
-                }
-                break;
-            }
-            case SIGUSR1:
-            {
-                return;
-            }
-            default:
-            {
-                assert(false);
-            }
-        }
-    }
-}
-
-} // namespace scoperegistry

=== removed file 'scoperegistry/SignalThread.h'
--- scoperegistry/SignalThread.h	2014-02-12 06:10:04 +0000
+++ scoperegistry/SignalThread.h	1970-01-01 00:00:00 +0000
@@ -1,55 +0,0 @@
-/*
- * Copyright (C) 2013 Canonical Ltd
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Lesser General Public License version 3 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- *
- * Authored by: Michi Henning <[e-mail]>
- */
-
-#ifndef SCOPEREGISTRY_SIGNAL_THREAD_H
-#define SCOPEREGISTRY_SIGNAL_THREAD_H
-
-#include <unity/util/NonCopyable.h>
-
-#include <mutex>
-#include <thread>
-
-#include <sys/types.h>
-
-namespace scoperegistry
-{
-
-class SignalThread final
-{
-public:
-    NONCOPYABLE(SignalThread);
-
-    SignalThread();
-    ~SignalThread();
-
-    void activate(std::function<void()> callback);
-    void stop();
-
-private:
-    void wait_for_sigs();
-
-    std::mutex mutex_;
-    std::function<void()> callback_;
-    bool done_;
-    std::thread handler_thread_;
-    sigset_t sigs_;
-};
-
-} // namespace scoperegistry
-
-#endif

=== modified file 'scoperegistry/scoperegistry.cpp'
--- scoperegistry/scoperegistry.cpp	2014-03-13 12:33:32 +0000
+++ scoperegistry/scoperegistry.cpp	2014-03-28 15:41:24 +0000
@@ -17,7 +17,6 @@
  */
 
 #include "FindFiles.h"
-#include "SignalThread.h"
 
 #include <unity/scopes/internal/MiddlewareFactory.h>
 #include <unity/scopes/internal/MWRegistry.h>
@@ -74,7 +73,7 @@
 
 // Return a map of <scope, config_file> pairs for all scopes (Canonical and OEM scopes).
 // If a Canonical scope is overrideable and the OEM has configured a scope with the
-// same name, the OEM scope overrides the Canonical one.
+// same id, the OEM scope overrides the Canonical one.
 
 map<string, string> find_local_scopes(string const& scope_installdir, string const& oem_installdir)
 {
@@ -90,22 +89,22 @@
     for (auto&& path : config_files)
     {
         string file_name = basename(const_cast<char*>(string(path).c_str()));    // basename() modifies its argument
-        string scope_name = strip_suffix(file_name, ".ini");
+        string scope_id = strip_suffix(file_name, ".ini");
         try
         {
             ScopeConfig config(path);
             if (config.overrideable())
             {
-                overrideable_scopes[scope_name] = path;
+                overrideable_scopes[scope_id] = path;
             }
             else
             {
-                fixed_scopes[scope_name] = path;
+                fixed_scopes[scope_id] = path;
             }
         }
         catch (unity::Exception const& e)
         {
-            error("ignoring scope \"" + scope_name + "\": configuration error:\n" + e.what());
+            error("ignoring scope \"" + scope_id + "\": configuration error:\n" + e.what());
         }
     }
 
@@ -117,10 +116,10 @@
             for (auto&& path : oem_paths)
             {
                 string file_name = basename(const_cast<char*>(string(path).c_str()));    // basename() modifies its argument
-                string scope_name = strip_suffix(file_name, ".ini");
-                if (fixed_scopes.find(scope_name) == fixed_scopes.end())
+                string scope_id = strip_suffix(file_name, ".ini");
+                if (fixed_scopes.find(scope_id) == fixed_scopes.end())
                 {
-                    overrideable_scopes[scope_name] = path;  // Replaces scope if it was present already
+                    overrideable_scopes[scope_id] = path;  // Replaces scope if it was present already
                 }
                 else
                 {
@@ -189,14 +188,18 @@
             catch (NotFoundException const&)
             {
             }
+
             ScopeProxy proxy = ScopeImpl::create(mw->create_scope_proxy(pair.first), mw->runtime(), pair.first);
             mi->set_proxy(proxy);
             auto meta = ScopeMetadataImpl::create(move(mi));
-            vector<string> spawn_command;
-            spawn_command.push_back(scoperunner_path);
-            spawn_command.push_back(config_file);
-            spawn_command.push_back(pair.second);
-            registry->add_local_scope(pair.first, move(meta), spawn_command);
+
+            RegistryObject::ScopeExecData exec_data;
+            exec_data.scope_id = pair.first;
+            exec_data.scoperunner_path = scoperunner_path;
+            exec_data.runtime_config = config_file;
+            exec_data.scope_config = pair.second;
+
+            registry->add_local_scope(pair.first, move(meta), exec_data);
         }
         catch (unity::Exception const& e)
         {
@@ -244,10 +247,33 @@
     char const* const config_file = argc > 1 ? argv[1] : "";
     int exit_status = 1;
 
-    SignalThread signal_thread;
-
     try
     {
+        // We shutdown the runtime whenever these signals happen.
+        auto termination_trap = core::posix::trap_signals_for_all_subsequent_threads(
+        {
+            core::posix::Signal::sig_int,
+            core::posix::Signal::sig_hup,
+            core::posix::Signal::sig_term
+        });
+
+        // And we maintain our list of processes with the help of sig_chld.
+        auto child_trap = core::posix::trap_signals_for_all_subsequent_threads(
+        {
+            core::posix::Signal::sig_chld
+        });
+
+        // The death observer is required to make sure that we reap all child processes
+        // whenever multiple sigchld's are compressed together.
+        auto death_observer =
+                core::posix::ChildProcess::DeathObserver::create_once_with_signal_trap(
+                    child_trap);
+
+        // Starting up both traps.
+        std::thread termination_trap_worker([termination_trap]() { termination_trap->run(); });
+        std::thread child_trap_worker([child_trap]() { child_trap->run(); });
+
+        // And finally creating our runtime.
         RuntimeImpl::UPtr runtime = RuntimeImpl::create("Registry", config_file);
 
         string identity = runtime->registry_identity();
@@ -278,10 +304,22 @@
 
         // Inform the signal thread that it should shutdown the middleware
         // if we get a termination signal.
-        signal_thread.activate([middleware]{ middleware->stop(); });
+        termination_trap->signal_raised().connect([middleware](core::posix::Signal signal)
+        {
+            switch(signal)
+            {
+            case core::posix::Signal::sig_int:
+            case core::posix::Signal::sig_hup:
+            case core::posix::Signal::sig_term:
+                middleware->stop();
+                break;
+            default:
+                break;
+            }
+        });
 
         // The registry object stores the local and remote scopes
-        RegistryObject::SPtr registry(new RegistryObject);
+        RegistryObject::SPtr registry(new RegistryObject(*death_observer));
 
         // Add the metadata for each scope to the lookup table.
         // We do this before starting any of the scopes, so aggregating scopes don't get a lookup failure if
@@ -295,11 +333,11 @@
         for (auto i = 2; i < argc; ++i)
         {
             string file_name = basename(const_cast<char*>(string(argv[i]).c_str()));  // basename() modifies its argument
-            string scope_name = strip_suffix(file_name, ".ini");
-            local_scopes[scope_name] = argv[i];                   // operator[] overwrites pre-existing entries
+            string scope_id = strip_suffix(file_name, ".ini");
+            local_scopes[scope_id] = argv[i];                   // operator[] overwrites pre-existing entries
         }
 
-        add_local_scopes(registry, local_scopes, middleware, scoperunner_path, config_file);
+        add_local_scopes(registry, local_scopes, middleware, scoperunner_path, runtime->configfile());
         if (ss_reg_id.empty() || ss_reg_endpoint.empty())
         {
             error("no remote registry configured, only local scopes will be available");
@@ -342,6 +380,19 @@
 
         // Wait until we are done, which happens if we receive a termination signal.
         middleware->wait_for_shutdown();
+
+        // Stop termination_trap
+        termination_trap->stop();
+        if (termination_trap_worker.joinable())
+            termination_trap_worker.join();
+
+        // Please note that the child_trap must only be stopped once the
+        // termination_trap thread has been joined. We otherwise will encounter
+        // a race between the middleware shutting down and not receiving sigchld anymore.
+        child_trap->stop();
+        if (child_trap_worker.joinable())
+            child_trap_worker.join();
+
         exit_status = 0;
     }
     catch (std::exception const& e)

=== modified file 'scoperunner/scoperunner.cpp'
--- scoperunner/scoperunner.cpp	2014-01-24 03:18:25 +0000
+++ scoperunner/scoperunner.cpp	2014-03-28 15:41:24 +0000
@@ -24,6 +24,8 @@
 #include <unity/scopes/ScopeExceptions.h>
 #include <unity/UnityExceptions.h>
 
+#include <core/posix/signal.h>
+
 #include <cassert>
 #include <future>
 #include <iostream>
@@ -92,7 +94,8 @@
 
 // Scope thread start function.
 
-void scope_thread(string const& runtime_config,
+void scope_thread(std::shared_ptr<core::posix::SignalTrap> trap,
+                  string const& runtime_config,
                   string const& scope_name,
                   string const& lib_dir,
                   promise<void> finished_promise)
@@ -114,6 +117,12 @@
         auto scope = unique_ptr<ScopeObject>(new ScopeObject(rt.get(), loader->scope_base()));
         auto proxy = mw->add_scope_object(scope_name, move(scope));
 
+        trap->signal_raised().connect([loader, mw](core::posix::Signal)
+        {
+            loader->stop();
+            mw->stop();
+        });
+
         mw->wait_for_shutdown();
 
         // Collect exit status from the run thread. If this throws, the ScopeLoader
@@ -135,6 +144,14 @@
 
 int run_scopes(string const& runtime_config, vector<string> config_files)
 {
+    auto trap = core::posix::trap_signals_for_all_subsequent_threads(
+    {
+        core::posix::Signal::sig_hup,
+        core::posix::Signal::sig_term
+    });
+
+    std::thread trap_worker([trap]() { trap->run(); });
+
     for (auto file : config_files)
     {
         string file_name = basename(const_cast<char*>(string(file).c_str()));    // basename() modifies its argument
@@ -152,7 +169,7 @@
         // We collect exit status from the thread via the future from each promise.
         promise<void> p;
         auto f = p.get_future();
-        thread t(scope_thread, runtime_config, scope_name, dir, move(p));
+        thread t(scope_thread, trap, runtime_config, scope_name, dir, move(p));
         auto id = t.get_id();
         threads[id] = ThreadFuture { move(t), move(f) };
     }
@@ -180,6 +197,12 @@
             ++num_errors;
         }
     }
+
+    trap->stop();
+
+    if (trap_worker.joinable())
+        trap_worker.join();
+
     return num_errors;
 }
 
@@ -188,6 +211,13 @@
 int
 main(int argc, char* argv[])
 {
+    // sig masks are inherited by child processes when forked.
+    // we do not want to inherit our parent's (scoperegistry)
+    // sig mask, hence we clear it immediately on entry.
+    sigset_t set;
+    ::sigemptyset(&set);
+    ::pthread_sigmask(SIG_SETMASK, &set, nullptr);
+
     prog_name = basename(argv[0]);
     if (argc < 3)
     {

=== modified file 'smartscopesproxy/smartscopesproxy.cpp'
--- smartscopesproxy/smartscopesproxy.cpp	2014-02-20 19:19:25 +0000
+++ smartscopesproxy/smartscopesproxy.cpp	2014-03-28 15:41:24 +0000
@@ -23,6 +23,8 @@
 
 #include <unity/scopes/internal/DfltConfig.h>
 
+#include <core/posix/signal.h>
+
 #include <cassert>
 #include <signal.h>
 #include <libgen.h>
@@ -74,10 +76,17 @@
 
     try
     {
+        auto trap = core::posix::trap_signals_for_all_subsequent_threads(
+        {
+            core::posix::Signal::sig_int,
+            core::posix::Signal::sig_hup,
+            core::posix::Signal::sig_term
+        });
+
         ///! TODO: get these from config
         std::string ss_reg_id = "SSRegistry";
         std::string ss_scope_id = "SmartScope";
-        uint const no_reply_timeout = 20000;
+        uint const http_reply_timeout = 20000;
         uint const ss_reg_refresh_rate = 60 * 60 * 24; // 24 hour refresh (in seconds)
 
         // Instantiate SS registry and scopes runtimes
@@ -93,9 +102,17 @@
         MiddlewareBase::SPtr reg_mw = reg_rt->factory()->find(ss_reg_id, mw_kind);
         MiddlewareBase::SPtr scope_mw = scope_rt->factory()->create(ss_scope_id, mw_kind, mw_configfile);
 
+        trap->signal_raised().connect([reg_mw, scope_mw](core::posix::Signal)
+        {
+            scope_mw->stop();
+            reg_mw->stop();
+        });
+
+        std::thread trap_worker([trap]() { trap->run(); });
+
         // Instantiate a SS registry object
         SSRegistryObject::SPtr reg(new SSRegistryObject(reg_mw, scope_mw->get_scope_endpoint(),
-                                                        no_reply_timeout, ss_reg_refresh_rate));
+                                                        http_reply_timeout, ss_reg_refresh_rate));
 
         // Instantiate a SS scope object
         SSScopeObject::UPtr scope(new SSScopeObject(ss_scope_id, scope_mw, reg));
@@ -114,6 +131,11 @@
         scope_mw->wait_for_shutdown();
         reg_mw->wait_for_shutdown();
 
+        trap->stop();
+
+        if (trap_worker.joinable())
+            trap_worker.join();
+
         exit_status = 0;
     }
     catch (std::exception const& e)

=== modified file 'src/scopes/ScopeMetadata.cpp'
--- src/scopes/ScopeMetadata.cpp	2014-03-13 12:33:32 +0000
+++ src/scopes/ScopeMetadata.cpp	2014-03-28 15:41:24 +0000
@@ -42,6 +42,7 @@
     assert(p->proxy());
     assert(!p->display_name().empty());
     assert(!p->description().empty());
+    assert(!p->author().empty());
 }
 
 ScopeMetadata::ScopeMetadata(ScopeMetadata const& other) :

=== modified file 'src/scopes/internal/RegistryObject.cpp'
--- src/scopes/internal/RegistryObject.cpp	2014-03-07 12:31:25 +0000
+++ src/scopes/internal/RegistryObject.cpp	2014-03-28 15:41:24 +0000
@@ -13,20 +13,18 @@
  * You should have received a copy of the GNU Lesser General Public License
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  *
- * Authored by: Michi Henning <[e-mail]>
+ * Authored by: Marcus Tomlinson <[e-mail]>
  */
 
 #include <unity/scopes/internal/RegistryObject.h>
 
+#include <core/posix/child_process.h>
+#include <core/posix/exec.h>
+
 #include <unity/scopes/internal/MWRegistry.h>
 #include <unity/scopes/ScopeExceptions.h>
 #include <unity/UnityExceptions.h>
 
-#include <signal.h>
-#include <cassert>
-#include <sys/wait.h>
-#include <unistd.h>
-
 using namespace std;
 
 namespace unity
@@ -38,43 +36,58 @@
 namespace internal
 {
 
-RegistryObject::RegistryObject()
+///! TODO: get from config
+static const int c_process_wait_timeout = 1000;
+
+RegistryObject::RegistryObject(core::posix::ChildProcess::DeathObserver& death_observer)
+    : death_observer_(death_observer),
+      death_observer_connection_
+      {
+          death_observer_.child_died().connect([this](const core::posix::ChildProcess& cp)
+          {
+              on_process_death(cp);
+          })
+      }
+
 {
 }
 
 RegistryObject::~RegistryObject()
 {
-    try
-    {
-        shutdown();
-    }
-    catch (std::exception const& e)
-    {
-        fprintf(stderr, "scoperegistry: shutdown error: %s\n", e.what());
-    }
-    catch (...)
-    {
-        fprintf(stderr, "scoperegistry: unknown exception during shutdown\n");
-    }
+    // kill all scope processes
+    for (auto& scope_process : scope_processes_)
+    {
+        try
+        {
+            scope_process.second.kill();
+        }
+        catch(std::exception const& e)
+        {
+            cerr << "RegistryObject::~RegistryObject(): " << e.what() << endl;
+        }
+    }
+
+    // clear scope maps
+    scopes_.clear();
+    scope_processes_.clear();
 }
 
-ScopeMetadata RegistryObject::get_metadata(std::string const& scope_id)
+ScopeMetadata RegistryObject::get_metadata(std::string const& scope_id) const
 {
     lock_guard<decltype(mutex_)> lock(mutex_);
     // If the id is empty, it was sent as empty by the remote client.
     if (scope_id.empty())
     {
-        throw unity::InvalidArgumentException("RegistryObject::get_metadata(): Cannot search for scope with empty name");
+        throw unity::InvalidArgumentException("RegistryObject::get_metadata(): Cannot search for scope with empty id");
     }
 
     // Look for the scope in both the local and the remote map.
     // Local scopes take precedence over remote ones of the same
-    // name. (Ideally, this will never happen, except maybe
-    // during development.)
-    auto const& it = scopes_.find(scope_id);
-    if (it != scopes_.end())
+    // id. (Ideally, this should never happen.)
+    auto const& scope_it = scopes_.find(scope_id);
+    if (scope_it != scopes_.end())
     {
-        return it->second;
+        return scope_it->second;
     }
 
     if (remote_registry_)
@@ -85,14 +98,14 @@
     throw NotFoundException("RegistryObject::get_metadata(): no such scope: ",  scope_id);
 }
 
-MetadataMap RegistryObject::list()
+MetadataMap RegistryObject::list() const
 {
     lock_guard<decltype(mutex_)> lock(mutex_);
     MetadataMap all_scopes(scopes_);  // Local scopes
 
-    // If a remote scope has the same name as a local one,
+    // If a remote scope has the same id as a local one,
     // this will not overwrite a local scope with a remote
-    // one if they have the same name.
+    // one if they have the same id.
     if (remote_registry_)
     {
         MetadataMap remote_scopes = remote_registry_->list();
@@ -102,34 +115,58 @@
     return all_scopes;
 }
 
+ScopeProxy RegistryObject::locate(std::string const& scope_id)
+{
+    decltype(scopes_.cbegin()) scope_it;
+    decltype(scope_processes_.begin()) proc_it;
+
+    {
+        lock_guard<decltype(mutex_)> lock(mutex_);
+        // If the id is empty, it was sent as empty by the remote client.
+        if (scope_id.empty())
+        {
+            throw unity::InvalidArgumentException("Registry::locate(): Cannot locate scope with empty id");
+        }
+
+        scope_it = scopes_.find(scope_id);
+        if (scope_it == scopes_.end())
+        {
+            throw NotFoundException("Registry::locate(): Tried to locate unknown local scope", scope_id);
+        }
+
+        proc_it = scope_processes_.find(scope_id);
+        if (proc_it == scope_processes_.end())
+        {
+            throw NotFoundException("Registry::locate(): Tried to exec unknown local scope", scope_id);
+        }
+    }
+
+    proc_it->second.exec(death_observer_);
+    return scope_it->second.proxy();
+}
+
 bool RegistryObject::add_local_scope(std::string const& scope_id, ScopeMetadata const& metadata,
-                                     std::vector<std::string> const& spawn_command)
+                                     ScopeExecData const& exec_data)
 {
     lock_guard<decltype(mutex_)> lock(mutex_);
     bool return_value = true;
     if (scope_id.empty())
     {
-        throw unity::InvalidArgumentException("RegistryObject::add_local_scope(): Cannot add scope with empty name");
+        throw unity::InvalidArgumentException("RegistryObject::add_local_scope(): Cannot add scope with empty id");
     }
-    if(scope_id.find('/') != std::string::npos) {
-        throw unity::InvalidArgumentException("RegistryObject::add_local_scope(): Cannot create a scope with "
-                                              "a slash in its name: " + scope_id);
+    if (scope_id.find('/') != std::string::npos)
+    {
+        throw unity::InvalidArgumentException("RegistryObject::add_local_scope(): Cannot create a scope with '/' in its id");
     }
 
     if (scopes_.find(scope_id) != scopes_.end())
     {
-        auto proc = scope_processes_.find(scope_id);
-        if (proc != scope_processes_.end())
-        {
-            kill_process(proc->second);
-            scope_processes_.erase(scope_id);
-        }
         scopes_.erase(scope_id);
-        commands_.erase(scope_id);
+        scope_processes_.erase(scope_id);
         return_value = false;
     }
     scopes_.insert(make_pair(scope_id, metadata));
-    commands_[scope_id] = spawn_command;
+    scope_processes_.insert(make_pair(scope_id, ScopeProcess(exec_data)));
     return return_value;
 }
 
@@ -140,10 +177,10 @@
     if (scope_id.empty())
     {
         throw unity::InvalidArgumentException("RegistryObject::remove_local_scope(): Cannot remove scope "
-                                              "with empty name");
+                                              "with empty id");
     }
 
-    commands_.erase(scope_id);
+    scope_processes_.erase(scope_id);
     return scopes_.erase(scope_id) == 1;
 }
 
@@ -153,96 +190,218 @@
     remote_registry_ = remote_registry;
 }
 
-ScopeProxy RegistryObject::locate(std::string const& scope_id)
-{
-    lock_guard<decltype(mutex_)> lock(mutex_);
-    // If the id is empty, it was sent as empty by the remote client.
-    if (scope_id.empty())
-        throw unity::InvalidArgumentException("RegistryObject::locate(): Cannot locate scope with empty name");
-    auto metadata = scopes_.find(scope_id);
-    if (metadata == scopes_.end())
-    {
-        throw NotFoundException("RegistryObject::locate(): Tried to obtain unknown scope: ", scope_id);
-    }
-    auto search = scope_processes_.find(scope_id);
-    if (search == scope_processes_.end() || is_dead(search->second))
-    {
-        spawn_scope(scope_id);
-    }
-    return metadata->second.proxy();
-}
-
-void RegistryObject::spawn_scope(std::string const& scope_id)
-{
-    if (scopes_.find(scope_id) == scopes_.end())
-    {
-        throw NotFoundException("RegistryObject::spawn_scope(): Tried to spawn an unknown scope: ", scope_id);
-    }
-    auto process = scope_processes_.find(scope_id);
-    if (process != scope_processes_.end())
-    {
-        assert(is_dead(process->second));
-        int status;
-        waitpid(process->second, &status, 0);
-        if (status != 0)
-        {
-            printf("scope %s has exited with nonzero error status %d.\n", scope_id.c_str(), status);
-        }
-        scope_processes_.erase(scope_id);
-    }
-
-    pid_t pid;
-    switch (pid = fork())
-    {
-        case -1:
-        {
-            throw SyscallException("RegistryObject::spawn_scope(): cannot fork", errno);
-        }
-        case 0: // child
-        {
-            const vector<string>& cmd = commands_[scope_id];
-            assert(cmd.size() == 3);
-            // Includes room for final NULL element.
-            unique_ptr<char const* []> argv(new char const*[4]);
-            argv[0] = cmd[0].c_str();
-            argv[1] = cmd[1].c_str();
-            argv[2] = cmd[2].c_str();
-            argv[3] = nullptr;
-            execv(argv[0], const_cast<char* const*>(argv.get()));
-            throw SyscallException("REgistryObject::spawn_scope(): cannot exec scoperunner", errno);
-        }
-    }
-    const vector<string>& cmd = commands_[scope_id];
-    printf("spawning scope %s to process number %d with command line %s %s %s.\n",
-           scope_id.c_str(), (int)pid, cmd[0].c_str(), cmd[1].c_str(), cmd[2].c_str());
-    scope_processes_[scope_id] = pid;
-}
-
-void RegistryObject::shutdown()
-{
-    for (const auto &i : scope_processes_)
-    {
-        kill_process(i.second);
-        // If and when we move to graceful shutdown, check that exit status
-        // was zero and print error message here.
-    }
-    scope_processes_.clear();
-    commands_.clear();
-}
-
-int RegistryObject::kill_process(pid_t pid) {
-    int exitcode;
-    // Currently just shoot children dead.
-    // If we want to get fancy and give them a graceful
-    // warning, this is the place to do it.
-    kill(pid, SIGKILL);
-    waitpid(pid, &exitcode, 0);
-    return exitcode;
-}
-
-bool RegistryObject::is_dead(pid_t pid)
-{
-    return kill(pid, 0) < 0 && errno == ESRCH;
+bool RegistryObject::is_scope_running( std::string const& scope_id )
+{
+    auto it = scope_processes_.find( scope_id );
+    if (it != scope_processes_.end())
+    {
+        return it->second.state() != ScopeProcess::ProcessState::Stopped;
+    }
+
+    throw NotFoundException("RegistryObject::is_scope_process_running(): no such scope: ",  scope_id);
+}
+
+void RegistryObject::on_process_death(core::posix::Process const& process)
+{
+    // the death observer has signaled that a child has died.
+    // broadcast this message to each scope process until we have found the process in question.
+    // (this is slightly more efficient than just connecting the signal to every scope process)
+    pid_t pid = process.pid();
+    for (auto& scope_process : scope_processes_)
+    {
+        if (scope_process.second.on_process_death(pid))
+            break;
+    }
+}
+
+RegistryObject::ScopeProcess::ScopeProcess(ScopeExecData exec_data)
+    : exec_data_(exec_data)
+{
+}
+
+RegistryObject::ScopeProcess::ScopeProcess(ScopeProcess const& other)
+    : exec_data_(other.exec_data_)
+{
+}
+
+RegistryObject::ScopeProcess::~ScopeProcess()
+{
+    try
+    {
+        kill();
+    }
+    catch(std::exception const& e)
+    {
+        cerr << "RegistryObject::ScopeProcess::~ScopeProcess(): " << e.what() << endl;
+    }
+}
+
+RegistryObject::ScopeProcess::ProcessState RegistryObject::ScopeProcess::state() const
+{
+    std::lock_guard<std::mutex> lock(process_mutex_);
+    return state_;
+}
+
+bool RegistryObject::ScopeProcess::wait_for_state(ProcessState state, int timeout_ms) const
+{
+    std::unique_lock<std::mutex> lock(process_mutex_);
+    return wait_for_state(lock, state, timeout_ms);
+}
+
+void RegistryObject::ScopeProcess::exec(core::posix::ChildProcess::DeathObserver& death_observer)
+{
+    std::unique_lock<std::mutex> lock(process_mutex_);
+
+    // 1. check if the scope is running.
+    //  1.1. if scope already running, return.
+    if (state_ == ScopeProcess::Running)
+    {
+        return;
+    }
+    //  1.2. if scope running but is “stopping”, wait for it to stop / kill it.
+    else if (state_ == ScopeProcess::Stopping)
+    {
+        if (!wait_for_state(lock, ScopeProcess::Stopped, c_process_wait_timeout))
+        {
+            cerr << "RegistryObject::ScopeProcess::exec(): Force killing process. Scope: \""
+                 << exec_data_.scope_id << "\" took too long to stop." << endl;
+            try
+            {
+                kill(lock);
+            }
+            catch(std::exception const& e)
+            {
+                cerr << "RegistryObject::ScopeProcess::exec(): " << e.what() << endl;
+            }
+        }
+    }
+
+    // 2. exec the scope.
+    update_state_unlocked(Starting);
+
+    const std::string program{exec_data_.scoperunner_path};
+    const std::vector<std::string> argv = {exec_data_.runtime_config, exec_data_.scope_config};
+
+    std::map<std::string, std::string> env;
+    core::posix::this_process::env::for_each([&env](const std::string& key, const std::string& value)
+    {
+        env.insert(std::make_pair(key, value));
+    });
+
+    {
+        process_ = core::posix::exec(program, argv, env,
+                                     core::posix::StandardStream::stdin | core::posix::StandardStream::stdout);
+        if (process_.pid() <= 0)
+        {
+            clear_handle_unlocked();
+            throw unity::ResourceException("RegistryObject::ScopeProcess::exec(): Failed to exec scope via command: \""
+                                           + exec_data_.scoperunner_path + " " + exec_data_.runtime_config + " "
+                                           + exec_data_.scope_config + "\"");
+        }
+    }
+
+    ///! TODO: This should not be here. A ready signal from the scope should trigger "running".
+    update_state_unlocked(Running);
+
+    // 3. wait for scope to be "running".
+    //  3.1. when ready, return.
+    //  3.2. OR if timeout, kill process and throw.
+    if (!wait_for_state(lock, ScopeProcess::Running, c_process_wait_timeout))
+    {
+        try
+        {
+            kill(lock);
+        }
+        catch(std::exception const& e)
+        {
+            cerr << "RegistryObject::ScopeProcess::exec(): " << e.what() << endl;
+        }
+        throw unity::ResourceException("RegistryObject::ScopeProcess::exec(): exec aborted. Scope: \""
+                                       + exec_data_.scope_id + "\" took too long to start.");
+    }
+
+    cout << "RegistryObject::ScopeProcess::exec(): Process for scope: \"" << exec_data_.scope_id << "\" started" << endl;
+
+    // 4. add the scope process to the death observer
+    death_observer.add(process_);
+}
+
+void RegistryObject::ScopeProcess::kill()
+{
+    std::unique_lock<std::mutex> lock(process_mutex_);
+    kill(lock);
+}
+
+bool RegistryObject::ScopeProcess::on_process_death(pid_t pid)
+{
+    std::lock_guard<std::mutex> lock(process_mutex_);
+
+    // check if this is the process reported to have died
+    if (pid == process_.pid())
+    {
+        cout << "RegistryObject::ScopeProcess::on_process_death(): Process for scope: \"" << exec_data_.scope_id
+             << "\" terminated" << endl;
+        clear_handle_unlocked();
+        return true;
+    }
+
+    return false;
+}
+
+void RegistryObject::ScopeProcess::clear_handle_unlocked()
+{
+    process_ = core::posix::ChildProcess::invalid();
+    update_state_unlocked(Stopped);
+}
+
+void RegistryObject::ScopeProcess::update_state_unlocked(ProcessState state)
+{
+    state_ = state;
+    state_change_cond_.notify_all();
+}
+
+bool RegistryObject::ScopeProcess::wait_for_state(std::unique_lock<std::mutex>& lock,
+                                                  ProcessState state, int timeout_ms) const
+{
+    return state_change_cond_.wait_for(lock,
+                                       std::chrono::milliseconds(timeout_ms),
+                                       [this, &state]{return state_ == state;});
+}
+
+void RegistryObject::ScopeProcess::kill(std::unique_lock<std::mutex>& lock)
+{
+    if (state_ == Stopped)
+    {
+        return;
+    }
+
+    try
+    {
+        // first try to close the scope process gracefully
+        process_.send_signal_or_throw(core::posix::Signal::sig_term);
+
+        if (!wait_for_state(lock, ScopeProcess::Stopped, c_process_wait_timeout))
+        {
+            std::error_code ec;
+
+            cerr << "RegistryObject::ScopeProcess::kill(): Scope: \"" << exec_data_.scope_id
+                 << "\" is taking longer than expected to terminate gracefully. "
+                 << "Killing the process instead." << endl;
+
+            // scope is taking too long to close, send kill signal
+            process_.send_signal(core::posix::Signal::sig_kill, ec);
+        }
+    }
+    catch (std::exception const&)
+    {
+        cerr << "RegistryObject::ScopeProcess::kill(): Failed to kill scope: \""
+             << exec_data_.scope_id << "\"" << endl;
+        throw;
+    }
+
+    // clear the process handle
+    // even on error, the previous handle will be unrecoverable at this point
+    clear_handle_unlocked();
 }
 
 } // namespace internal

=== modified file 'src/scopes/internal/RuntimeImpl.cpp'
--- src/scopes/internal/RuntimeImpl.cpp	2014-03-07 04:19:32 +0000
+++ src/scopes/internal/RuntimeImpl.cpp	2014-03-28 15:41:24 +0000
@@ -28,7 +28,10 @@
 #include <unity/scopes/ScopeExceptions.h>
 #include <unity/UnityExceptions.h>
 
+#include <signal.h>
+
 #include <cassert>
+#include <cstring>
 #include <future>
 #include <iostream> // TODO: remove this once logging is added
 
@@ -56,6 +59,7 @@
     }
 
     string config_file(configfile.empty() ? DFLT_RUNTIME_INI : configfile);
+    configfile_ = config_file;
 
     try
     {
@@ -135,6 +139,11 @@
     return scope_id_;
 }
 
+string RuntimeImpl::configfile() const
+{
+    return configfile_;
+}
+
 MiddlewareFactory const* RuntimeImpl::factory() const
 {
     if (destroyed_.load())

=== modified file 'src/scopes/internal/smartscopes/SSRegistryObject.cpp'
--- src/scopes/internal/smartscopes/SSRegistryObject.cpp	2014-03-20 16:19:04 +0000
+++ src/scopes/internal/smartscopes/SSRegistryObject.cpp	2014-03-28 15:41:24 +0000
@@ -45,12 +45,12 @@
 
 SSRegistryObject::SSRegistryObject(MiddlewareBase::SPtr middleware,
                                    std::string const& ss_scope_endpoint,
-                                   uint no_reply_timeout,
+                                   uint http_reply_timeout,
                                    uint refresh_rate_in_sec,
                                    std::string const& sss_url,
                                    bool caching_enabled)
     : ssclient_(std::make_shared<SmartScopesClient>(
-                    std::make_shared<HttpClientQt>(no_reply_timeout),
+                    std::make_shared<HttpClientQt>(http_reply_timeout),
                     std::make_shared<JsonCppNode>(), sss_url))
     , refresh_stopped_(false)
     , middleware_(middleware)
@@ -85,12 +85,12 @@
     refresh_thread_.join();
 }
 
-ScopeMetadata SSRegistryObject::get_metadata(std::string const& scope_id)
+ScopeMetadata SSRegistryObject::get_metadata(std::string const& scope_id) const
 {
-    // If the name is empty, it was sent as empty by the remote client.
+    // If the id is empty, it was sent as empty by the remote client.
     if (scope_id.empty())
     {
-        throw unity::InvalidArgumentException("SSRegistryObject: Cannot search for scope with empty name");
+        throw unity::InvalidArgumentException("SSRegistryObject: Cannot search for scope with empty id");
     }
 
     std::lock_guard<std::mutex> lock(scopes_mutex_);
@@ -103,7 +103,7 @@
     return it->second;
 }
 
-MetadataMap SSRegistryObject::list()
+MetadataMap SSRegistryObject::list() const
 {
     std::lock_guard<std::mutex> lock(scopes_mutex_);
     return scopes_;
@@ -274,7 +274,8 @@
     if (changed)
     {
         // something has changed, send invalidate signal
-        system(c_dbussend_cmd);
+        int result = system(c_dbussend_cmd);
+        (void)result;
     }
 }
 
@@ -282,7 +283,7 @@
 {
     if (metadata.scope_id().empty())
     {
-        throw unity::InvalidArgumentException("SSRegistryObject: Cannot add scope with empty name");
+        throw unity::InvalidArgumentException("SSRegistryObject: Cannot add scope with empty id");
     }
 
     // store the base url under a scope name key

=== modified file 'test/gtest/scopes/internal/zmq_middleware/RegistryI/RegistryI_test.cpp'
--- test/gtest/scopes/internal/zmq_middleware/RegistryI/RegistryI_test.cpp	2014-03-07 12:31:25 +0000
+++ test/gtest/scopes/internal/zmq_middleware/RegistryI/RegistryI_test.cpp	2014-03-28 15:41:24 +0000
@@ -42,6 +42,30 @@
 using namespace unity::scopes::internal;
 using namespace unity::scopes::internal::zmq_middleware;
 
+namespace
+{
+struct Scope
+{
+    Scope()
+        : trap(core::posix::trap_signals_for_all_subsequent_threads({core::posix::Signal::sig_chld})),
+          death_observer(core::posix::ChildProcess::DeathObserver::create_once_with_signal_trap(trap)),
+          worker([this]() { trap->run(); })
+    {
+    }
+
+    ~Scope()
+    {
+        trap->stop();
+
+        if (worker.joinable())
+            worker.join();
+    }
+
+    std::shared_ptr<core::posix::SignalTrap> trap;
+    std::unique_ptr<core::posix::ChildProcess::DeathObserver> death_observer;
+    std::thread worker;
+} scope;
+
 ScopeMetadata make_meta(const string& name, MWScopeProxy const& proxy, MiddlewareBase::SPtr const& mw)
 {
     unique_ptr<ScopeMetadataImpl> mi(new ScopeMetadataImpl(mw.get()));
@@ -56,10 +80,11 @@
     mi->set_proxy(p);
     return ScopeMetadataImpl::create(move(mi));
 }
+}
 
 TEST(RegistryI, get_metadata)
 {
-    vector<string> dummy_spawn_command;
+    RegistryObject::ScopeExecData dummy_exec_data;
     RuntimeImpl::UPtr runtime = RuntimeImpl::create(
         "TestRegistry", TEST_BUILD_ROOT "/gtest/scopes/internal/zmq_middleware/RegistryI/Runtime.ini");
 
@@ -70,11 +95,11 @@
     string mw_configfile = c.mw_configfile();
 
     MiddlewareBase::SPtr middleware = runtime->factory()->create(identity, mw_kind, mw_configfile);
-    RegistryObject::SPtr ro(make_shared<RegistryObject>());
+    RegistryObject::SPtr ro(make_shared<RegistryObject>(*scope.death_observer));
     auto registry = middleware->add_registry_object(identity, ro);
     auto p = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
     EXPECT_TRUE(ro->add_local_scope("scope1", move(make_meta("scope1", p, middleware)),
-            dummy_spawn_command));
+            dummy_exec_data));
 
     auto r = runtime->registry();
     auto scope = r->get_metadata("scope1");
@@ -93,17 +118,17 @@
     string mw_configfile = c.mw_configfile();
 
     MiddlewareBase::SPtr middleware = runtime->factory()->create(identity, mw_kind, mw_configfile);
-    RegistryObject::SPtr ro(make_shared<RegistryObject>());
+    RegistryObject::SPtr ro(make_shared<RegistryObject>(*scope.death_observer));
     auto registry = middleware->add_registry_object(identity, ro);
 
     auto r = runtime->registry();
     auto scopes = r->list();
     EXPECT_TRUE(scopes.empty());
 
-    vector<string> dummy_spawn_command;
+    RegistryObject::ScopeExecData dummy_exec_data;
     auto proxy = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
     EXPECT_TRUE(ro->add_local_scope("scope1", move(make_meta("scope1", proxy, middleware)),
-            dummy_spawn_command));
+            dummy_exec_data));
     scopes = r->list();
     EXPECT_EQ(1u, scopes.size());
     EXPECT_NE(scopes.end(), scopes.find("scope1"));
@@ -117,7 +142,7 @@
     {
         string long_id = "0000000000000000000000000000000000000000000000" + to_string(i);
         EXPECT_TRUE(ro->add_local_scope(long_id, move(make_meta(long_id, proxy, middleware)),
-                dummy_spawn_command));
+                dummy_exec_data));
         ids.insert(long_id);
     }
     scopes = r->list();
@@ -142,22 +167,22 @@
     string mw_configfile = c.mw_configfile();
 
     MiddlewareBase::SPtr middleware = runtime->factory()->create(identity, mw_kind, mw_configfile);
-    RegistryObject::SPtr ro(make_shared<RegistryObject>());
+    RegistryObject::SPtr ro(make_shared<RegistryObject>(*scope.death_observer));
     auto registry = middleware->add_registry_object(identity, ro);
 
     auto r = runtime->registry();
     auto scopes = r->list();
     EXPECT_TRUE(scopes.empty());
 
-    vector<string> dummy_spawn_command;
+    RegistryObject::ScopeExecData dummy_exec_data;
     auto proxy = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
     EXPECT_TRUE(ro->add_local_scope("scope1", move(make_meta("scope1", proxy, middleware)),
-            dummy_spawn_command));
+            dummy_exec_data));
     scopes = r->list();
     EXPECT_EQ(1u, scopes.size());
     EXPECT_NE(scopes.end(), scopes.find("scope1"));
     EXPECT_FALSE(ro->add_local_scope("scope1", move(make_meta("scope1", proxy, middleware)),
-            dummy_spawn_command));
+            dummy_exec_data));
 
     EXPECT_TRUE(ro->remove_local_scope("scope1"));
     scopes = r->list();
@@ -169,7 +194,7 @@
     {
         string long_id = "0000000000000000000000000000000000000000000000" + to_string(i);
         ro->add_local_scope(long_id, move(make_meta(long_id, proxy, middleware)),
-                dummy_spawn_command);
+                dummy_exec_data);
         ids.insert(long_id);
     }
     scopes = r->list();
@@ -194,11 +219,11 @@
     string mw_configfile = c.mw_configfile();
 
     MiddlewareBase::SPtr middleware = runtime->factory()->create(identity, mw_kind, mw_configfile);
-    RegistryObject::SPtr ro(make_shared<RegistryObject>());
-    vector<string> dummy_spawn_command;
+    RegistryObject::SPtr ro(make_shared<RegistryObject>(*scope.death_observer));
+    RegistryObject::ScopeExecData dummy_exec_data;
     auto registry = middleware->add_registry_object(identity, ro);
     auto proxy = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
-    ro->add_local_scope("scope1", move(make_meta("scope1", proxy, middleware)), dummy_spawn_command);
+    ro->add_local_scope("scope1", move(make_meta("scope1", proxy, middleware)), dummy_exec_data);
 
     auto r = runtime->registry();
 
@@ -221,19 +246,19 @@
     catch (MiddlewareException const& e)
     {
         EXPECT_STREQ("unity::scopes::MiddlewareException: unity::InvalidArgumentException: "
-                     "RegistryObject::get_metadata(): Cannot search for scope with empty name",
+                     "RegistryObject::get_metadata(): Cannot search for scope with empty id",
                      e.what());
     }
 
     try
     {
         auto proxy = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
-        ro->add_local_scope("", move(make_meta("blah", proxy, middleware)), dummy_spawn_command);
+        ro->add_local_scope("", move(make_meta("blah", proxy, middleware)), dummy_exec_data);
         FAIL();
     }
     catch (InvalidArgumentException const& e)
     {
-        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::add_local_scope(): Cannot add scope with empty name",
+        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::add_local_scope(): Cannot add scope with empty id",
                      e.what());
     }
 
@@ -244,7 +269,7 @@
     }
     catch (InvalidArgumentException const& e)
     {
-        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::remove_local_scope(): Cannot remove scope with empty name",
+        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::remove_local_scope(): Cannot remove scope with empty id",
                      e.what());
     }
 }
@@ -255,25 +280,26 @@
 class MockRegistryObject : public RegistryObject
 {
 public:
-    MockRegistryObject()
+    MockRegistryObject(core::posix::ChildProcess::DeathObserver& death_observer)
+        : RegistryObject(death_observer)
     {
     }
 
-    virtual ScopeProxy locate(string const& scope_name) override
+    virtual ScopeProxy locate(string const& scope_id) override
     {
-        if (scope_name == "no_such_scope")
+        if (scope_id == "no_such_scope")
         {
-            throw NotFoundException("no can find", scope_name);
+            throw NotFoundException("no can find", scope_id);
         }
-        if (scope_name == "error_scope")
+        if (scope_id == "error_scope")
         {
             throw RegistryException("Couldn't start error_scope");
         }
-        return get_metadata(scope_name).proxy();
+        return get_metadata(scope_id).proxy();
     }
 };
 
-TEST(RegistryI, locate)
+TEST(RegistryI, locate_mock)
 {
     RuntimeImpl::UPtr runtime = RuntimeImpl::create(
         "TestRegistry", TEST_BUILD_ROOT "/gtest/scopes/internal/zmq_middleware/RegistryI/Runtime.ini");
@@ -283,15 +309,15 @@
     string mw_kind = c.mw_kind();
     string mw_endpoint = c.endpoint();
     string mw_configfile = c.mw_configfile();
-    vector<string> dummy_spawn_command;
+    RegistryObject::ScopeExecData dummy_exec_data;
 
     MiddlewareBase::SPtr middleware = runtime->factory()->create(identity, mw_kind, mw_configfile);
-    MockRegistryObject::SPtr mro(make_shared<MockRegistryObject>());
+    MockRegistryObject::SPtr mro(make_shared<MockRegistryObject>(*scope.death_observer));
     auto r = middleware->add_registry_object(identity, mro);
     auto r_proxy = dynamic_pointer_cast<ZmqRegistry>(r);
     auto proxy = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
     mro->add_local_scope("scope1", move(make_meta("scope1", proxy, middleware)),
-            dummy_spawn_command);
+            dummy_exec_data);
 
     auto p = r_proxy->locate("scope1");
     EXPECT_EQ("scope1", p->identity());
@@ -322,12 +348,12 @@
     {
         auto proxy = middleware->create_scope_proxy("scope1", "ipc:///tmp/scope1");
         mro->add_local_scope("", move(make_meta("blah", proxy, middleware)),
-                 dummy_spawn_command);
+                 dummy_exec_data);
         FAIL();
     }
     catch (InvalidArgumentException const& e)
     {
-        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::add_local_scope(): Cannot add scope with empty name",
+        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::add_local_scope(): Cannot add scope with empty id",
                      e.what());
     }
 
@@ -338,7 +364,185 @@
     }
     catch (InvalidArgumentException const& e)
     {
-        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::remove_local_scope(): Cannot remove scope with empty name",
+        EXPECT_STREQ("unity::InvalidArgumentException: RegistryObject::remove_local_scope(): Cannot remove scope with empty id",
                      e.what());
     }
 }
+
+std::string exec_cmd(std::string const& cmd)
+{
+    FILE* pipe = popen(cmd.c_str(), "r");
+    if (!pipe)
+    {
+        return "";
+    }
+
+    char buffer[128];
+    std::string result;
+    while (!feof(pipe))
+    {
+        if (fgets(buffer, 128, pipe) != NULL)
+        {
+            result += buffer;
+        }
+    }
+    pclose(pipe);
+
+    return result;
+}
+
+int first_child_pid()
+{
+    return stoi(exec_cmd("ps --ppid " + std::to_string(getpid()) + " --no-headers"));
+}
+
+int process_count()
+{
+    return stoi(exec_cmd("ps --ppid " + std::to_string(getpid()) + " | wc -l"));
+}
+
+TEST(RegistryI, locate)
+{
+    // get number of processes belonging to this test instance
+    int start_process_count = process_count();
+    int current_process_count = 0;
+    {
+        // configure registry
+        std::string rt_config = TEST_BUILD_ROOT "/gtest/scopes/internal/zmq_middleware/RegistryI/Runtime.ini";
+
+        RuntimeImpl::UPtr rt = RuntimeImpl::create("TestRegistry", rt_config);
+        string reg_id = rt->registry_identity();
+
+        RegistryConfig c(reg_id, rt->registry_configfile());
+        string mw_kind = c.mw_kind();
+        string scoperunner_path = c.scoperunner_path();
+
+        MiddlewareBase::SPtr mw = rt->factory()->find(reg_id, mw_kind);
+
+        RegistryObject::SPtr reg(new RegistryObject(*scope.death_observer));
+        mw->add_registry_object(reg_id, reg);
+
+        // configure scopes
+        std::array<std::string, 6> scope_ids = {"scope-A", "scope-B", "scope-C", "scope-D", "scope-N", "scope-S"};
+        std::map<std::string, ScopeProxy> proxies;
+
+        for (auto& scope_id : scope_ids)
+        {
+            proxies[scope_id] = ScopeImpl::create(mw->create_scope_proxy(scope_id), mw->runtime(), scope_id);
+
+            unique_ptr<ScopeMetadataImpl> mi(new ScopeMetadataImpl(mw.get()));
+            mi->set_scope_id(scope_id);
+            mi->set_display_name(scope_id);
+            mi->set_description(scope_id);
+            mi->set_author("Canonical Ltd.");
+            mi->set_proxy(proxies[scope_id]);
+            auto meta = ScopeMetadataImpl::create(move(mi));
+
+            RegistryObject::ScopeExecData exec_data;
+            exec_data.scope_id = scope_id;
+            exec_data.scoperunner_path = scoperunner_path;
+            exec_data.runtime_config = rt_config;
+            exec_data.scope_config = TEST_BUILD_ROOT "/../demo/scopes/" + scope_id + "/" + scope_id + ".ini";
+
+            reg->add_local_scope(scope_id, move(meta), exec_data);
+        }
+
+        // test initial state
+        {
+            // check that no scope processes are running
+            for (auto const& scope_id : scope_ids)
+            {
+                EXPECT_FALSE(reg->is_scope_running(scope_id));
+            }
+
+            // check that no new processes have been started yet
+            current_process_count = process_count();
+            EXPECT_EQ(0, current_process_count - start_process_count);
+        }
+
+        // test scope death and re-locate
+        {
+            // locate first scope
+            EXPECT_EQ(proxies[scope_ids[0]], reg->locate(scope_ids[0]));
+
+            // check that the first scope is running
+            EXPECT_TRUE(reg->is_scope_running(scope_ids[0]));
+
+            // check that 1 new process was started
+            current_process_count = process_count();
+            EXPECT_EQ(1, current_process_count - start_process_count);
+
+            // kill first scope
+            int scope1_pid = first_child_pid();
+            kill(scope1_pid, SIGKILL);
+
+            // wait for the SIGCHLD signal to reach the registry
+            while (reg->is_scope_running(scope_ids[0]))
+            {
+                std::this_thread::sleep_for(std::chrono::milliseconds{10});
+            }
+
+            // check that we now have no running scopes
+            current_process_count = process_count();
+            EXPECT_EQ(0, current_process_count - start_process_count);
+
+            // locate first scope
+            EXPECT_EQ(proxies[scope_ids[0]], reg->locate(scope_ids[0]));
+
+            // check that the first scope is running
+            EXPECT_TRUE(reg->is_scope_running(scope_ids[0]));
+
+            // check that 1 new process was started
+            current_process_count = process_count();
+            EXPECT_EQ(1, current_process_count - start_process_count);
+        }
+
+        // test locating all scopes
+        {
+            // locate all scopes (hense starting all scope processes)
+            for (auto const& scope_id : scope_ids)
+            {
+                EXPECT_EQ(proxies[scope_id], reg->locate(scope_id));
+            }
+
+            // check that all scopes processes are running
+            for (auto const& scope_id : scope_ids)
+            {
+                EXPECT_TRUE(reg->is_scope_running(scope_id));
+            }
+
+            // check that 6 new processes were started
+            current_process_count = process_count();
+            EXPECT_EQ(6, current_process_count - start_process_count);
+        }
+
+        // test locating the same scope multiple times
+        {
+            // locate the second scope multiple times
+            for (int i = 0; i < 1000; ++i)
+            {
+                EXPECT_EQ(proxies[scope_ids[1]], reg->locate(scope_ids[1]));
+            }
+
+            // check that no new processes were started
+            current_process_count = process_count();
+            EXPECT_EQ(6, current_process_count - start_process_count);
+        }
+
+        // test removing a scope
+        {
+            // remove a scope (hense killing the process)
+            EXPECT_TRUE(reg->remove_local_scope(scope_ids[0]));
+
+            // check that we now have 5 scopes running
+            current_process_count = process_count();
+            EXPECT_EQ(5, current_process_count - start_process_count);
+        }
+
+        // reg falls out of scope here and gets deleted (hense closing all scope processes)
+    }
+
+    // check that we are back to the original number of processes
+    current_process_count = process_count();
+    EXPECT_EQ(0, current_process_count - start_process_count);
+}

=== modified file 'test/gtest/scopes/internal/zmq_middleware/RegistryI/TestRegistry.ini.in'
--- test/gtest/scopes/internal/zmq_middleware/RegistryI/TestRegistry.ini.in	2014-01-14 12:54:15 +0000
+++ test/gtest/scopes/internal/zmq_middleware/RegistryI/TestRegistry.ini.in	2014-03-28 15:41:24 +0000
@@ -7,4 +7,4 @@
 Zmq.EndpointDir = /tmp
 Zmq.ConfigFile = Zmq.ini
 Scope.InstallDir = /tmp
-Scoperunner.Path = /tmp
+Scoperunner.Path = @CMAKE_BINARY_DIR@/scoperunner/scoperunner