Merge ~epics-core/epics-base/+git/Com:thread-join into ~epics-core/epics-base/+git/epics-base:7.0

[mdavidsaver]

Add epicsThreadJoin() to allow waiting for thread completion. Threads must be explicitly created as join-able. The default is not join-able.

Add epicsThreadCreateOpt() which accepts arguments as a structure for expandability.

The epicsThread class is modified to use epicsThreadJoin() automatically.

Implemented for all targets except vxWorks where epicsThreadJoin() is a no-op and EPICS_THREAD_CAN_JOIN==0.

[Andrew Johnson (anj)]

I'm going to need persuading why we want epicsThreadJoin() if it isn't going to be available on all platforms. I am concerned that by providing it, we allow people to write code that won't work without it, making that code inherently incompatible with the targets that don't support it.

Is there any particular reason that it can't be implemented on VxWorks, or is it just that you don't have the time and access? Doing so might be a good project to give Dirk, after adding test code to confirm that the implementation works.

If we do allow it on a subset of platforms, it would be better for VxWorks to not provide the routine at all (including declaring no function prototype for it — this is important since we can't detect someone calling a non-existent routine until boot/load time with VxWorks). That way any code that tries to call it despite the EPICS_THREAD_CAN_JOIN==0 will get a build-time error on VxWorks targets instead of them failing weirdly at runtime because the original author didn't think anyone would ever want to run this code on a platform without it.

review: Needs Information

[mdavidsaver (mdavidsaver)]

> Is there any particular reason that it can't be implemented on VxWorks, or is it just that you don't have the time and access?

I made it work for RTEMS, so I see no reason why it couldn't be made to work for vxWorks as well.

> If we do allow it on a subset of platforms

I would rather support all platforms.

[Andrew Johnson (anj)]

I noticed the other day that VxWorks 6.9 added this API:

    STATUS taskWait(TASK_ID taskId, _Vx_ticks_t timeout);

This behaves like pthread_join(), but without copying the thread's exit status to the caller.

Should the epicsThreadJoin() API take a timeout argument? That seems like a useful thing which could be hard or impossible to implement in portable application code, although I haven't looked too closely at how easy it would be to implement for the other OSs (RTEMS looks easy though). Adding this would also require adding a return status so the application knows what happened. If the routine returned a status value we could avoid the need to call cantProceed(). We could provide an epicsThreadMustJoin() API which does the cantProceed() thing...

Shouldn't the C++ class also have a join() method? The destructor acts as one so maybe it isn't essential, but there's no way to provide it with timeout value or get errors back.

I see that the epicsThread class always sets opts.joinable=1 even when the underlying OS defines EPICS_THREAD_CAN_JOIN (0) and only provides an empty epicsThreadJoin() routine. Is that what you implemented in the "epicsThread fix join" commit 149ab118?

Looking at the Posix implementation, a joinable thread that calls epicsThreadJoin(epicsThreadGetIdSelf()) should become unjoinable, but the code doesn't reset the id->joinable flag despite calling pthread_detach(). I didn't check the others for similar issues.

Do you have any test code for this? I just created an implementation for VxWorks 6.9 and later but it may be too simple, so adding some tests to epicsThreadTest.cpp would be helpful.

Will eventually need documentation and a Release Notes entry.

[mdavidsaver (mdavidsaver)]

> I noticed the other day that VxWorks 6.9 added this API:

8-()

> Should the epicsThreadJoin() API take a timeout argument?

I would say no. Given the nature of threads, join is only useful if the caller has already done something to ensure the thread will (eventually) return. If there is some doubt about this, then it should be handled at this earlier phase.

Otherwise it seems a bit like allowing free() to timeout.

> Shouldn't the C++ class also have a join() method?

I suppose this could be added. I was thinking first of a non-invasive change.

https://en.cppreference.com/w/cpp/thread/thread/join

> EPICS_THREAD_CAN_JOIN

This will go away now.

> epicsThreadJoin(epicsThreadGetIdSelf()) should become unjoinable

I'll take a look.

> adding some tests to epicsThreadTest.cpp would be helpful

Any thoughts on what to test? I couldn't think of any meaningful unittest to see if resources have actually been freed. Mostly I added prints while developing (especially for RTEMS), and ran epicsThreadTest through valgrind on Linux.

[Andrew Johnson (anj)]

I was thinking that a timeout would at least allow unit test programs to call testAbort() in the event that their threads don't die due to some other bug — this API might turn out to be more useful in unit tests than inside the IOC or other applications, and this could simplify making them more robust against not exiting properly.

I'm starting to think that calling cantProceed() instead of returning an error status might be a mistake, and as I suggested we can always add an epicsThreadMustJoin() routine for those applications that want that behavior. You have already had to change some of our older APIs that only returned void, it would be good to not make that mistake again.

VxWorks 6.8 and earlier still define EPICS_THREAD_CAN_JOIN (0)...

I would start with checks of the basic functionality, and that things the APIs are supposed to reject get rejected (such as self-calling epicsThreadJoin(epicsThreadGetIdSelf()) or on an unjoinable thread; Posix says you can't call pthread_detach() twice, etc. I agree that you probably can't check for resources being freed, but you could check timeout functionality...

[Martin Konrad (info-martin-konrad)]

> Should the epicsThreadJoin() API take a timeout argument?
Definitely no.

> Given the nature of threads, join is only useful if the caller has already done something to ensure the thread will (eventually) return.
I agree. We might want to spend some time making it easier for developers to get the "signal the thread to shutdown" part right. Maybe by providing a good example in the Application Developer's Guide?

> Otherwise it seems a bit like allowing free() to timeout.
It's actually worse: Not freeing something leads to a memory leak (usually a minor problem) but not joining a thread might result in data races which takes us straight to undefined behavior land.

[Martin Konrad (info-martin-konrad)]

I would argue that a stuck test exposes a problem the same way "FAILED" does. The same goes for threads: A thread that doesn't join usually points to a thread function that got stuck. If you want to ignore this problem temporarily you might want to disable/exclude the corresponding test.

[Andrew Johnson (anj)]

Disabling individual tests on the specific architectures where they fail isn't something we have done in the past. I'm not saying it can't be done, but that merely removes the test lockups from ever being seen. When tests fail to exit that mostly only affects me because I have to see that they've been running for hours and kill that job on the APS Jenkins server.

I would like to make it easier for test authors to build safeguards like timeouts into their code. If a test never fails for an author they aren't as likely to want to work on fixing the actual problem, and adding a timeout is an easy workaround that would save work for me. A test program will be reported as failing if it times out, whereas if I have to kill the job the test results don't get analyzed at all so the author doesn't see the problem (Jenkins doesn't send email when I kill a job).

In a perfect world I might agree that epicsThreadJoin() doesn't need a timeout, but it could make life a little easier in this world IMHO.

If it's too hard to implement a timeout on Posix I would accept that as a reason for not adding it, but I still think that epicsThreadJoin() should return a status, and epicsThreadMustJoin() should be added with the call to cantProceed() — this has been our practice in other APIs e.g. callocMustSucceed(), epicsMutexMustCreate(), epicsMutexMustLock(), epicsEventMustWait(), epicsEventMustTrigger(), epicsSpinMustCreate().

[Martin Konrad (info-martin-konrad)]

Keep in mind that we don't need threads to shoot ourselves in the foot (I can just as well call something equivalent to "while (true)" from a test directly) but I have to admit that multithreading makes it easier ;-)

The problem of CI runs that don't complete is bigger than joining threads. But there is a cure: https://plugins.jenkins.io/build-timeout

BTW: In Google Test I get a yellow "DISABLED" instead of a green "PASS"/a red "FAIL" when I prefix the name of a test with "DISABLE". This makes it hard to forget a disabled test.

[mdavidsaver (mdavidsaver)]

> But there is a cure: https://plugins.jenkins.io/build-timeout

I agree an overall timeout would be a better way to solve the Bothering Andrew part of this problem.

If Jenkins can't be made to work, then maybe makeTestfile.pl could be taught about SIGALRM.

> If it's too hard to implement a timeout on Posix I would accept that as a reason for not adding it

The standard pthread_join() doesn't have a timeout. A quick search finds pthread_tryjoin_np(), which is a glibc extension.

> that epicsThreadJoin() should return a status

What status? How would a caller react?

Part of my reasoning for not having a return value is that by the time a join is performed, there often isn't anything sensible to be done.

eg. I regard epicsMutexLock() returning as a mis-feature. There is nothing sensible which user code can do with a epicsMutexLockError. So why offer up the option? Seems an invitation to ignore a serious error.

[Andrew Johnson (anj)]

The jenkins "Build Timeout" plugin wouldn't help much, the documentation says:

  Once the timeout is reached, Jenkins behaves as if an invisible hand has
  clicked the "abort build" button.

So as I wrote above Jenkins won't send out any emails as a result, and now nobody knows that the job failed.

The problem with implementing a watchdog timeout in the wrapper script is that the timeout period really should vary with the specific test and the machine it's running on. As Lewis Muir pointed out, the code asking for the join knows best how long to wait before giving up, and also what to do if it fails.

The epicsThread class provides an exitWait(double seconds) method which is equivalent to join with a timeout. Even Python's Thread.join() method has a timeout parameter.

A return status value from epicsThreadJoin() would provide the same kinds of information to the caller that pthread_join() and taskWait() return: "Not a joinable thread", "Cannot wait on self", "Invalid thread ID", and even "Timeout". A fundamental library routine shouldn't dictate that the only way to handle errors is for it to ignore them or call cantProceed(); a unit test program at least should be able to call testAbort() instead. Just step back a moment and look at your 3 implementations of epicsThreadJoin() — a significant proportion of the code in them is just handling error conditions.

[Martin Konrad (info-martin-konrad)]

The problem I have with this is that

// ...
t1.joinWithTimeout(timeout);
// read something modified by t1

is a very common pattern. This is safe because join() acts as a barrier in terms of the memory model. If we proceed without ensuring the thread has shut down we need to ensure we never read/write anything that might be written/read by the thread because we can't be sure t1 has shut down. Otherwise we might have a data race (in other words: anything can happen).

Simple example:

void foo(atomic<bool>& terminate, int& shared) {
  while (!terminate)
    shared = ...;
}

{
  atomic<bool> terminate = false;
  int shared;
  thread t1(foo, terminate, shared);
  // ...
  terminate = true;
  t1.joinWithTimeout(timeout);
} // shared goes out of scope - but foo() might keep writing to it!

It's hard to use this correctly. Should you really implement a joinWithTimeout() I think it should come with a big fat warning in the documentation. I would certainly stay away from doing anything beyond calling "exit" if join times out since I'm pretty sure I wouldn't get it right.

P.S.: I like the idea of leveraging SIGALRM.

[J. Lewis Muir (jlmuir)]

(Hmm, my email reply to the special <email address hidden> email
address did not seem to generate a comment here, so I'm adding the
comment again via the web UI.)

> I would like to make it easier for test authors to build safeguards
> like timeouts into their code.

Agreed. The test case can know that a join should succeed within some
amount of time, and if it doesn't, it can fail the test explicitly with
a message saying exactly what went wrong. This is better than a test
that just hangs.

I think it also has value in production code for a similar reason: if
the caller knows that the join should succeed within a certain amount
of time, it allows the caller to handle the case where it doesn't. It
could log an error message, it could abort, it could do both, it could
try again, it could tell the thread to stop and then try again, etc.

[J. Lewis Muir (jlmuir)]

> It's hard to use this correctly. Should you really implement a
> joinWithTimeout() I think it should come with a big fat warning in the
> documentation. I would certainly stay away from doing anything beyond calling
> "exit" if join times out since I'm pretty sure I wouldn't get it right.

I don't understand the concern here. If you're dealing with threads, you have to understand the memory model; that's a given. There are a practically infinite number of ways I can write wrong code with this API as well as the entire existing EPICS Base API.

[Ralph Lange (ralph-lange)]

I like using clear names to distinguish stuff.
"tryjoin" (that Michael found to be a pthread extension) is a good example.

// ...
ret = t1.tryJoin(timeout);
// read something modified by t1

makes it pretty obvious that you should check the returned status before trying to read stuff that t1 should have modified.

[Andrew Johnson (anj)]

The name epicsThreadMustJoin() would tell someone who knows our other API names that the routine will call cantProceed() if it fails.

I would expect a routine called epicsThreadTryJoin() to be functionally equivalent to the glibc extension pthread_tryjoin_np() that Michael found. It should also be equivalent to passing in a timeout of zero to a timed version.

Might pthreads allow us to write our own equivalent to pthread_timedjoin_np() using waitpid()? I'm hoping for something that might also work on RTEMS-5, MacOS and non-glibc Linux targets...

[mdavidsaver (mdavidsaver)]

Ok, I think we are getting side tracked. I will renamed epicsThreadJoin() -> epicsThreadMustJoin(). However, I leave epicsThreadTryJoin() as future work from someone who wants to do it, and can find a portable way. I don't want this to hold up merging.

wrt. epicsThreadMustJoin() returning an error. The one case I can see which could be indicated in a race free way is self join. This is also a case which a caller can easily check and handle beforehand. So I don't see this as sufficient motivation.

wrt. clearing joinable. I'll change this so that trying to self join twice will be defined. However, other cases of double join, or join of unjoinable, will remain undefined (aka. race to crash).

wrt. vxWorks and taskWait(). I can't find any documentation for this. How does it handled a self wait? For that matter, how does it handle waiting for a thread which might already have returned? Does vxWorks have a notion of (not) detached threads?

wrt. Testing. I'll extend epicsThreadTest with at least an explicit test of self join.

review: Needs Fixing

[Andrew Johnson (anj)]

taskWait( )

NAME

taskWait( ) - wait for task to terminate

SYNOPSIS

STATUS taskWait
    (
    TASK_ID tid, /* task id */
    _Vx_ticks_t timeout /* timeout */
    )

DESCRIPTION

This routine will wait/pend until a task has been terminated.

WARNING

This routine may not be used from interrupt level

SMP CONSIDERATIONS

This API is spinlock and intCpuLock restricted. These restrictions are not enforced by the implementation so it is the responsibility of the caller to ensure they are complied with. Future implementations may enforce these restrictions.

RETURNS

OK when timeout or done waiting for task to be destroyed, ERROR on
         failure

ERRNO

S_intLib_NOT_ISR_CALLABLE
    Routine is not callable from an ISR.

S_taskLib_ILLEGAL_OPERATION
    Cannot wait on self

S_objLib_OBJ_ID_ERROR
    Invalid task ID

I think it may also set errno to one of these (NO_WAIT is a macro used for 0 ticks, WAIT_FOREVER is its counterpart):

S_objLib_OBJ_TIMEOUT
    Timeout occured while pending on sempahore.

S_objLib_OBJ_UNAVAILABLE
    Would have blocked but NO_WAIT was specified.

[mdavidsaver (mdavidsaver)]

> This almost seems too simple...

Unfortunately, yes. I think this is open to races involving re-use of thread IDs (and maybe use after free?). This potential is why the RTEMS version uses a barrier to prevent the thread from actually returning until it is joined (the rtems_id stays valid). Alternately, as with WIN32, allocated a separate epicsThreadId, with a ref. counter, that can outlive the OS task.

[Andrew Johnson (anj)]

The VxWorks implementation of osdThread is pretty thin — our thread ID is the OS's task ID, and in various places the code assumes the system is UP, so no memory barriers are required. I don't understand where else you could see races or user-after-frees that don't also exist in the native taskWait() API. Wind River implements taskWait() by adding the calling task to a wait queue that lives in the TCB of the task being waited for. The queue gets flushed (by a different task) to wake up all waiters when the task is deleted.

Now I agree that the current code is too simple since it doesn't check the status returned by taskWait() or look at errno, which obviously needs to be done and the subsequent behavior made to match the other implementations. I skipped doing that to have the discussion about returning an error status.

[mdavidsaver (mdavidsaver)]

Maybe I'm not understanding. What is the lifetime of the handle returned by taskSpawn()? Will it ever be free()'d? Will it ever be reused? eg. Will a second call to taskSpawn() ever return the same ID?

My assumption is that IDs are not free()d, and can be reused (like a file descriptor). If this is the case, then it is possible for a race to occur between the time I get hold of an epicsThreadId, and the time I call epicsThreadMustJoin() on it. If in this interval the original thread exits, and suppose a new thread is started which happens to get the same ID. In this case I end up trying to join a different thread!

Resolving this is what necessitates joinable vs. detached as user configuration. Either the handle should remain valid, for the same thread, until joined. Or it should be freed immediately when the thread returns.

So the fact that the vxWorks epicsThreadCreateOpt() does nothing with epicsThreadOpts::joinable tells me that there is a bug.

[mdavidsaver (mdavidsaver)]

Various changes, including the promised rename. I've also used epicsThreadMustJoin() in db_close_events(), for a considerable simplification.

review: Approve

[mdavidsaver (mdavidsaver)]

Was forgetting the vxworks issue

review: Needs Fixing

[Andrew Johnson (anj)]

AI on MAD to document semantics of epicsThreadJoin().
AI on AJ to finish VxWorks implementation.

[Andrew Johnson (anj)]

A working VxWorks 6.9+ implementation of epicsThreadMustJoin() is now included, and I added some more tests.

The changes made to asCaStop, dbCa and dbEvent mean the IOC now requires EPICS_THREAD_CAN_JOIN=1, so that macro is now obsolete and should be deleted. The minimum VxWorks version will also have to be bumped up to 6.9, which I documented in the Release Notes. The APS doesn't have a problem with that, although PSI might be unhappy.

On VxWorks a thread which sits waiting to be joined for longer than 60 seconds will log a warning message, as will a call to epicsThreadMustJoin(). The RTEMS and Windows implementations could probably do the same thing.

Could we replace the epicsThreadOptsDefaults() routine with a macro named EPICS_THREAD_OPTS_INIT which would be defined in osdThread.h (or even in epicsThread.h, see below). I'm looking to simplify this code:

    epicsThreadOpts opts;
    epicsThreadOptsDefaults(&opts);

into this:

    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;

which removes the need for that epicsThreadOptsDefaults() routine. This name matches similar ones elsewhere in libCom (ELLLIST_INIT, EPICS_THREAD_ONCE_INIT etc.).

The only reason we need an OS-specific definition of those defaults is for the stack size parameter, which is an unsigned integer. Since we're making changes in this area, why don't we modify the epicsThreadCreateOpt() routines to check opts->stackSize and pass it through epicsThreadGetStackSize() if the value given matches any of the epicsThreadStackSizeClass values (none of which are legal stack sizes on any architecture)? Code could still pass in an arch-specific value, but this simplifies the API which I have always found very clunky.

[mdavidsaver (mdavidsaver)]

Thanks Andrew.

> IOC now requires EPICS_THREAD_CAN_JOIN=1

Do you see tests failing with this change? If I was careful, then they shouldn't. The change should only result in transient resource leaks triggering false positives on a hypothetical target with EPICS_THREAD_CAN_JOIN=0 and valgrind support.

That said, I'm happy to be rid of EPICS_THREAD_CAN_JOIN.

> The RTEMS and Windows implementations could probably do the same thing.

I've no objection to this, but I'm not motivated to do this myself.

> ... EPICS_THREAD_OPTS_INIT ...

Also no objection.

review: Approve

[Andrew Johnson (anj)]

> The change should only result in transient resource leaks

Oops, sorry - you're right, I assumed that things would break if epicsThreadMustJoin() was a no-op, but looking more closely I guess they shouldn't, sorry! I have not looked in detail at the shutdown logic changes in dbEvent.c, I trust you to have tested that reasonably.

I have significantly adjusted and expanded the Release Notes again.

I have changed the EPICS_THREAD_CAN_JOIN macro, it is now only defined when a working epicsThreadMustJoin() is available, so external code can use a simple #ifdef to determine that the API is available and works on this Base version & target.

> ... EPICS_THREAD_OPTS_INIT ...

Done. External code can now use #ifdef EPICS_THREAD_OPTS_INIT to determine whether the epicsThreadCreateOpt() API is available on this Base version.

I just added another test to epicsThreadTest.cpp to check that a join actually delays the caller; the empty epicsThreadMustJoin() as compiled for VxWorks < 6.9 will return immediately. This check should detect that and fail on VxWorks 6.8 say, but is marked as ToDo when that's expected.

I'm happy with this now, others may wish to take a final look given my recent updates.

review: Approve

[mdavidsaver (mdavidsaver)]

> the shutdown logic changes in dbEvent.c

This is in part reverting d2b0e920012d4c9bc0eaecc0e092cedd776db578 now that a better solution is available.

> I trust you to have tested that reasonably.

Do I get some rope too?

anyway... Your changes look fine.

I would ask that future formatt cleanup be done in a separate commit. I note that the posix osdThread.c cleanup is going to cause conflicts if/when merging https://github.com/epics-base/epics-base/pull/24 If this was done with a separate commit, the merge process might be simplified by first reverting the cleanup.

diff --git a/documentation/RELEASE_NOTES.html b/documentation/RELEASE_NOTES.html
index df78360..178afa4 100644
--- a/documentation/RELEASE_NOTES.html
+++ b/documentation/RELEASE_NOTES.html
@@ -30,6 +30,7 @@ release.</p>
 
 -->
 
+<<<<<<< documentation/RELEASE_NOTES.html
 <h1 align="center">EPICS Release 7.0.2.2</h1>
 
 <h3>Build System changes</h3>
@@ -127,6 +128,89 @@ than it used to be.</p>
 
 
 <h1 align="center">EPICS Release 7.0.2</h1>
+=======
+<h3>VxWorks Minimum Version Recommendation</h3>
+
+<p>The implementation of the <code>epicsThreadMustJoin()</code> feature
+described below uses facilities that were added to VxWorks in version 6.9. When
+built against an older version of VxWorks the join functionality will not be
+available and calls to <code>epicsThreadMustJoin()</code> will return
+immediately. In this case the epicsThread.h header will not define the C macro
+<code>EPICS_THREAD_CAN_JOIN</code> to allow alternate code to be provided for
+these targets. The IOC's use of the join feature has been designed to work for
+either situation.</p>
+
+
+<h3>New and modified epicsThread APIs</h3>
+
+<h4><code>epicsThreadCreateOpt()</code></h4>
+
+<p>A new routine <code>epicsThreadCreateOpt()</code> is an alternative to
+<code>epicsThreadCreate()</code> which takes some arguments via a structure
+(<code>struct epicsThreadOpts</code>) to allow for future extensions.</p>
+
+<blockquote><pre>
+typedef struct epicsThreadOpts {
+    unsigned int priority;
+    unsigned int stackSize;
+    unsigned int joinable;
+} epicsThreadOpts;
+#define EPICS_THREAD_OPTS_INIT { \
+    epicsThreadPriorityLow, epicsThreadStackMedium, 0}
+
+epicsThreadId epicsThreadCreateOpt(const char * name,
+    EPICSTHREADFUNC funptr, void * parm, const epicsThreadOpts *opts);
+</pre></blockquote>
+
+<p>The final <code>opts</code> parameter may be <code>NULL</code> to use the
+default values of thread priority (low) and stack size (medium). Callers wishing
+to provide alternative settings for these thread options or to create a joinable
+thread (see below) should create and pass in an <code>epicsThreadOpts</code>
+structure as shown below. Always initialize one of these structures using the
+<code>EPICS_THREAD_OPTS_INIT</code> macro to ensure that any additional fields
+that get added in the future are set to their default values.</p>
+
+<blockquote><pre>
+void startitup(void) {
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+    epicsThreadId tid;
+
+    opts.priority = epicsThreadPriorityMedium;
+    tid = epicsThreadCreateOpt("my thread", &amp;threadMain, NULL, &amp;opts);
+}
+</pre></blockquote>
+
+<p>C or C++ Code that also needs to build on earlier versions of Base can use
+<code>#ifdef&nbsp;EPICS_THREAD_OPTS_INIT</code> to determine whether the
+<code>epicsThreadCreateOpt()</code> API is available on this Base version.</p>
+
+<h4>Thread stack sizes</h4>
+
+<p>The <code>stackSize</code> member of the <code>epicsThreadOpts</code>
+structure and the equivalent parameters to the <code>epicsThreadCreate()</code>
+and <code>epicsThreadMustCreate()</code> routines can now be passed either one
+of the <code>epicsThreadStackSizeClass</code> enum values or a value returned
+from the <code>epicsThreadGetStackSize()</code> routine.</p>
+
+<h4><code>epicsThreadMustJoin()</code></h4>
+
+<p>If the new <code>joinable</code> flag of an <code>epicsThreadOpts</code>
+structure is non-zero (the default value is zero), the new API routine
+<code>epicsThreadMustJoin()</code> <em>must</em> be called with the thread's
+<code>epicsThreadId</code> when/after the thread exits, to free up thread
+resources. This function will block until the thread's main function has
+returned, allowing the parent to wait for its child thread. The child's
+<code>epicsThreadId</code> will no longer be valid and should not be used after
+the <code>epicsThreadMustJoin()</code> routine returns.</p>
+
+<p>A thread that was originally created with its joinable flag set may itself
+call <code>epicsThreadMustJoin()</code>, passing in its own epicsThreadId. This
+marks the thread as no longer being joinable, so it will then free the thread
+resources itself when its main function returns. The <code>epicsThreadId</code>
+of a thread that is not joinable gets invalidated as soon as its main function
+returns.</p>
+
+>>>>>>> documentation/RELEASE_NOTES.html
 
 <h3>Launchpad Bugs</h3>
 
diff --git a/modules/database/src/ioc/as/asCa.c b/modules/database/src/ioc/as/asCa.c
index d018044..21bb47f 100644
--- a/modules/database/src/ioc/as/asCa.c
+++ b/modules/database/src/ioc/as/asCa.c
@@ -229,20 +229,23 @@ static void asCaTask(void)
 
    
 void asCaStart(void)
 {
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+
+    opts.stackSize = epicsThreadGetStackSize(epicsThreadStackBig);
+    opts.priority = epicsThreadPriorityScanLow - 3;
+    opts.joinable = 1;
+
     if(asCaDebug) printf("asCaStart called\n");
     if(firstTime) {
-	firstTime = FALSE;
+        firstTime = FALSE;
         asCaTaskLock=epicsMutexMustCreate();
         asCaTaskWait=epicsEventMustCreate(epicsEventEmpty);
         asCaTaskAddChannels=epicsEventMustCreate(epicsEventEmpty);
         asCaTaskClearChannels=epicsEventMustCreate(epicsEventEmpty);
-        threadid = epicsThreadCreate("asCaTask",
-            (epicsThreadPriorityScanLow - 3),
-            epicsThreadGetStackSize(epicsThreadStackBig),
-            (EPICSTHREADFUNC)asCaTask,0);
-	if(threadid==0) {
-	    errMessage(0,"asCaStart: taskSpawn Failure\n");
-	}
+        threadid = epicsThreadCreateOpt("asCaTask", (EPICSTHREADFUNC)asCaTask, 0, &opts);
+        if(threadid==0) {
+            errMessage(0,"asCaStart: taskSpawn Failure\n");
+        }
     }
     epicsMutexMustLock(asCaTaskLock);
     epicsEventSignal(asCaTaskAddChannels);
@@ -260,6 +263,8 @@ void asCaStop(void)
     epicsEventMustWait(asCaTaskWait);
     if(asCaDebug) printf("asCaStop done\n");
     epicsMutexUnlock(asCaTaskLock);
+    epicsThreadMustJoin(threadid);
+    threadid = 0;
 }
 
 int ascar(int level) { return ascarFP(stdout,level);}
diff --git a/modules/database/src/ioc/db/dbCa.c b/modules/database/src/ioc/db/dbCa.c
index 843fbfc..45e178c 100644
--- a/modules/database/src/ioc/db/dbCa.c
+++ b/modules/database/src/ioc/db/dbCa.c
@@ -68,6 +68,7 @@ static volatile enum dbCaCtl_t {
     ctlInit, ctlRun, ctlPause, ctlExit
 } dbCaCtl;
 static epicsEventId startStopEvent;
+static epicsThreadId dbCaWorker;
 
 struct ca_client_context * dbCaClientContext;
 
@@ -258,10 +259,18 @@ void dbCaShutdown(void)
     dbCaCtl = ctlExit;
     epicsEventSignal(workListEvent);
     epicsEventMustWait(startStopEvent);
+    if(dbCaWorker)
+        epicsThreadMustJoin(dbCaWorker);
 }
 
 static void dbCaLinkInitImpl(int isolate)
 {
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+
+    opts.stackSize = epicsThreadGetStackSize(epicsThreadStackBig);
+    opts.priority = epicsThreadPriorityMedium;
+    opts.joinable = 1;
+
     dbServiceIsolate = isolate;
     dbServiceIOInit();
 
@@ -274,9 +283,8 @@ static void dbCaLinkInitImpl(int isolate)
         startStopEvent = epicsEventMustCreate(epicsEventEmpty);
     dbCaCtl = ctlPause;
 
-    epicsThreadCreate("dbCaLink", epicsThreadPriorityMedium,
-        epicsThreadGetStackSize(epicsThreadStackBig),
-        dbCaTask, NULL);
+    dbCaWorker = epicsThreadCreateOpt("dbCaLink", dbCaTask, NULL, &opts);
+    /* wait for worker to startup and initialize dbCaClientContext */
     epicsEventMustWait(startStopEvent);
 }
 
diff --git a/modules/database/src/ioc/db/dbEvent.c b/modules/database/src/ioc/db/dbEvent.c
index 77b12de..5e0f4db 100644
--- a/modules/database/src/ioc/db/dbEvent.c
+++ b/modules/database/src/ioc/db/dbEvent.c
@@ -84,7 +84,6 @@ struct event_user {
     epicsMutexId        lock;
     epicsEventId        ppendsem;       /* Wait while empty */
     epicsEventId        pflush_sem;     /* wait for flush */
-    epicsEventId        pexitsem;       /* wait for event task to join */
 
     EXTRALABORFUNC      *extralabor_sub;/* off load to event task */
     void                *extralabor_arg;/* parameter to above */
@@ -123,8 +122,6 @@ static char *EVENT_PEND_NAME = "eventTask";
 
 static struct evSubscrip canceledEvent;
 
-static epicsMutexId stopSync;
-
 static unsigned short ringSpace ( const struct event_que *pevq )
 {
     if ( pevq->evque[pevq->putix] == EVENTQEMPTY ) {
@@ -266,10 +263,6 @@ dbEventCtx db_init_events (void)
 {
     struct event_user * evUser;
 
-    if (!stopSync) {
-        stopSync = epicsMutexMustCreate();
-    }
-
     if (!dbevEventUserFreeList) {
         freeListInitPvt(&dbevEventUserFreeList,
             sizeof(struct event_user),8);
@@ -293,9 +286,6 @@ dbEventCtx db_init_events (void)
         return NULL;
     }
 
-    /* Flag will be cleared when event task starts */
-    evUser->pendexit = TRUE;
-
     evUser->firstque.evUser = evUser;
     evUser->firstque.writelock = epicsMutexCreate();
     if (!evUser->firstque.writelock)
@@ -310,9 +300,6 @@ dbEventCtx db_init_events (void)
     evUser->lock = epicsMutexCreate();
     if (!evUser->lock)
         goto fail;
-    evUser->pexitsem = epicsEventCreate(epicsEventEmpty);
-    if (!evUser->pexitsem)
-        goto fail;
 
     evUser->flowCtrlMode = FALSE;
     evUser->extraLaborBusy = FALSE;
@@ -327,8 +314,6 @@ fail:
         epicsEventDestroy (evUser->ppendsem);
     if(evUser->pflush_sem)
         epicsEventDestroy (evUser->pflush_sem);
-    if(evUser->pexitsem)
-        epicsEventDestroy (evUser->pexitsem);
     freeListFree(dbevEventUserFreeList,evUser);
     return NULL;
 }
@@ -349,7 +334,6 @@ epicsShareFunc void db_cleanup_events(void)
     dbevFieldLogFreeList = NULL;
 }
 
-    /* intentionally leak stopSync to avoid possible shutdown races */
 /*
  *  DB_CLOSE_EVENTS()
  *
@@ -371,30 +355,15 @@ void db_close_events (dbEventCtx ctx)
      * hazardous to the system's health.
      */
     epicsMutexMustLock ( evUser->lock );
-    if(!evUser->pendexit) { /* event task running */
-        evUser->pendexit = TRUE;
-        epicsMutexUnlock ( evUser->lock );
-
-        /* notify the waiting task */
-        epicsEventSignal(evUser->ppendsem);
-        /* wait for task to exit */
-        epicsEventMustWait(evUser->pexitsem);
-
-        epicsMutexMustLock ( evUser->lock );
-    }
-
+    evUser->pendexit = TRUE;
     epicsMutexUnlock ( evUser->lock );
 
-    epicsMutexMustLock (stopSync);
-
-    epicsEventDestroy(evUser->pexitsem);
-    epicsEventDestroy(evUser->ppendsem);
-    epicsEventDestroy(evUser->pflush_sem);
-    epicsMutexDestroy(evUser->lock);
-
-    epicsMutexUnlock (stopSync);
+    /* notify the waiting task */
+    epicsEventSignal(evUser->ppendsem);
 
-    freeListFree(dbevEventUserFreeList, evUser);
+    if(evUser->taskid)
+        epicsThreadMustJoin(evUser->taskid);
+    /* evUser has been deleted by the worker */
 }
 
 /*
@@ -1074,16 +1043,13 @@ static void event_task (void *pParm)
         }
     }
 
-    taskwdRemove(epicsThreadGetIdSelf());
-
-    /* use stopSync to ensure pexitsem is not destroy'd
-     * until epicsEventSignal() has returned.
-     */
-    epicsMutexMustLock (stopSync);
+    epicsEventDestroy(evUser->ppendsem);
+    epicsEventDestroy(evUser->pflush_sem);
+    epicsMutexDestroy(evUser->lock);
 
-    epicsEventSignal(evUser->pexitsem);
+    freeListFree(dbevEventUserFreeList, evUser);
 
-    epicsMutexUnlock(stopSync);
+    taskwdRemove(epicsThreadGetIdSelf());
 
     return;
 }
@@ -1096,6 +1062,11 @@ int db_start_events (
     void *init_func_arg, unsigned osiPriority )
 {
      struct event_user * const evUser = (struct event_user *) ctx;
+     epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+
+     opts.stackSize = epicsThreadGetStackSize(epicsThreadStackMedium);
+     opts.priority = osiPriority;
+     opts.joinable = 1;
 
      epicsMutexMustLock ( evUser->lock );
 
@@ -1113,15 +1084,12 @@ int db_start_events (
      if (!taskname) {
          taskname = EVENT_PEND_NAME;
      }
-     evUser->taskid = epicsThreadCreate (
-         taskname, osiPriority, 
-         epicsThreadGetStackSize(epicsThreadStackMedium),
-         event_task, (void *)evUser);
+     evUser->taskid = epicsThreadCreateOpt (
+         taskname, event_task, (void *)evUser, &opts);
      if (!evUser->taskid) {
          epicsMutexUnlock ( evUser->lock );
          return DB_EVENT_ERROR;
      }
-     evUser->pendexit = FALSE;
      epicsMutexUnlock ( evUser->lock );
      return DB_EVENT_OK;
 }
diff --git a/modules/database/test/ioc/db/dbChArrTest.cpp b/modules/database/test/ioc/db/dbChArrTest.cpp
index 8a788be..8255fdc 100644
--- a/modules/database/test/ioc/db/dbChArrTest.cpp
+++ b/modules/database/test/ioc/db/dbChArrTest.cpp
@@ -36,7 +36,7 @@
 #include "iocInit.h"
 #include "iocsh.h"
 #include "dbChannel.h"
-#include "epicsUnitTest.h"
+#include "dbUnitTest.h"
 #include "testMain.h"
 #include "osiFileName.h"
 
@@ -197,50 +197,33 @@ static void check(short dbr_type) {
     dbChannelDelete(pch);
 }
 
-static dbEventCtx evtctx;
-
-extern "C" {
-static void dbChArrTestCleanup(void* junk)
-{
-    dbFreeBase(pdbbase);
-    registryFree();
-    pdbbase=0;
-
-    db_close_events(evtctx);
-
-    dbmfFreeChunks();
-}
-}
-
 MAIN(dbChArrTest)
 {
     testPlan(102);
 
     /* Prepare the IOC */
+    testdbPrepare();
 
     epicsEnvSet("EPICS_CA_SERVER_PORT", server_port);
 
-    if (dbReadDatabase(&pdbbase, "dbChArrTest.dbd",
-            "." OSI_PATH_LIST_SEPARATOR ".." OSI_PATH_LIST_SEPARATOR
-            "../O.Common" OSI_PATH_LIST_SEPARATOR "O.Common", NULL))
-        testAbort("Database description not loaded");
+    testdbReadDatabase("dbChArrTest.dbd",
+                       "." OSI_PATH_LIST_SEPARATOR ".." OSI_PATH_LIST_SEPARATOR
+                       "../O.Common" OSI_PATH_LIST_SEPARATOR "O.Common", NULL);
 
     dbChArrTest_registerRecordDeviceDriver(pdbbase);
 
-    if (dbReadDatabase(&pdbbase, "dbChArrTest.db",
-            "." OSI_PATH_LIST_SEPARATOR "..", NULL))
-        testAbort("Test database not loaded");
-
-    epicsAtExit(&dbChArrTestCleanup,NULL);
+    testdbReadDatabase("dbChArrTest.db",
+                       "." OSI_PATH_LIST_SEPARATOR "..", NULL);
 
-    /* Start the IOC */
-
-    iocInit();
-    evtctx = db_init_events();
+    testIocInitOk();
 
     check(DBR_LONG);
     check(DBR_DOUBLE);
     check(DBR_STRING);
 
+    testIocShutdownOk();
+
+    testdbCleanup();
+
     return testDone();
 }
diff --git a/modules/libcom/RTEMS/rtems_config.c b/modules/libcom/RTEMS/rtems_config.c
index 147c08b..796b104 100644
--- a/modules/libcom/RTEMS/rtems_config.c
+++ b/modules/libcom/RTEMS/rtems_config.c
@@ -27,6 +27,7 @@
 #endif
 
 #define CONFIGURE_MAXIMUM_TASKS             rtems_resource_unlimited(30)
+#define CONFIGURE_MAXIMUM_BARRIERS          rtems_resource_unlimited(30)
 #define CONFIGURE_MAXIMUM_SEMAPHORES        rtems_resource_unlimited(500)
 #define CONFIGURE_MAXIMUM_TIMERS            rtems_resource_unlimited(20)
 #define CONFIGURE_MAXIMUM_MESSAGE_QUEUES    rtems_resource_unlimited(5)
diff --git a/modules/libcom/src/osi/epicsThread.cpp b/modules/libcom/src/osi/epicsThread.cpp
index 892d73d..92f8338 100644
--- a/modules/libcom/src/osi/epicsThread.cpp
+++ b/modules/libcom/src/osi/epicsThread.cpp
@@ -31,6 +31,18 @@
 
 using namespace std;
 
+epicsThreadId epicsShareAPI epicsThreadCreate (
+    const char * name, unsigned int priority, unsigned int stackSize,
+    EPICSTHREADFUNC funptr,void * parm )
+{
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+    opts.priority = priority;
+    opts.stackSize = stackSize;
+    opts.joinable = 0;
+
+    return epicsThreadCreateOpt(name, funptr, parm, &opts);
+}
+
 epicsThreadRunable::~epicsThreadRunable () {}
 void epicsThreadRunable::run () {}
 void epicsThreadRunable::show ( unsigned int ) const {}
@@ -141,6 +153,13 @@ bool epicsThread::exitWait ( const double delay ) throw ()
             if ( this->pThreadDestroyed ) {
                 *this->pThreadDestroyed = true;
             }
+            if(!joined) {
+                {
+                    epicsGuard < epicsMutex > guard ( this->mutex );
+                    joined = true;
+                }
+                epicsThreadMustJoin(this->id);
+            }
             return true;
         }
         epicsTime exitWaitBegin = epicsTime::getCurrent ();
@@ -154,6 +173,12 @@ bool epicsThread::exitWait ( const double delay ) throw ()
             epicsTime current = epicsTime::getCurrent ();
             exitWaitElapsed = current - exitWaitBegin;
         }
+        if(this->terminated && !joined) {
+            joined = true;
+
+            epicsGuardRelease < epicsMutex > unguard ( guard );
+            epicsThreadMustJoin(this->id);
+        }
     }
     catch ( std :: exception & except ) {
         errlogPrintf (
@@ -177,11 +202,18 @@ epicsThread::epicsThread (
     epicsThreadRunable & runableIn, const char * pName,
         unsigned stackSize, unsigned priority ) :
     runable ( runableIn ), id ( 0 ), pThreadDestroyed ( 0 ),
-    begin ( false ), cancel ( false ), terminated ( false )
+    begin ( false ), cancel ( false ), terminated ( false ),
+    joined ( false )
 {
-    this->id = epicsThreadCreate (
-        pName, priority, stackSize, epicsThreadCallEntryPoint,
-        static_cast < void * > ( this ) );
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+    opts.stackSize = stackSize;
+    opts.priority = priority;
+    opts.joinable = 1;
+
+    this->id = epicsThreadCreateOpt(
+        pName, epicsThreadCallEntryPoint,
+        static_cast < void * > ( this ),
+        &opts);
     if ( ! this->id ) {
         throw unableToCreateThread ();
     }
diff --git a/modules/libcom/src/osi/epicsThread.h b/modules/libcom/src/osi/epicsThread.h
index 84b2c47..da16a0b 100644
--- a/modules/libcom/src/osi/epicsThread.h
+++ b/modules/libcom/src/osi/epicsThread.h
@@ -45,6 +45,7 @@ typedef enum {
     epicsThreadBooleanStatusFail, epicsThreadBooleanStatusSuccess
 } epicsThreadBooleanStatus;
 
+/** Lookup target specific default stack size */
 epicsShareFunc unsigned int epicsShareAPI epicsThreadGetStackSize(
     epicsThreadStackSizeClass size);
 
@@ -54,6 +55,20 @@ typedef struct epicsThreadOSD *epicsThreadId;
 typedef epicsThreadId epicsThreadOnceId;
 #define EPICS_THREAD_ONCE_INIT 0
 
+/** Perform one-time initialization.
+ *
+ * Run the provided function if it has not run, and is not running.
+ *
+ * @post The provided function has been run.
+ *
+ * @code
+ * static epicsThreadOnceId onceId = EPICS_THREAD_ONCE_INIT;
+ * static void myInitFunc(void *arg) { ... }
+ * static void some Function(void) {
+ *     epicsThreadOnce(&onceId, &myInitFunc, NULL);
+ * }
+ * @endcode
+ */
 epicsShareFunc void epicsShareAPI epicsThreadOnce(
     epicsThreadOnceId *id, EPICSTHREADFUNC, void *arg);
 
@@ -65,46 +80,140 @@ epicsShareFunc void epicsThreadRealtimeLock(void);
 
 epicsShareFunc void epicsShareAPI epicsThreadExitMain(void);
 
+/** For use with epicsThreadCreateOpt() */
+typedef struct epicsThreadOpts {
+    /** Thread priority in OSI range (cf. epicsThreadPriority*) */
+    unsigned int priority;
+    /** Thread stack size, either in bytes for this architecture or
+     * an enum epicsThreadStackSizeClass value.
+     */
+    unsigned int stackSize;
+    /** Should thread be joinable? (default (0) is not joinable).
+     * If joinable=1, then epicsThreadMustJoin() must be called for cleanup thread resources.
+     */
+    unsigned int joinable;
+} epicsThreadOpts;
+
+/** Default initial values for epicsThreadOpts
+ * Applications should always use this macro to initialize an epicsThreadOpts
+ * structure. Additional fields may be added in the future, and the order of
+ * the fields might also change, thus code that assumes the above definition
+ * might break if these rules are not followed.
+ */
+#define EPICS_THREAD_OPTS_INIT { \
+    epicsThreadPriorityLow, epicsThreadStackMedium, 0}
+
+/** @brief Allocate and start a new OS thread.
+ * @param name A name describing this thread.  Appears in various log and error message.
+ * @param funptr The thread main function.
+ * @param parm Passed to thread main function.
+ * @param opts Modifiers for the new thread, or NULL to use target specific defaults.
+ * @return NULL on error
+ */
+epicsShareFunc epicsThreadId epicsThreadCreateOpt (
+    const char * name,
+    EPICSTHREADFUNC funptr, void * parm,
+    const epicsThreadOpts *opts );
+/** Short-hand for epicsThreadCreateOpt() to create an un-joinable thread. */
 epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate (
     const char * name, unsigned int priority, unsigned int stackSize,
     EPICSTHREADFUNC funptr,void * parm );
+/** Short-hand for epicsThreadCreateOpt() to create an un-joinable thread.
+ * On error calls cantProceed()
+ */
 epicsShareFunc epicsThreadId epicsShareAPI epicsThreadMustCreate (
     const char * name, unsigned int priority, unsigned int stackSize,
-    EPICSTHREADFUNC funptr,void * parm ); 
+    EPICSTHREADFUNC funptr,void * parm );
+
+/* This gets undefined in osdThread.h on VxWorks < 6.9 */
+#define EPICS_THREAD_CAN_JOIN
+/** Wait for a joinable thread to exit (return from its main function) */
+epicsShareFunc void epicsThreadMustJoin(epicsThreadId id);
+/** Block the current thread until epicsThreadResume(). */
 epicsShareFunc void epicsShareAPI epicsThreadSuspendSelf(void);
+/** Resume a thread suspended with epicsThreadSuspendSelf() */
 epicsShareFunc void epicsShareAPI epicsThreadResume(epicsThreadId id);
+/** Return thread OSI priority */
 epicsShareFunc unsigned int epicsShareAPI epicsThreadGetPriority(
     epicsThreadId id);
+/** Return thread OSI priority */
 epicsShareFunc unsigned int epicsShareAPI epicsThreadGetPrioritySelf(void);
+/** Change OSI priority of target thread. */
 epicsShareFunc void epicsShareAPI epicsThreadSetPriority(
     epicsThreadId id,unsigned int priority);
+/** Lookup the next usage OSI priority such that priority > *pPriorityJustBelow
+ *  if this is possible with the current target configuration and privlages.
+ */
 epicsShareFunc epicsThreadBooleanStatus epicsShareAPI
     epicsThreadHighestPriorityLevelBelow (
         unsigned int priority, unsigned *pPriorityJustBelow);
+/** Lookup the next usage OSI priority such that priority < *pPriorityJustBelow
+ *  if this is possible with the current target configuration and privlages.
+ */
 epicsShareFunc epicsThreadBooleanStatus epicsShareAPI
     epicsThreadLowestPriorityLevelAbove (
         unsigned int priority, unsigned *pPriorityJustAbove);
+/** Test if two thread IDs actually refer to the same OS thread */
 epicsShareFunc int epicsShareAPI epicsThreadIsEqual(
     epicsThreadId id1, epicsThreadId id2);
+/** Test if thread has been suspended with epicsThreadSuspendSelf() */
 epicsShareFunc int epicsShareAPI epicsThreadIsSuspended(epicsThreadId id);
+/** @brief Block the calling thread for at least the specified time.
+ * @param seconds Time to wait in seconds.  Values <=0 blocks for the shortest possible time.
+ */
 epicsShareFunc void epicsShareAPI epicsThreadSleep(double seconds);
+/** @brief Query a value approximating the OS timer/scheduler resolution.
+ * @return A value in seconds >=0
+ *
+ * @warning On targets other than vxWorks and RTEMS, the quantum value often isn't
+ *          meaningful.  Use of this function is discouraged in portable code.
+ */
 epicsShareFunc double epicsShareAPI epicsThreadSleepQuantum(void);
+/** Find an epicsThreadId associated with the current thread.
+ * For non-EPICS threads, a new epicsThreadId may be allocated.
+ */
 epicsShareFunc epicsThreadId epicsShareAPI epicsThreadGetIdSelf(void);
+/** Attempt to find the first instance of a thread by name.
+ * @return An epicsThreadId, or NULL if no such thread is currently running.
+ *         Note that a non-NULL ID may still be invalid if this call races
+ *         with thread exit.
+ *
+ * @warning Safe use of this function requires external knowledge that this
+ *          thread will not return.
+ */
 epicsShareFunc epicsThreadId epicsShareAPI epicsThreadGetId(const char *name);
+/** Return a value approximating the number of threads which this target
+ *  can run in parallel.  This value is advisory.
+ * @return >=1
+ */
 epicsShareFunc int epicsThreadGetCPUs(void);
 
+/** Return the name of the current thread.
+ *
+ * @return Never NULL.  Storage lifetime tied to epicsThreadId.
+ *
+ * This is either a copy of the string passed to epicsThread*Create*(),
+ * or an arbitrary unique string for non-EPICS threads.
+ */
 epicsShareFunc const char * epicsShareAPI epicsThreadGetNameSelf(void);
 
-/* For epicsThreadGetName name is guaranteed to be null terminated */
-/* size is size of buffer to hold name (including terminator) */
-/* Failure results in an empty string stored in name */
+/** Copy out the thread name into the provided buffer.
+ *
+ * Guaranteed to be null terminated.
+ * size is number of bytes in buffer to hold name (including terminator).
+ * Failure results in an empty string stored in name.
+ */
 epicsShareFunc void epicsShareAPI epicsThreadGetName(
     epicsThreadId id, char *name, size_t size);
 
 epicsShareFunc int epicsShareAPI epicsThreadIsOkToBlock(void);
 epicsShareFunc void epicsShareAPI epicsThreadSetOkToBlock(int isOkToBlock);
 
+/** Print to stdout information about all running EPICS threads.
+ * @param level 0 prints minimal output.  Higher values print more details.
+ */
 epicsShareFunc void epicsShareAPI epicsThreadShowAll(unsigned int level);
+/** Print info about a single EPICS thread. */
 epicsShareFunc void epicsShareAPI epicsThreadShow(
     epicsThreadId id,unsigned int level);
 
@@ -115,10 +224,17 @@ epicsShareFunc int epicsThreadHookDelete(EPICS_THREAD_HOOK_ROUTINE hook);
 epicsShareFunc void epicsThreadHooksShow(void);
 epicsShareFunc void epicsThreadMap(EPICS_THREAD_HOOK_ROUTINE func);
 
+/** Thread local storage */
 typedef struct epicsThreadPrivateOSD * epicsThreadPrivateId;
+/** Allocate a new thread local variable.
+ * This variable will initially hold NULL for each thread.
+ */
 epicsShareFunc epicsThreadPrivateId epicsShareAPI epicsThreadPrivateCreate(void);
+/** Free a thread local variable */
 epicsShareFunc void epicsShareAPI epicsThreadPrivateDelete(epicsThreadPrivateId id);
+/** Update thread local variable */
 epicsShareFunc void epicsShareAPI epicsThreadPrivateSet(epicsThreadPrivateId,void *);
+/** Fetch the current value of a thread local variable */
 epicsShareFunc void * epicsShareAPI epicsThreadPrivateGet(epicsThreadPrivateId);
 
 #ifdef __cplusplus
@@ -130,33 +246,64 @@ epicsShareFunc void * epicsShareAPI epicsThreadPrivateGet(epicsThreadPrivateId);
 #include "epicsEvent.h"
 #include "epicsMutex.h"
 
+//! Interface used with class epicsThread
 class epicsShareClass epicsThreadRunable {
 public:
     virtual ~epicsThreadRunable () = 0;
+    //! Thread main function.
+    //! C++ exceptions which propagate from this method will be caught and a warning printed.
+    //! No other action is taken.
     virtual void run () = 0;
+    //! Optional.  Called via epicsThread::show()
     virtual void show ( unsigned int level ) const;
 };
 
 extern "C" void epicsThreadCallEntryPoint ( void * );
 
+/** @brief An OS thread
+ *
+ * A wrapper around the epicsThread* C API.
+ *
+ * @note Threads must be start() ed.
+ */
 class epicsShareClass epicsThread {
 public:
+    /** Create a new thread with the provided information.
+     *
+     * cf. epicsThreadOpts
+     * @note Threads must be start() ed.
+     * @throws epicsThread::unableToCreateThread on error.
+     */
     epicsThread ( epicsThreadRunable &,const char *name, unsigned int stackSize,
         unsigned int priority=epicsThreadPriorityLow );
     ~epicsThread () throw ();
+    //! Actually start the thread.
     void start () throw ();
+    //! Wait for the thread epicsRunnable::run() to return.
     void exitWait () throw ();
+    //! Wait for the thread epicsRunnable::run() to return.
+    //! @param delay Wait up to this many seconds.
+    //! @returns true if run() returned.  false on timeout.
     bool exitWait ( const double delay ) throw (); 
+    //! @throws A special exitException which will be caught and ignored.
+    //! @note This exitException doesn't not derive from std::exception
     static void exit ();
+    //! cf. epicsThreadResume()
     void resume () throw ();
+    //! cf. epicsThreadGetName();
     void getName ( char * name, size_t size ) const throw ();
+    //! cf. epicsThreadGetIdSelf()()
     epicsThreadId getId () const throw ();
+    //! cf. epicsThreadGetPriority()
     unsigned int getPriority () const throw ();
+    //! cf. epicsThreadSetPriority()
     void setPriority ( unsigned int ) throw ();
     bool priorityIsEqual ( const epicsThread & ) const throw ();
     bool isSuspended () const throw ();
+    //! @return true if call through this thread's epicsRunnable::run()
     bool isCurrentThread () const throw ();
     bool operator == ( const epicsThread & ) const throw ();
+    //! Say something interesting about this thread to stdout.
     void show ( unsigned level ) const throw ();
 
     /* these operate on the current thread */
@@ -178,6 +325,7 @@ private:
     bool begin;
     bool cancel;
     bool terminated;
+    bool joined;
 
     bool beginWait () throw ();
     epicsThread ( const epicsThread & );
diff --git a/modules/libcom/src/osi/os/Linux/osdThread.h b/modules/libcom/src/osi/os/Linux/osdThread.h
index 7d2a486..bb1fdcb 100644
--- a/modules/libcom/src/osi/os/Linux/osdThread.h
+++ b/modules/libcom/src/osi/os/Linux/osdThread.h
@@ -22,6 +22,7 @@ extern "C" {
 
 typedef struct epicsThreadOSD {
     ELLNODE            node;
+    int                refcnt;
     pthread_t          tid;
     pid_t              lwpId;
     pthread_attr_t     attr;
@@ -35,6 +36,7 @@ typedef struct epicsThreadOSD {
     int                isRealTimeScheduled;
     int                isOnThreadList;
     unsigned int       osiPriority;
+    int                joinable;
     char               name[1];     /* actually larger */
 } epicsThreadOSD;
 
diff --git a/modules/libcom/src/osi/os/RTEMS/osdThread.c b/modules/libcom/src/osi/os/RTEMS/osdThread.c
index 769e958..bdcd8c1 100644
--- a/modules/libcom/src/osi/os/RTEMS/osdThread.c
+++ b/modules/libcom/src/osi/os/RTEMS/osdThread.c
@@ -35,6 +35,7 @@
 #include "osiUnistd.h"
 #include "osdInterrupt.h"
 #include "epicsExit.h"
+#include "epicsAtomic.h"
 
 epicsShareFunc void osdThreadHooksRun(epicsThreadId id);
 epicsShareFunc void osdThreadHooksRunMain(epicsThreadId id);
@@ -47,6 +48,9 @@ struct taskVar {
     struct taskVar  *back;
     char            *name;
     rtems_id             id;
+    rtems_id             join_barrier; /* only valid if joinable */
+    int                refcnt;
+    int                joinable;
     EPICSTHREADFUNC              funptr;
     void                *parm;
     unsigned int        threadVariableCapacity;
@@ -163,6 +167,22 @@ taskVarUnlock (void)
     epicsMutexOsdUnlock (taskVarMutex);
 }
 
+static
+void taskUnref(struct taskVar *v)
+{
+    int ref = epicsAtomicDecrIntT(&v->refcnt);
+    assert(ref>=0);
+    if(ref>0) return;
+
+
+    if (v->joinable) {
+        rtems_barrier_delete(v->join_barrier);
+    }
+    free (v->threadVariables);
+    free (v->name);
+    free (v);
+}
+
 /*
  * EPICS threads destroy themselves by returning from the thread entry function.
  * This simple wrapper provides the same semantics on RTEMS.
@@ -183,9 +203,12 @@ threadWrapper (rtems_task_argument arg)
     if (v->forw)
         v->forw->back = v->back;
     taskVarUnlock ();
-    free (v->threadVariables);
-    free (v->name);
-    free (v);
+    if(v->joinable) {
+        rtems_status_code sc = rtems_barrier_wait(v->join_barrier, RTEMS_NO_TIMEOUT);
+        if(sc!=RTEMS_SUCCESSFUL)
+            cantProceed("oops %s\n", rtems_status_text(sc));
+    }
+    taskUnref(v);
     rtems_task_delete (RTEMS_SELF);
 }
 
@@ -196,21 +219,34 @@ void epicsThreadExitMain (void)
 {
 }
 
-static void
+static rtems_status_code
 setThreadInfo(rtems_id tid, const char *name, EPICSTHREADFUNC funptr,
-    void *parm)
+    void *parm, int joinable)
 {
     struct taskVar *v;
     uint32_t note;
-    rtems_status_code sc;
+    rtems_status_code sc = RTEMS_SUCCESSFUL;
 
     v = mallocMustSucceed (sizeof *v, "epicsThreadCreate_vars");
     v->name = epicsStrDup(name);
     v->id = tid;
     v->funptr = funptr;
     v->parm = parm;
+    v->joinable = joinable;
+    v->refcnt = joinable ? 2 : 1;
     v->threadVariableCapacity = 0;
     v->threadVariables = NULL;
+    if (joinable) {
+        char c[3];
+        strncpy(c, v->name, 3);
+        sc = rtems_barrier_create(rtems_build_name('~', c[0], c[1], c[2]),
+                RTEMS_BARRIER_AUTOMATIC_RELEASE | RTEMS_LOCAL,
+                2, &v->join_barrier);
+        if (sc != RTEMS_SUCCESSFUL) {
+            free(v);
+            return sc;
+        }
+    }
     note = (uint32_t)v;
     rtems_task_set_note (tid, RTEMS_NOTEPAD_TASKVAR, note);
     taskVarLock ();
@@ -222,10 +258,14 @@ setThreadInfo(rtems_id tid, const char *name, EPICSTHREADFUNC funptr,
     taskVarUnlock ();
     if (funptr) {
         sc = rtems_task_start (tid, threadWrapper, (rtems_task_argument)v);
-        if (sc !=  RTEMS_SUCCESSFUL)
-            errlogPrintf ("setThreadInfo:  Can't start  %s: %s\n",
-                name, rtems_status_text(sc));
     }
+    if (sc != RTEMS_SUCCESSFUL) {
+        if (joinable) {
+            rtems_barrier_delete(v->join_barrier);
+        }
+        free(v);
+    }
+    return sc;
 }
 
 /*
@@ -247,7 +287,8 @@ epicsThreadInit (void)
         if (!onceMutex || !taskVarMutex)
             cantProceed("epicsThreadInit() can't create global mutexes\n");
         rtems_task_ident (RTEMS_SELF, 0, &tid);
-        setThreadInfo (tid, "_main_", NULL, NULL);
+        if(setThreadInfo (tid, "_main_", NULL, NULL, 0) != RTEMS_SUCCESSFUL)
+            cantProceed("epicsThreadInit() unable to setup _main_");
         osdThreadHooksRunMain((epicsThreadId)tid);
         initialized = 1;
         epicsThreadCreate ("ImsgDaemon", 99,
@@ -263,15 +304,26 @@ void epicsThreadRealtimeLock(void)
  * Create and start a new thread
  */
 epicsThreadId
-epicsThreadCreate (const char *name,
-    unsigned int priority, unsigned int stackSize,
-    EPICSTHREADFUNC funptr,void *parm)
+epicsThreadCreateOpt (
+    const char * name,
+    EPICSTHREADFUNC funptr, void * parm, const epicsThreadOpts *opts )
 {
+    unsigned int stackSize;
     rtems_id tid;
     rtems_status_code sc;
     char c[4];
 
-    if (!initialized) epicsThreadInit();
+    if (!initialized)
+        epicsThreadInit();
+
+    if (!opts) {
+        static const epicsThreadOpts opts_default = EPICS_THREAD_OPTS_INIT;
+        opts = &opts_default;
+    }
+    stackSize = opts->stackSize;
+    if (stackSize <= epicsThreadStackBig)
+        stackSize = epicsThreadGetStackSize(stackSize);
+
     if (stackSize < RTEMS_MINIMUM_STACK_SIZE) {
         errlogPrintf ("Warning: epicsThreadCreate %s illegal stackSize %d\n",
             name, stackSize);
@@ -279,7 +331,7 @@ epicsThreadCreate (const char *name,
     }
     strncpy (c, name, sizeof c);
     sc = rtems_task_create (rtems_build_name (c[0], c[1], c[2], c[3]),
-         epicsThreadGetOssPriorityValue (priority),
+         epicsThreadGetOssPriorityValue (opts->priority),
          stackSize,
          RTEMS_PREEMPT|RTEMS_NO_TIMESLICE|RTEMS_NO_ASR|RTEMS_INTERRUPT_LEVEL(0),
          RTEMS_FLOATING_POINT|RTEMS_LOCAL,
@@ -289,7 +341,13 @@ epicsThreadCreate (const char *name,
             name, rtems_status_text(sc));
         return 0;
     }
-    setThreadInfo (tid, name, funptr,parm);
+    sc = setThreadInfo (tid, name, funptr, parm, opts->joinable);
+    if (sc != RTEMS_SUCCESSFUL) {
+        errlogPrintf ("epicsThreadCreate create failure during setup for %s: %s\n",
+            name, rtems_status_text(sc));
+        rtems_task_delete(tid);
+        return 0;
+    }
     return (epicsThreadId)tid;
 }
 
@@ -305,6 +363,51 @@ threadMustCreate (const char *name,
     return tid;
 }
 
+void epicsThreadMustJoin(epicsThreadId id)
+{
+    rtems_id target_tid = (rtems_id)id, self_tid;
+    struct taskVar *v = 0;
+
+    rtems_task_ident (RTEMS_SELF, 0, &self_tid);
+
+    {
+        uint32_t note;
+        rtems_task_get_note (target_tid, RTEMS_NOTEPAD_TASKVAR, &note);
+        v = (void *)note;
+    }
+    /* 'v' may be NULL if 'id' represents a non-EPICS thread other than _main_. */
+
+    if(!v || !v->joinable) {
+        if(epicsThreadGetIdSelf()==id) {
+            errlogPrintf("Warning: %s thread self-join of unjoinable\n", v ? v->name : "non-EPICS thread");
+
+        } else {
+            /* try to error nicely, however in all likelyhood de-ref of
+             * 'id' has already caused SIGSEGV as we are racing thread exit,
+             * which free's 'id'.
+             */
+            cantProceed("Error: %s thread not joinable.\n", v->name);
+        }
+        return;
+
+    } else if(target_tid!=self_tid) {
+        /* wait for target to complete */
+        rtems_status_code sc = rtems_barrier_wait(v->join_barrier, RTEMS_NO_TIMEOUT);
+        if(sc!=RTEMS_SUCCESSFUL)
+            cantProceed("oopsj %s\n", rtems_status_text(sc));
+
+        if(sc != RTEMS_SUCCESSFUL) {
+            errlogPrintf("epicsThreadMustJoin('%s') -> %s\n", v->name, rtems_status_text(sc));
+        }
+    }
+
+    v->joinable = 0;
+    taskUnref(v);
+    /* target task may be deleted.
+     * self task is not deleted, even for self join.
+     */
+}
+
 void
 epicsThreadSuspendSelf (void)
 {
diff --git a/modules/libcom/src/osi/os/RTEMS/osdThread.h b/modules/libcom/src/osi/os/RTEMS/osdThread.h
index 4451f84..4eef8c0 100644
--- a/modules/libcom/src/osi/os/RTEMS/osdThread.h
+++ b/modules/libcom/src/osi/os/RTEMS/osdThread.h
@@ -3,10 +3,21 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
-* EPICS BASE Versions 3.13.7
-* and higher are distributed subject to a Software License Agreement found
-* in file LICENSE that is included with this distribution. 
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
 \*************************************************************************/
 
+#ifndef INC_osdThread_H
+#define INC_osdThread_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
 int epicsThreadGetOssPriorityValue(unsigned int osiPriority);
 
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* INC_osdThread_H */
diff --git a/modules/libcom/src/osi/os/WIN32/osdThread.c b/modules/libcom/src/osi/os/WIN32/osdThread.c
index 8cdb4a3..1cfa1ec 100644
--- a/modules/libcom/src/osi/os/WIN32/osdThread.c
+++ b/modules/libcom/src/osi/os/WIN32/osdThread.c
@@ -32,6 +32,7 @@
 #include "epicsAssert.h"
 #include "ellLib.h"
 #include "epicsExit.h"
+#include "epicsAtomic.h"
 
 epicsShareFunc void osdThreadHooksRun(epicsThreadId id);
 
@@ -46,6 +47,7 @@ typedef struct win32ThreadGlobal {
 
 typedef struct epicsThreadOSD {
     ELLNODE node;
+    int refcnt;
     HANDLE handle;
     EPICSTHREADFUNC funptr;
     void * parm;
@@ -53,6 +55,7 @@ typedef struct epicsThreadOSD {
     DWORD id;
     unsigned epicsPriority;
     char isSuspended;
+    char joinable;
 } win32ThreadParam;
 
 typedef struct epicsThreadPrivateOSD {
@@ -238,6 +241,8 @@ static void epicsParmCleanupWIN32 ( win32ThreadParam * pParm )
     }
 
     if ( pParm ) {
+        if(epicsAtomicDecrIntT(&pParm->refcnt) > 0) return;
+
         /* fprintf ( stderr, "thread %s is exiting\n", pParm->pName ); */
         EnterCriticalSection ( & pGbl->mutex );
         ellDelete ( & pGbl->threadList, & pParm->node );
@@ -526,6 +531,7 @@ static win32ThreadParam * epicsThreadParmCreate ( const char *pName )
         pParmWIN32->pName = (char *) ( pParmWIN32 + 1 );
         strcpy ( pParmWIN32->pName, pName );
         pParmWIN32->isSuspended = 0;
+        epicsAtomicIncrIntT(&pParmWIN32->refcnt);
     }
     return pParmWIN32;
 }
@@ -579,11 +585,14 @@ static win32ThreadParam * epicsThreadImplicitCreate ( void )
 /*
  * epicsThreadCreate ()
  */
-epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate (const char *pName,
-    unsigned int priority, unsigned int stackSize, EPICSTHREADFUNC pFunc,void *pParm)
+epicsThreadId epicsThreadCreateOpt (
+    const char * pName,
+    EPICSTHREADFUNC pFunc, void * pParm,
+    const epicsThreadOpts *opts )
 {
     win32ThreadGlobal * pGbl = fetchWin32ThreadGlobal ();
     win32ThreadParam * pParmWIN32;
+    unsigned int stackSize;
     int osdPriority;
     DWORD wstat;
     BOOL bstat;
@@ -592,18 +601,26 @@ epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate (const char *pName,
         return NULL;
     }
 
+    if (!opts) {
+        static const epicsThreadOpts opts_default = EPICS_THREAD_OPTS_INIT;
+        opts = &opts_default;
+    }
+    stackSize = opts->stackSize;
+    if (stackSize <= epicsThreadStackBig)
+        stackSize = epicsThreadGetStackSize(stackSize);
+
     pParmWIN32 = epicsThreadParmCreate ( pName );
     if ( pParmWIN32 == 0 ) {
         return ( epicsThreadId ) pParmWIN32;
     }
     pParmWIN32->funptr = pFunc;
     pParmWIN32->parm = pParm;
-    pParmWIN32->epicsPriority = priority;
+    pParmWIN32->epicsPriority = opts->priority;
 
     {
         unsigned threadId;
         pParmWIN32->handle = (HANDLE) _beginthreadex ( 
-            0, stackSize, epicsWin32ThreadEntry, 
+            0, stackSize, epicsWin32ThreadEntry,
             pParmWIN32, 
             CREATE_SUSPENDED | STACK_SIZE_PARAM_IS_A_RESERVATION, 
             & threadId );
@@ -615,7 +632,7 @@ epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate (const char *pName,
         pParmWIN32->id = ( DWORD ) threadId ;
     }
 
-    osdPriority = epicsThreadGetOsdPriorityValue (priority);
+    osdPriority = epicsThreadGetOsdPriorityValue (opts->priority);
     bstat = SetThreadPriority ( pParmWIN32->handle, osdPriority );
     if (!bstat) {
         CloseHandle ( pParmWIN32->handle ); 
@@ -637,9 +654,48 @@ epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate (const char *pName,
         return NULL;
     }
 
+    if(opts->joinable) {
+        pParmWIN32->joinable = 1;
+        epicsAtomicIncrIntT(&pParmWIN32->refcnt);
+    }
+
     return ( epicsThreadId ) pParmWIN32;
 }
 
+void epicsThreadMustJoin(epicsThreadId id)
+{
+    win32ThreadParam * pParmWIN32 = id;
+
+    if(!id) {
+        /* no-op */
+    } else if(!pParmWIN32->joinable) {
+        if(epicsThreadGetIdSelf()==id) {
+            fprintf(stderr, "Warning: %s thread self-join of unjoinable\n", pParmWIN32->pName);
+
+        } else {
+            /* try to error nicely, however in all likelyhood de-ref of
+             * 'id' has already caused SIGSEGV as we are racing thread exit,
+             * which free's 'id'.
+             */
+            cantProceed("Error: %s thread not joinable.\n", pParmWIN32->pName);
+        }
+        return;
+
+    } else if(epicsThreadGetIdSelf() != id) {
+        DWORD status = WaitForSingleObject(pParmWIN32->handle, INFINITE);
+        if(status != WAIT_OBJECT_0) {
+            /* TODO: signal error? */
+        }
+
+        pParmWIN32->joinable = 0;
+        epicsParmCleanupWIN32(pParmWIN32);
+    } else {
+        /* join self silently does nothing */
+        pParmWIN32->joinable = 0;
+        epicsParmCleanupWIN32(pParmWIN32);
+    }
+}
+
 /*
  * epicsThreadSuspendSelf ()
  */
diff --git a/modules/libcom/src/osi/os/WIN32/osdThread.h b/modules/libcom/src/osi/os/WIN32/osdThread.h
index 136e96b..69bc364 100644
--- a/modules/libcom/src/osi/os/WIN32/osdThread.h
+++ b/modules/libcom/src/osi/os/WIN32/osdThread.h
@@ -3,13 +3,11 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
-* EPICS BASE Versions 3.13.7
-* and higher are distributed subject to a Software License Agreement found
-* in file LICENSE that is included with this distribution. 
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
 \*************************************************************************/
 
 #ifndef osdThreadh
 #define osdThreadh
 
- 
 #endif /* osdThreadh */
diff --git a/modules/libcom/src/osi/os/posix/osdThread.c b/modules/libcom/src/osi/os/posix/osdThread.c
index e3e0fe3..a13d387 100644
--- a/modules/libcom/src/osi/os/posix/osdThread.c
+++ b/modules/libcom/src/osi/os/posix/osdThread.c
@@ -38,6 +38,7 @@
 #include "errlog.h"
 #include "epicsAssert.h"
 #include "epicsExit.h"
+#include "epicsAtomic.h"
 
 epicsShareFunc void epicsThreadShowInfo(epicsThreadOSD *pthreadInfo, unsigned int level);
 epicsShareFunc void osdThreadHooksRun(epicsThreadId id);
@@ -167,12 +168,14 @@ static epicsThreadOSD * create_threadInfo(const char *name)
         return NULL;
     }
     strcpy(pthreadInfo->name, name);
+    epicsAtomicIncrIntT(&pthreadInfo->refcnt); /* initial ref for the thread itself */
     return pthreadInfo;
 }
 
 static epicsThreadOSD * init_threadInfo(const char *name,
     unsigned int priority, unsigned int stackSize,
-    EPICSTHREADFUNC funptr,void *parm)
+    EPICSTHREADFUNC funptr,void *parm,
+    unsigned joinable)
 {
     epicsThreadOSD *pthreadInfo;
     int status;
@@ -182,12 +185,15 @@ static epicsThreadOSD * init_threadInfo(const char *name,
         return NULL;
     pthreadInfo->createFunc = funptr;
     pthreadInfo->createArg = parm;
+    pthreadInfo->joinable = joinable;
     status = pthread_attr_init(&pthreadInfo->attr);
     checkStatusOnce(status,"pthread_attr_init");
     if(status) return 0;
-    status = pthread_attr_setdetachstate(
-        &pthreadInfo->attr, PTHREAD_CREATE_DETACHED);
-    checkStatusOnce(status,"pthread_attr_setdetachstate");
+    if(!joinable){
+        status = pthread_attr_setdetachstate(
+            &pthreadInfo->attr, PTHREAD_CREATE_DETACHED);
+        checkStatusOnce(status,"pthread_attr_setdetachstate");
+    }
 #if defined (_POSIX_THREAD_ATTR_STACKSIZE)
 #if ! defined (OSITHREAD_USE_DEFAULT_STACK)
     status = pthread_attr_setstacksize( &pthreadInfo->attr,(size_t)stackSize);
@@ -204,6 +210,8 @@ static void free_threadInfo(epicsThreadOSD *pthreadInfo)
 {
     int status;
 
+    if(epicsAtomicDecrIntT(&pthreadInfo->refcnt) > 0) return;
+
     status = mutexLock(&listLock);
     checkStatusQuit(status,"pthread_mutex_lock","free_threadInfo");
     if(pthreadInfo->isOnThreadList) ellDelete(&pthreadList,&pthreadInfo->node);
@@ -366,7 +374,7 @@ static void once(void)
     if(errVerbose) fprintf(stderr,"task priorities are not implemented\n");
 #endif /* _POSIX_THREAD_PRIORITY_SCHEDULING */
 
-    pthreadInfo = init_threadInfo("_main_",0,epicsThreadGetStackSize(epicsThreadStackSmall),0,0);
+    pthreadInfo = init_threadInfo("_main_",0,epicsThreadGetStackSize(epicsThreadStackSmall),0,0,0);
     assert(pthreadInfo!=NULL);
     status = pthread_setspecific(getpthreadInfo,(void *)pthreadInfo);
     checkStatusOnceQuit(status,"pthread_setspecific","epicsThreadInit");
@@ -445,15 +453,22 @@ void epicsThreadRealtimeLock(void)
 #endif
 }
 
-epicsShareFunc unsigned int epicsShareAPI epicsThreadGetStackSize (epicsThreadStackSizeClass stackSizeClass)
-{
 #if defined (OSITHREAD_USE_DEFAULT_STACK)
-    return 0;
+#define STACK_SIZE(f) (0)
 #elif defined(_POSIX_THREAD_ATTR_STACKSIZE) && _POSIX_THREAD_ATTR_STACKSIZE > 0
     #define STACK_SIZE(f) (f * 0x10000 * sizeof(void *))
     static const unsigned stackSizeTable[epicsThreadStackBig+1] = {
         STACK_SIZE(1), STACK_SIZE(2), STACK_SIZE(4)
     };
+#else
+#define STACK_SIZE(f) (0)
+#endif /*_POSIX_THREAD_ATTR_STACKSIZE*/
+
+epicsShareFunc unsigned int epicsShareAPI epicsThreadGetStackSize (epicsThreadStackSizeClass stackSizeClass)
+{
+#if defined (OSITHREAD_USE_DEFAULT_STACK)
+    return 0;
+#elif defined(_POSIX_THREAD_ATTR_STACKSIZE) && _POSIX_THREAD_ATTR_STACKSIZE > 0
     if (stackSizeClass<epicsThreadStackSmall) {
         errlogPrintf("epicsThreadGetStackSize illegal argument (too small)");
         return stackSizeTable[epicsThreadStackBig];
@@ -511,42 +526,66 @@ epicsShareFunc void epicsShareAPI epicsThreadOnce(epicsThreadOnceId *id, void (*
     checkStatusQuit(status,"pthread_mutex_unlock","epicsThreadOnce");
 }
 
-epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate(const char *name,
-    unsigned int priority, unsigned int stackSize,
-    EPICSTHREADFUNC funptr,void *parm)
+epicsThreadId
+epicsThreadCreateOpt(const char * name,
+    EPICSTHREADFUNC funptr, void * parm, const epicsThreadOpts *opts )
 {
+    unsigned int stackSize;
     epicsThreadOSD *pthreadInfo;
     int status;
     sigset_t blockAllSig, oldSig;
 
     epicsThreadInit();
     assert(pcommonAttr);
+
+    if (!opts) {
+        static const epicsThreadOpts opts_default = EPICS_THREAD_OPTS_INIT;
+        opts = &opts_default;
+    }
+    stackSize = opts->stackSize;
+    if (stackSize <= epicsThreadStackBig)
+        stackSize = epicsThreadGetStackSize(stackSize);
+
     sigfillset(&blockAllSig);
-    pthread_sigmask(SIG_SETMASK,&blockAllSig,&oldSig);
-    pthreadInfo = init_threadInfo(name,priority,stackSize,funptr,parm);
-    if(pthreadInfo==0) return 0;
+    pthread_sigmask(SIG_SETMASK, &blockAllSig, &oldSig);
+
+    pthreadInfo = init_threadInfo(name, opts->priority, stackSize, funptr,
+        parm, opts->joinable);
+    if (pthreadInfo==0)
+        return 0;
+
     pthreadInfo->isEpicsThread = 1;
-    setSchedulingPolicy(pthreadInfo,SCHED_FIFO);
+    setSchedulingPolicy(pthreadInfo, SCHED_FIFO);
     pthreadInfo->isRealTimeScheduled = 1;
-    status = pthread_create(&pthreadInfo->tid,&pthreadInfo->attr,
-                start_routine,pthreadInfo);
-    if(status==EPERM){
+
+    status = pthread_create(&pthreadInfo->tid, &pthreadInfo->attr,
+        start_routine, pthreadInfo);
+    if (status==EPERM) {
         /* Try again without SCHED_FIFO*/
         free_threadInfo(pthreadInfo);
-        pthreadInfo = init_threadInfo(name,priority,stackSize,funptr,parm);
-        if(pthreadInfo==0) return 0;
+
+        pthreadInfo = init_threadInfo(name, opts->priority, stackSize,
+            funptr, parm, opts->joinable);
+        if (pthreadInfo==0)
+            return 0;
+
         pthreadInfo->isEpicsThread = 1;
-        status = pthread_create(&pthreadInfo->tid,&pthreadInfo->attr,
-                start_routine,pthreadInfo);
+        status = pthread_create(&pthreadInfo->tid, &pthreadInfo->attr,
+            start_routine, pthreadInfo);
     }
-    checkStatusOnce(status,"pthread_create");
-    if(status) {
+    checkStatusOnce(status, "pthread_create");
+    if (status) {
         free_threadInfo(pthreadInfo);
         return 0;
     }
-    status = pthread_sigmask(SIG_SETMASK,&oldSig,NULL);
-    checkStatusOnce(status,"pthread_sigmask");
-    return(pthreadInfo);
+
+    status = pthread_sigmask(SIG_SETMASK, &oldSig, NULL);
+    checkStatusOnce(status, "pthread_sigmask");
+    if (pthreadInfo->joinable) {
+        /* extra ref for epicsThreadMustJoin() */
+        epicsAtomicIncrIntT(&pthreadInfo->refcnt);
+    }
+    return pthreadInfo;
 }
 
 /*
@@ -584,7 +623,40 @@ static epicsThreadOSD *createImplicit(void)
     }
     return pthreadInfo;
 }
-

+
+void epicsThreadMustJoin(epicsThreadId id)
+{
+    void *ret = NULL;
+    int status;
+
+    if(!id) {
+        return;
+    } else if(!id->joinable) {
+        if(epicsThreadGetIdSelf()==id) {
+            errlogPrintf("Warning: %s thread self-join of unjoinable\n", id->name);
+
+        } else {
+            /* try to error nicely, however in all likelyhood de-ref of
+             * 'id' has already caused SIGSEGV as we are racing thread exit,
+             * which free's 'id'.
+             */
+            cantProceed("Error: %s thread not joinable.\n", id->name);
+        }
+        return;
+    }
+
+    status = pthread_join(id->tid, &ret);
+    if(status == EDEADLK) {
+        /* Thread can't join itself (directly or indirectly)
+         * so we detach instead.
+         */
+        status = pthread_detach(id->tid);
+        checkStatusOnce(status, "pthread_detach");
+    } else checkStatusOnce(status, "pthread_join");
+    id->joinable = 0;
+    free_threadInfo(id);
+}
+
 epicsShareFunc void epicsShareAPI epicsThreadSuspendSelf(void)
 {
     epicsThreadOSD *pthreadInfo;
diff --git a/modules/libcom/src/osi/os/posix/osdThread.h b/modules/libcom/src/osi/os/posix/osdThread.h
index 3a80b53..8fe8f14 100644
--- a/modules/libcom/src/osi/os/posix/osdThread.h
+++ b/modules/libcom/src/osi/os/posix/osdThread.h
@@ -3,9 +3,8 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
-* EPICS BASE Versions 3.13.7
-* and higher are distributed subject to a Software License Agreement found
-* in file LICENSE that is included with this distribution. 
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
 \*************************************************************************/
 #ifndef osdThreadh
 #define osdThreadh
@@ -22,6 +21,7 @@ extern "C" {
 
 typedef struct epicsThreadOSD {
     ELLNODE            node;
+    int                refcnt;
     pthread_t          tid;
     pthread_attr_t     attr;
     struct sched_param schedParam;
@@ -34,6 +34,7 @@ typedef struct epicsThreadOSD {
     int                isRealTimeScheduled;
     int                isOnThreadList;
     unsigned int       osiPriority;
+    int                joinable;
     char               name[1];     /* actually larger */
 } epicsThreadOSD;
 
diff --git a/modules/libcom/src/osi/os/vxWorks/osdThread.c b/modules/libcom/src/osi/os/vxWorks/osdThread.c
index ce01ea6..0ed3138 100644
--- a/modules/libcom/src/osi/os/vxWorks/osdThread.c
+++ b/modules/libcom/src/osi/os/vxWorks/osdThread.c
@@ -34,6 +34,22 @@
 #include "vxLib.h"
 #include "epicsExit.h"
 
+#ifdef EPICS_THREAD_CAN_JOIN
+  /* The implementation of epicsThreadMustJoin() here uses 2 features
+   * of VxWorks that were first introduced in VxWorks 6.9: taskWait(),
+   * and the taskSpareFieldGet/Set routines in taskUtilLib.
+   */
+  #include <taskUtilLib.h>
+
+  #define JOIN_WARNING_TIMEOUT (60 * sysClkRateGet())
+
+  static SPARE_NUM joinField;
+  #define ALLOT_JOIN(tid) taskSpareNumAllot(tid, &joinField)
+#else
+  #define ALLOT_JOIN(tid)
+#endif
+
+
 epicsShareFunc void osdThreadHooksRun(epicsThreadId id);
 
 #if CPU_FAMILY == MC680X0
@@ -109,6 +125,7 @@ static void epicsThreadInit(void)
         assert(taskIdList);
         taskIdListSize = ID_LIST_CHUNK;
         atRebootRegister();
+        ALLOT_JOIN(0);
         done = 1;
     }
     lock = 0;
@@ -170,19 +187,58 @@ void epicsThreadOnce(epicsThreadOnceId *id, void (*func)(void *), void *arg)
     }
     semGive(epicsThreadOnceMutex);
 }
-

+
+#ifdef EPICS_THREAD_CAN_JOIN
+
+/* This routine is not static so it appears in the back-trace
+ * of a thread that is waiting to be joined.
+ */
+void epicsThreadAwaitingJoin(int tid)
+{
+    SEM_ID joinSem = (SEM_ID) taskSpareFieldGet(tid, joinField);
+    STATUS status;
+
+    if (!joinSem || (int) joinSem == ERROR)
+        return;
+
+    /* Wait for our supervisor */
+    status = semTake(joinSem, JOIN_WARNING_TIMEOUT);
+    if (status && errno == S_objLib_OBJ_TIMEOUT) {
+        errlogPrintf("Warning: epicsThread '%s' still awaiting join\n",
+            epicsThreadGetNameSelf());
+        status = semTake(joinSem, WAIT_FOREVER);
+    }
+    if (status)
+        perror("epicsThreadAwaitingJoin");
+
+    semDelete(joinSem);
+    taskSpareFieldSet(tid, joinField, 0);
+}
+  #define PREPARE_JOIN(tid, joinable) \
+    taskSpareFieldSet(tid, joinField, \
+        joinable ? (int) semBCreate(SEM_Q_FIFO, SEM_EMPTY) : 0)
+  #define AWAIT_JOIN(tid) epicsThreadAwaitingJoin(tid)
+#else
+  #define PREPARE_JOIN(tid, joinable)
+  #define AWAIT_JOIN(tid)
+#endif
+
 static void createFunction(EPICSTHREADFUNC func, void *parm)
 {
     int tid = taskIdSelf();
 
     taskVarAdd(tid,(int *)(char *)&papTSD);
-    /*Make sure that papTSD is still 0 after that call to taskVarAdd*/
-    papTSD = 0;
+    papTSD = NULL; /* Initialize for this thread */
+
     osdThreadHooksRun((epicsThreadId)tid);
+
     (*func)(parm);
+
     epicsExitCallAtThreadExits ();
     free(papTSD);
     taskVarDelete(tid,(int *)(char *)&papTSD);
+
+    AWAIT_JOIN(tid);
 }
 
 #ifdef ALTIVEC
@@ -190,27 +246,101 @@ static void createFunction(EPICSTHREADFUNC func, void *parm)
 #else
   #define TASK_FLAGS (VX_FP_TASK)
 #endif
-epicsThreadId epicsThreadCreate(const char *name,
-    unsigned int priority, unsigned int stackSize,
-    EPICSTHREADFUNC funptr,void *parm)
+epicsThreadId epicsThreadCreateOpt(const char * name,
+    EPICSTHREADFUNC funptr, void * parm, const epicsThreadOpts *opts )
 {
+    unsigned int stackSize;
     int tid;
 
     epicsThreadInit();
-    if(stackSize<100) {
-        errlogPrintf("epicsThreadCreate %s illegal stackSize %d\n",name,stackSize);
-        return(0);
+
+    if (!opts) {
+        static const epicsThreadOpts opts_default = EPICS_THREAD_OPTS_INIT;
+        opts = &opts_default;
     }
-    tid = taskSpawn((char *)name,getOssPriorityValue(priority),
+    stackSize = opts->stackSize;
+    if (stackSize <= epicsThreadStackBig)
+        stackSize = epicsThreadGetStackSize(stackSize);
+
+    if (stackSize < 100) {
+        errlogPrintf("epicsThreadCreate %s illegal stackSize %d\n",
+            name, stackSize);
+        return 0;
+    }
+
+    tid = taskCreate((char *)name,getOssPriorityValue(opts->priority),
         TASK_FLAGS, stackSize,
-        (FUNCPTR)createFunction,(int)funptr,(int)parm,
+        (FUNCPTR)createFunction, (int)funptr, (int)parm,
         0,0,0,0,0,0,0,0);
-    if(tid==ERROR) {
+    if (tid == ERROR) {
         errlogPrintf("epicsThreadCreate %s failure %s\n",
-            name,strerror(errno));
-        return(0);
+            name, strerror(errno));
+        return 0;
     }
-    return((epicsThreadId)tid);
+
+    PREPARE_JOIN(tid, opts->joinable);
+    taskActivate(tid);
+
+    return (epicsThreadId)tid;
+}
+
+void epicsThreadMustJoin(epicsThreadId id)
+{
+#ifdef EPICS_THREAD_CAN_JOIN
+    const char *fn = "epicsThreadMustJoin";
+    int tid = (int) id;
+    SEM_ID joinSem;
+    STATUS status;
+
+    if (!tid)
+        return;
+
+    joinSem = (SEM_ID) taskSpareFieldGet(tid, joinField);
+    if ((int) joinSem == ERROR) {
+        errlogPrintf("%s: Thread '%s' no longer exists.\n",
+            fn, taskName(tid));
+        return;
+    }
+
+    if (tid == taskIdSelf()) {
+        if (joinSem) {
+            semDelete(joinSem);
+            taskSpareFieldSet(tid, joinField, 0);
+        }
+        else {
+            errlogPrintf("%s: Self-join of unjoinable thread '%s'\n",
+                fn, taskName(tid));
+        }
+        return;
+    }
+
+    if (!joinSem) {
+        cantProceed("%s: Thread '%s' is not joinable.\n",
+            fn, taskName(tid));
+        return;
+    }
+
+    semGive(joinSem);   /* Rendezvous with thread */
+
+    status = taskWait(tid, JOIN_WARNING_TIMEOUT);
+    if (status && errno == S_objLib_OBJ_TIMEOUT) {
+        errlogPrintf("Warning: %s still waiting for thread '%s'\n",
+            fn, taskName(tid));
+        status = taskWait(tid, WAIT_FOREVER);
+    }
+    if (status) {
+        if (errno == S_taskLib_ILLEGAL_OPERATION) {
+            errlogPrintf("%s: This shouldn't happen!\n", fn);
+        }
+        else if (errno == S_objLib_OBJ_ID_ERROR) {
+            errlogPrintf("%s: %x is not a known thread\n", fn, tid);
+        }
+        else {
+            perror(fn);
+        }
+        cantProceed(fn);
+    }
+#endif
 }
 
 void epicsThreadSuspendSelf()
diff --git a/modules/libcom/src/osi/os/vxWorks/osdThread.h b/modules/libcom/src/osi/os/vxWorks/osdThread.h
index 2ee9f2d..1514566 100644
--- a/modules/libcom/src/osi/os/vxWorks/osdThread.h
+++ b/modules/libcom/src/osi/os/vxWorks/osdThread.h
@@ -3,11 +3,17 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
-* EPICS BASE Versions 3.13.7
-* and higher are distributed subject to a Software License Agreement found
-* in file LICENSE that is included with this distribution. 
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
 \*************************************************************************/
+
 #ifndef osdThreadh
 #define osdThreadh
 
+/* VxWorks 6.9 and later can support joining threads */
+
+#if (_WRS_VXWORKS_MAJOR == 6 && _WRS_VXWORKS_MINOR < 9)
+#undef EPICS_THREAD_CAN_JOIN
+#endif
+
 #endif /* osdThreadh */
diff --git a/modules/libcom/test/epicsThreadTest.cpp b/modules/libcom/test/epicsThreadTest.cpp
index eb26cc8..4796438 100644
--- a/modules/libcom/test/epicsThreadTest.cpp
+++ b/modules/libcom/test/epicsThreadTest.cpp
@@ -18,11 +18,14 @@
 #include <math.h>
 
 #include "epicsThread.h"
+#include "epicsEvent.h"
 #include "epicsTime.h"
 #include "errlog.h"
 #include "epicsUnitTest.h"
 #include "testMain.h"
 
+namespace {
+
 static epicsThreadPrivate<int> privateKey;
 
 class myThread: public epicsThreadRunable {
@@ -37,7 +40,7 @@ private:
 };
 
 myThread::myThread(int arg,const char *name) :
-    thread(*this,name,epicsThreadGetStackSize(epicsThreadStackSmall),50+arg),
+    thread(*this,name,epicsThreadStackSmall,50+arg),
     argvalue(0)
 {
     argvalue = new int;
@@ -52,7 +55,7 @@ void myThread::run()
     startEvt.signal();
     int *pset = argvalue;
     privateKey.set(argvalue);
-    epicsThreadSleep(2.0);
+
     int *pget = privateKey.get();
     testOk1(pget == pset);
 
@@ -60,9 +63,113 @@ void myThread::run()
     testOk1(thread.getPriority() == epicsThreadGetPriority(self));
 }
 
+void testMyThread()
+{
+
+    const int ntasks = 3;
+    myThread *myThreads[ntasks];
+
+    int startPriority = 0;
+    for (int i = 0; i < ntasks; i++) {
+        char name[10];
+        sprintf(name, "t%d", i);
+        myThreads[i] = new myThread(i, name);
+        if (i == 0)
+            startPriority = myThreads[i]->thread.getPriority();
+        myThreads[i]->thread.setPriority(startPriority + i);
+    }
+
+    for (int i = 0; i < ntasks; i++) {
+        myThreads[i]->startEvt.wait();
+        myThreads[i]->thread.exitWait();
+        delete myThreads[i];
+    }
+}
+
+struct joinStuff {
+    epicsThreadOpts *opts;
+    epicsEvent *trigger;
+    epicsEvent *finished;
+};
+
+void donothing(void *arg)
+{}
+
+void dowait(void *arg)
+{
+    epicsEvent *trigger = (epicsEvent *) arg;
+    trigger->wait();
+    epicsThreadSleep(0.1);
+}
+
+void dodelay(void *arg)
+{
+    epicsThreadSleep(2.0);
+}
+
+void joinTests(void *arg)
+{
+    struct joinStuff *stuff = (struct joinStuff *) arg;
+
+    // Task finishes before parent joins
+    epicsThreadId tid = epicsThreadCreateOpt("nothing",
+        &donothing, 0, stuff->opts);
+    epicsThreadSleep(0.1);
+    epicsThreadMustJoin(tid);
+
+    // Parent joins before task finishes
+    tid = epicsThreadCreateOpt("await",
+        &dowait, stuff->trigger, stuff->opts);
+    stuff->trigger->signal();
+    epicsThreadMustJoin(tid);
+
+    // Parent gets delayed until task finishes
+    epicsTime start, end;
+    start = epicsTime::getCurrent();
+    tid = epicsThreadCreateOpt("delay",
+        &dodelay, 0, stuff->opts);
+    epicsThreadMustJoin(tid);
+    end = epicsTime::getCurrent();
+    double duration = end - start;
+#ifndef EPICS_THREAD_CAN_JOIN
+    testTodoBegin("Thread join doesn't work");
+#endif
+    testOk(duration > 1.0, "Join delayed parent (%g seconds)", duration);
+    testTodoEnd();
+
+    // This is a no-op
+    epicsThreadId self = epicsThreadGetIdSelf();
+    epicsThreadMustJoin(self);
+
+    // This is a no-op as well, except for a warning.
+    eltc(0);
+    epicsThreadMustJoin(self);
+    eltc(1);
+
+    stuff->finished->signal();
+}
+
+void testJoining()
+{
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
+    epicsEvent finished, trigger;
+    struct joinStuff stuff = {
+        &opts, &trigger, &finished
+    };
+
+    opts.priority = 50;
+    opts.joinable = 1;
+    epicsThreadCreateOpt("parent", &joinTests, &stuff, &opts);
+
+    // as selfjoin joins itself, we can't.
+    testOk(finished.wait(10.0), "Join tests completed");
+}
+
+} // namespace
 
 typedef struct info {
     int  isOkToBlock;
+    int  didSomething;
 } info;
 
 extern "C" {
@@ -71,52 +178,55 @@ static void thread(void *arg)
     info *pinfo = (info *)arg;
 
     epicsThreadSetOkToBlock(pinfo->isOkToBlock);
-    epicsThreadSleep(1.0);
 
     testOk(epicsThreadIsOkToBlock() == pinfo->isOkToBlock,
         "%s epicsThreadIsOkToBlock() = %d",
         epicsThreadGetNameSelf(), pinfo->isOkToBlock);
-    epicsThreadSleep(0.1);
+
+    pinfo->didSomething = 1;
 }
 }
 
-
-MAIN(epicsThreadTest)
+static void testOkToBlock()
 {
-    testPlan(9);
+    epicsThreadOpts opts = EPICS_THREAD_OPTS_INIT;
 
-    unsigned int ncpus = epicsThreadGetCPUs();
-    testDiag("System has %u CPUs", ncpus);
-    testOk1(ncpus > 0);
+    opts.priority = 50;
+    opts.joinable = 1;
 
-    const int ntasks = 3;
-    myThread *myThreads[ntasks];
+    info infoA = {0, 0};
+    epicsThreadId threadA = epicsThreadCreateOpt("threadA", thread, &infoA, &opts);
 
-    int startPriority = 0;
-    for (int i = 0; i < ntasks; i++) {
-        char name[10];
-        sprintf(name, "t%d", i);
-        myThreads[i] = new myThread(i, name);
-        if (i == 0)
-            startPriority = myThreads[i]->thread.getPriority();
-        myThreads[i]->thread.setPriority(startPriority + i);
-    }
+    info infoB = {1, 0};
+    epicsThreadId threadB = epicsThreadCreateOpt("threadB", thread, &infoB, &opts);
 
-    for (int i = 0; i < ntasks; i++) {
-        myThreads[i]->startEvt.wait();
-        myThreads[i]->thread.exitWait();
-        delete myThreads[i];
-    }
+    // join B first to better our chance of detecting if it never runs.
+    epicsThreadMustJoin(threadB);
+    testOk1(infoB.didSomething);
 
-    unsigned int stackSize = epicsThreadGetStackSize(epicsThreadStackSmall);
+    epicsThreadMustJoin(threadA);
+    testOk1(infoA.didSomething);
+}
 
-    info infoA = {0};
-    epicsThreadCreate("threadA", 50, stackSize, thread, &infoA);
 
-    info infoB = {1};
-    epicsThreadCreate("threadB", 50, stackSize, thread, &infoB);
+MAIN(epicsThreadTest)
+{
+    testPlan(13);
 
-    epicsThreadSleep(2.0);
+    unsigned int ncpus = epicsThreadGetCPUs();
+    testDiag("System has %u CPUs", ncpus);
+    testOk1(ncpus > 0);
+    testDiag("main() thread %p", epicsThreadGetIdSelf());
+
+    testJoining(); // Do this first, ~epicsThread() uses it...
+    testMyThread();
+    testOkToBlock();
+
+    // attempt to self-join from a non-EPICS thread
+    // to make sure it does nothing as expected
+    eltc(0);
+    epicsThreadMustJoin(epicsThreadGetIdSelf());
+    eltc(1);
 
     return testDone();
 }