Merge lp:~epics-core/epics-base/ioc-shutdown into lp:~epics-core/epics-base/3.15

[Ralph Lange]

Add iocBuildNoCA() and iocShutdown() to iocInit API to allow running an IOC from withing a test program that is part of a test harness.

Doc changes in https://code.launchpad.net/~epics-documenters/epics-appdev/ioc-shutdown

[mdavidsaver (mdavidsaver)]

Ralph, have you tested re-initialization? I see that callbackOnceFlag in callback.c is not being reset.

[mdavidsaver (mdavidsaver)]

> static volatile enum ctl cbCtl;

Also, while I don't think this case will cause any problem, do we want to start using atomic ops for global flags like this as a matter of good practice?

[mdavidsaver (mdavidsaver)]

Looks like re-init does quite work. Remaining issues include:

> Warning: Registration already done.

From the generated registrar code. Only a warning.

> Warning -- iocshRegisterVariable redefining asCaDebug.

From iocsh.c. The lists for registered shell functions and variables need to be cleared.

> Access security is NOT enabled. Was asSetFilename specified before iocInit?
> iocBuild: asInit Failed.

The 'firstTime' flag (at least) in asDbLib.c also needs to be reset.

[mdavidsaver (mdavidsaver)]

> Looks like re-init does quite work.

doesn't

[lp:~epics-core/epics-base/ioc-shutdown]

12446. By mdavidsaver on 2013-10-11

dbShutdownTest

12447. By mdavidsaver on 2013-10-11

asShutdown

12448. By mdavidsaver on 2013-10-11

cleanup iocsh

12449. By mdavidsaver on 2013-10-11

cleanup dbScan

[mdavidsaver (mdavidsaver)]

re-init crashes and errors are fixed (I think). Still leaking memory in dbReadCOM().

I've added a new test dbShutdownTest which does the complete startup and shutdown sequence twice. This is added in src/ioc/db/test despite the fact that it depends on src/std (ie base.dbd) because I'm too lazy to create src/std/test (or similar).

[lp:~epics-core/epics-base/ioc-shutdown]

12450. By Ralph Lange on 2013-10-22

ioc/db/test: Add tests for common IOC threads to dbShutdownTest.c

[Ralph Lange (ralph-lange)]

Seems that only the scan threads are restarted.
errlog, timerQueue, taskwd, and callback threads are missing in the second cycle().

Added a test that shows this.

[lp:~epics-core/epics-base/ioc-shutdown]

12451. By Ralph Lange on 2013-10-22

ioc/db: Fix restart issue for callback threads.

12452. By Ralph Lange on 2013-10-22

libCom/error: Fix restart issue for errlog facility

12453. By Ralph Lange on 2013-10-22

libCom/taskwd: Fix restart issue.

12454. By Ralph Lange on 2013-10-22

ioc/db/test: Add testPlan number for dbShutdownTest

[Ralph Lange (ralph-lange)]

Fixed the restart issues - the extended dbShutdownTests pass now.

[lp:~epics-core/epics-base/ioc-shutdown]

12455. By Ralph Lange on 2013-10-22

ioc/misc: add missing include in iocInit.c

[Andrew Johnson (anj)]

For future reference: With almost 2**10 lines of diff this branch mixes together (admittedly related) changes to several different subsystems, which makes it hard to review (Jeff's 3.15_libcom_from_cvs_trunk branch has the same problem). The easier you make it to review a branch, the sooner and quicker it will likely be reviewed.

The documentation branch does not cover all the changes made here, it only talks about the modifications to the callback subsystem.

Is it wise to provide public APIs to all these Shutdown functions? I'm not so much concerned that this code doesn't work as whether by making them available we're letting ourselves in for a maintenance nightmare in the future. Do all these subsystems *need* to be shut down and restarted? What are the criteria for choosing which ones should and shouldn't be included? Could we shut everything down by calling epicsExitCallAtExits() instead, and just ensure that each system's registered shutdown function leaves the subsystem in a state that can be properly restarted?

The idea of resetting OnceFlags makes me cringe; they were never designed for that, and I'm not convinced that doing so is completely safe. The state change from EPICS_THREAD_ONCE_INIT through <active> to EPICS_THREAD_ONCE_DONE happens within the protection of a mutex inside the epicsThreadOnce() routines, the mutex providing a memory barrier on SMP. The reversals here are unprotected by either, and don't check that the subsystems have even been initialized either — could some other thread such as a time provider on a different CPU call a subsystem function at the wrong time while the subsystem is shutting down and crash it?

I'm not convinced that restarting an IOC inside the same process is a good idea, given the code we have (I know you're trying to change that, but I think it needs a properly planned solution, which this isn't [yet]). Just making the shutdown APIs available will mean that someone will want/try to use them on a running hard IOC, which is not going to work since we can't communicate the shutdown request to any device support or driver threads (other than by calling their epicsExitCallAtExits() routines, but existing support don't use that to prepare themselves for a restart).

IMHO if you need to test two independent IOCs, they need to be in two separate processes. You could split them into two test programs, or write the test as a Perl script which starts, controls and stops the IOCs as sub-processes; I have a Perl module which can do that on a host OS and could be extended to control an embedded IOC.

This isn't a definite NO, but I will need convincing that this is the right way to go.

- Andrew

review: Needs Fixing

[mdavidsaver (mdavidsaver)]

On 12/03/2013 02:35 PM, Andrew Johnson wrote:
> For future reference: With almost 2**10 lines of diff this branch mixes together (admittedly related) changes to several different subsystems,

??? The merge proposal shows:

961 lines (+337/-53) 21 files modified

Also, the branch consists of many small, focused, changesets.

> The documentation branch does not cover all the changes made here, it only talks about the modifications to the callback subsystem.

Agreed, this needs to change.

> Is it wise to provide public APIs to all these Shutdown functions?

I would like to wrap all of this up into a single function (say
"testdbCleanup"). We could then make private header(s) for the many
shutdown calls. Would this make the change more acceptable.

> Could we shut everything down by calling epicsExitCallAtExits() instead, and just ensure that each system's registered shutdown function leaves the subsystem in a state that can be properly restarted?

This is an option I suppose.

> The idea of resetting OnceFlags makes me cringe;

A good point.

> I'm not convinced that restarting an IOC inside the same process is a good idea, given the code we have (I know you're trying to change that, but I think it needs a properly planned solution, which this isn't [yet]). Just making the shutdown APIs available will mean that someone will want/try to use them on a running hard IOC, which is not going to work since we can't communicate the shutdown request to any device support or driver threads (other than by calling their epicsExitCallAtExits() routines, but existing support don't use that to prepare themselves for a restart).

The goal is unittests. It it would help in your mind we can only allow
shutdown/cleanup between calling testPlan() and testDone().

[Andrew Johnson (anj)]

On 12/03/2013 03:39 PM, mdavidsaver wrote:
> On 12/03/2013 02:35 PM, Andrew Johnson wrote:
>> For future reference: With almost 2**10 lines of diff this branch mixes
> together (admittedly related) changes to several different subsystems,
>
> ??? The merge proposal shows:
>
> 961 lines (+337/-53) 21 files modified

961 ≈ 1024

I'm just saying that the longer the diff is, the more reviewers will be
put off looking at it. If the changes were split into smaller dependent
branches, reviewing them might look like less work and could be done in
smaller chunks (this was a human psychology comment, not a programming
issue at all).

>> Is it wise to provide public APIs to all these Shutdown functions?
>
> I would like to wrap all of this up into a single function (say
> "testdbCleanup"). We could then make private header(s) for the many
> shutdown calls. Would this make the change more acceptable.

Hmm, maybe. How about an alternative API to iocInit, for test programs
only, and implemented outside of iocInit.c? I would be quite happy for
the various init*Sup() and related routines to be moved out of there and
into ioc/db somewhere, dbInit.c maybe?

>> Could we shut everything down by calling epicsExitCallAtExits() instead,
> and just ensure that each system's registered shutdown function leaves
> the subsystem in a state that can be properly restarted?
>
> This is an option I suppose.

The advantage of using epicsExitCallAtExits() is that we don't have to
expose the various Shutdown symbols at all, and we automatically stop
everything that we've started, in the (correct) reverse order.

The disadvantage is this will slow down IOC shutdown unnecessarily,
since we only need the 'clean up ready to start up again' actions in
those exit routines when we're running tests that need restart.

I'm getting visions of a subsystem management API; subsystems would
register both a fast exit and a slow clean-up routine with it. It could
even incorporate the taskwd, and provide subsystem/task status & health
info on request. This could also somewhere to centralize all of the
init/run/pause/exit enum task controls that are proliferating (which I
agree should probably become atomics).

Too much work?

- Andrew
--
Advertising may be described as the science of arresting the human
intelligence long enough to get money from it. -- Stephen Leacock

[Ralph Lange (ralph-lange)]

> On 12/03/2013 03:39 PM, mdavidsaver wrote:
> > On 12/03/2013 02:35 PM, Andrew Johnson wrote:
> >> For future reference: With almost 2**10 lines of diff this branch mixes
> > together (admittedly related) changes to several different subsystems,
> >
> > ??? The merge proposal shows:
> >
> > 961 lines (+337/-53) 21 files modified
>
> 961 ≈ 1024
>
> I'm just saying that the longer the diff is, the more reviewers will be
> put off looking at it. If the changes were split into smaller dependent
> branches, reviewing them might look like less work and could be done in
> smaller chunks (this was a human psychology comment, not a programming
> issue at all).

I was considering this, but did not find a useful subset of the changes that would have added a functionality that could be documented and tested for.

Which in my imagination would have led to rejection of each individual branch and you commenting that a single functionality should have been presented as a single branch instead of multiple dependent branches that don't add anything useful by itself. ;-)

Admittedly, I might have been able to split the callback related things off. But still with a high change of not being able to add the mandatory tests.

[Ralph Lange (ralph-lange)]

> The documentation branch does not cover all the changes made here, it only
> talks about the modifications to the callback subsystem.

If you are talking about the Application Developer's Guide, this change is IMHO the only one that is part of an API that Application Developers use.

Base development and internal APIs are not supposed to be covered by this document, correct?

[Ralph Lange (ralph-lange)]

> The advantage of using epicsExitCallAtExits() is that we don't have to
> expose the various Shutdown symbols at all, and we automatically stop
> everything that we've started, in the (correct) reverse order.
>
> The disadvantage is this will slow down IOC shutdown unnecessarily,
> since we only need the 'clean up ready to start up again' actions in
> those exit routines when we're running tests that need restart.
>
> I'm getting visions of a subsystem management API; subsystems would
> register both a fast exit and a slow clean-up routine with it. It could
> even incorporate the taskwd, and provide subsystem/task status & health
> info on request. This could also somewhere to centralize all of the
> init/run/pause/exit enum task controls that are proliferating (which I
> agree should probably become atomics).
>
> Too much work?

If you think further, that becomes a major design change of iocCore.

It basically means going from somewhat modular spaghetti toward a component model where we have a mostly administrative core framework, and everything else becomes a component, possibly with separate versioning, providing a component API that handles component admin (version), compatibility (components requiring certain versions of other components), status/health, init/run/pause/exit, etc.
The lists of available RSETs and DSETs would basically become extension points.

Existing support modules should have the choice of registering as a component. Plus maybe some generic (generated) code that handles existing record and device support to become (pseudo) components.

That would be a lot of work.
But I like the idea.

[Andrew Johnson (anj)]

On 12/04/2013 03:49 AM, Ralph Lange wrote:
>> The documentation branch does not cover all the changes made here,
>> it only talks about the modifications to the callback subsystem.
>
> If you are talking about the Application Developer's Guide, this
> change is IMHO the only one that is part of an API that Application
> Developers use.

8.6.4 documents asInit(), so should mention asShutdown() nearby.
15.8.1 documents dbCaLinkInit(), so should mention dbCaShutdown(). Mea
culpa that it doesn't cover dbCaRun() or dbCaPause(), although they are
mentioned in section 7.
17.3 documents scanInit(), so should mention scanShutdown() nearby.
10.5 documents errlogInit(), so should mention errlogShutdown() nearby.

> Base development and internal APIs are not supposed to be covered by
> this document, correct?

Maybe, but as demonstrated above the documentation already covers
several internal APIs. If we don't modify those sections when we're
updating the code, the existing docs could start to contain wrong
information, which is a Bad Thing™ and may lead to Broken Window Syndrome.

--
Advertising may be described as the science of arresting the human
intelligence long enough to get money from it. -- Stephen Leacock

[Andrew Johnson (anj)]

On 12/04/2013 04:56 AM, Ralph Lange wrote:
>> Too much work?
>
> If you think further, that becomes a major design change of iocCore.

I agree; maybe a Codeathon project for someone to look at?

> That would be a lot of work. But I like the idea.

I created a Blueprint on LP to keep the idea alive.

--
Advertising may be described as the science of arresting the human
intelligence long enough to get money from it. -- Stephen Leacock

[Andrew Johnson (anj)]

So, where to go with this from here?

My preference would be to look further at the epicsExitCallAtExits() solution, instead of adding all the public Shutdown routines (which also reduces the documentation changes needed). Maybe we could add a global flag to epicsExit.c which the registered routines can check to decide whether they need to do a full clean-up ready for a restart, or just a quick shutdown (I want to avoid them all wasting time freeing memory if the process is about to exit or VxWorks is about to reboot).

[Andrew Johnson (anj)]

epicsExitCallAtExits() is not currently re-usable, but can be made so relatively easily, I have the necessary changes here.

[lp:~epics-core/epics-base/ioc-shutdown]

12456. By mdavidsaver on 2013-12-05

epicsThreadOnceReset

12457. By mdavidsaver on 2013-12-05

use epicsThreadOnceReset

12458. By mdavidsaver on 2013-12-05

cleanup initHook

12459. By mdavidsaver on 2013-12-05

add dbUnitTest.h

hide boilerplate of tests using the PDB

[mdavidsaver (mdavidsaver)]

I've added epicsThreadOnceReset() to provide a safe way to reset once flags.

Also I introduce dbUnitTest.h to provide the boilerplate which unittests running in an IOC should use.

[lp:~epics-core/epics-base/ioc-shutdown]

12460. By Andrew Johnson on 2013-12-05

Make epicsExit subsystem reusable.

Calling epicsAtExit() after epicsExitCallAtExits() now
recreates the per-process list and registers the routine.

12461. By mdavidsaver on 2013-12-14

deletePeriodic: fix list free

Can't iterate on free'd node

12462. By mdavidsaver on 2013-12-14

dbScan: avoid double shutdown

scanShutdown will be called (later) by iocShutdown
via exitDatabase()

12463. By mdavidsaver on 2013-12-14

epicsExit: optional debug printing

Add a flag to cause a string to be printed
before each handler is run to show the order.

12464. By mdavidsaver on 2013-12-14

errlog: nicer exit handler name

[Ralph Lange (ralph-lange)]

Where are we on this?

I would like to get this branch into a mergeable state, but I have lost track a bit, and I'm not sure about the line between long-term goals and mandatory features for this to be merged.

[Ralph Lange (ralph-lange)]

OK ... I think hiding the additional shutdown calls is what is left to do. I will

- move the various init*Sup() and related routines to ioc/db/dbInit.c
- create a private header dbInit.h
- call appropriately from iocInit.c and dbUnitTest.c through the dbInit.h API

- document dbUnitTest.h

That will not preclude us from doing more sophisticated module start/shutdown management later on, and be sufficient for the merge.

Correct?

[Andrew Johnson (anj)]

Just merged the replacement ioc-shutdown2 branch, this merge is no longer applicable.

review: Disapprove

Unmerged revisions

12464. By mdavidsaver on 2013-12-14

errlog: nicer exit handler name

12463. By mdavidsaver on 2013-12-14

epicsExit: optional debug printing

Add a flag to cause a string to be printed
before each handler is run to show the order.

12462. By mdavidsaver on 2013-12-14

dbScan: avoid double shutdown

scanShutdown will be called (later) by iocShutdown
via exitDatabase()

12461. By mdavidsaver on 2013-12-14

deletePeriodic: fix list free

Can't iterate on free'd node

12460. By Andrew Johnson on 2013-12-05

Make epicsExit subsystem reusable.

Calling epicsAtExit() after epicsExitCallAtExits() now
recreates the per-process list and registers the routine.

12459. By mdavidsaver on 2013-12-05

add dbUnitTest.h

hide boilerplate of tests using the PDB

12458. By mdavidsaver on 2013-12-05

cleanup initHook

12457. By mdavidsaver on 2013-12-05

use epicsThreadOnceReset

12456. By mdavidsaver on 2013-12-05

epicsThreadOnceReset

12455. By Ralph Lange on 2013-10-22

ioc/misc: add missing include in iocInit.c

=== modified file 'src/Makefile'
--- src/Makefile	2012-06-22 23:16:26 +0000
+++ src/Makefile	2013-12-14 18:55:29 +0000
@@ -62,7 +62,7 @@
 ioc_DEPEND_DIRS = libCom ca/client
 
 DIRS += ioc/db/test
-ioc/db/test_DEPEND_DIRS = ioc libCom/RTEMS
+ioc/db/test_DEPEND_DIRS = ioc std libCom/RTEMS
 
 DIRS += ioc/dbtemplate/test
 ioc/dbtemplate/test_DEPEND_DIRS = ioc

=== modified file 'src/ioc/as/asDbLib.c'
--- src/ioc/as/asDbLib.c	2012-10-01 05:54:10 +0000
+++ src/ioc/as/asDbLib.c	2013-12-14 18:55:29 +0000
@@ -152,6 +152,15 @@
     return(asInitCommon());
 }
 
+int asShutdown(void) {
+    volatile ASBASE *pbase = pasbase;
+    pasbase = NULL;
+    firstTime = TRUE;
+    if(pbase)
+        asFreeAll((ASBASE*)pbase);
+    return 0;
+}
+
 static void wdCallback(void *arg)
 {
     ASDBCALLBACK *pcallback = (ASDBCALLBACK *)arg;

=== modified file 'src/ioc/as/asDbLib.h'
--- src/ioc/as/asDbLib.h	2012-07-15 21:08:50 +0000
+++ src/ioc/as/asDbLib.h	2013-12-14 18:55:29 +0000
@@ -32,6 +32,7 @@
 epicsShareFunc int asSetSubstitutions(const char *substitutions);
 epicsShareFunc int asInit(void);
 epicsShareFunc int asInitAsyn(ASDBCALLBACK *pcallback);
+epicsShareFunc int asShutdown(void);
 epicsShareFunc int asDbGetAsl(struct dbChannel *chan);
 epicsShareFunc void * asDbGetMemberPvt(struct dbChannel *chan);
 epicsShareFunc int asdbdump(void);

=== modified file 'src/ioc/db/Makefile'
--- src/ioc/db/Makefile	2012-11-29 18:53:21 +0000
+++ src/ioc/db/Makefile	2013-12-14 18:55:29 +0000
@@ -37,6 +37,7 @@
 INC += dbState.h
 INC += db_access_routines.h
 INC += db_convert.h
+INC += dbUnitTest.h
 
 # Generate menuGlobal.dbd automatically
 DBD += menuGlobal.dbd
@@ -86,4 +87,4 @@
 dbCore_SRCS += dbIocRegister.c
 dbCore_SRCS += chfPlugin.c
 dbCore_SRCS += dbState.c
-
+dbCore_SRCS += dbUnitTest.c

=== modified file 'src/ioc/db/callback.c'
--- src/ioc/db/callback.c	2012-05-04 22:34:48 +0000
+++ src/ioc/db/callback.c	2013-12-14 18:55:29 +0000
@@ -53,6 +53,8 @@
 static epicsTimerQueueId timerQueue;
 
 /* Shutdown handling */
+enum ctl {ctlInit, ctlRun, ctlPause, ctlExit};
+static volatile enum ctl cbCtl;
 static epicsEventId startStopEvent;
 static void *exitCallback;
 
@@ -101,10 +103,13 @@
     epicsEventSignal(startStopEvent);
 }
 
-static void callbackShutdown(void *arg)
+void callbackShutdown(void)
 {
     int i;
 
+    if (cbCtl == ctlExit) return;
+    cbCtl = ctlExit;
+
     for (i = 0; i < NUM_CALLBACK_PRIORITIES; i++) {
         int lockKey = epicsInterruptLock();
         int ok = epicsRingPointerPush(callbackQ[i], &exitCallback);
@@ -112,6 +117,13 @@
         epicsEventSignal(callbackSem[i]);
         if (ok) epicsEventWait(startStopEvent);
     }
+    epicsTimerQueueRelease(timerQueue);
+    epicsThreadOnceReset(&callbackOnceFlag);
+}
+
+static void callbackExit(void *arg)
+{
+    callbackShutdown();
 }
 
 static void callbackInitOnce(void *arg)
@@ -119,6 +131,7 @@
     int i;
 
     startStopEvent = epicsEventMustCreate(epicsEventEmpty);
+    cbCtl = ctlRun;
     timerQueue = epicsTimerQueueAllocate(0,epicsThreadPriorityScanHigh);
     for (i = 0; i < NUM_CALLBACK_PRIORITIES; i++) {
         epicsThreadId tid;
@@ -137,7 +150,7 @@
         else
             epicsEventWait(startStopEvent);
     }
-    epicsAtExit(callbackShutdown, NULL);
+    epicsAtExit(callbackExit, NULL);
 }
 
 void callbackInit(void)

=== modified file 'src/ioc/db/callback.h'
--- src/ioc/db/callback.h	2010-10-05 19:27:37 +0000
+++ src/ioc/db/callback.h	2013-12-14 18:55:29 +0000
@@ -57,6 +57,7 @@
 
 epicsShareFunc void callbackInit(void);
 epicsShareFunc void callbackRequest(CALLBACK *pCallback);
+epicsShareFunc void callbackShutdown(void);
 epicsShareFunc void callbackSetProcess(
     CALLBACK *pcallback, int Priority, void *pRec);
 epicsShareFunc void callbackRequestProcessCallback(

=== modified file 'src/ioc/db/dbCa.c'
--- src/ioc/db/dbCa.c	2012-06-22 23:16:26 +0000
+++ src/ioc/db/dbCa.c	2013-12-14 18:55:29 +0000
@@ -173,15 +173,22 @@
     dbScanUnlock(pdbCommon);
 }
 
-static void dbCaShutdown(void *arg)
+void dbCaShutdown(void)
 {
-    if (dbCaCtl == ctlRun) {
+    if (dbCaCtl == ctlRun || dbCaCtl == ctlPause) {
         dbCaCtl = ctlExit;
         epicsEventSignal(workListEvent);
         epicsEventMustWait(startStopEvent);
+        epicsEventDestroy(startStopEvent);
+        epicsEventDestroy(workListEvent);
     }
 }
 
+static void dbCaExit(void *arg)
+{
+    dbCaShutdown();
+}
+
 void dbCaLinkInit(void)
 {
     dbServiceIOInit();
@@ -194,19 +201,23 @@
         epicsThreadGetStackSize(epicsThreadStackBig),
         dbCaTask, NULL);
     epicsEventMustWait(startStopEvent);
-    epicsAtExit(dbCaShutdown, NULL);
+    epicsAtExit(dbCaExit, NULL);
 }
 
 void dbCaRun(void)
 {
-    dbCaCtl = ctlRun;
-    epicsEventSignal(workListEvent);
+    if (dbCaCtl == ctlPause) {
+        dbCaCtl = ctlRun;
+        epicsEventSignal(workListEvent);
+    }
 }
 
 void dbCaPause(void)
 {
-    dbCaCtl = ctlPause;
-    epicsEventSignal(workListEvent);
+    if (dbCaCtl == ctlRun) {
+        dbCaCtl = ctlPause;
+        epicsEventSignal(workListEvent);
+    }
 }
 
 void dbCaAddLinkCallback(struct link *plink,

=== modified file 'src/ioc/db/dbCa.h'
--- src/ioc/db/dbCa.h	2012-04-27 17:21:47 +0000
+++ src/ioc/db/dbCa.h	2013-12-14 18:55:29 +0000
@@ -26,6 +26,7 @@
 epicsShareFunc void dbCaLinkInit(void);
 epicsShareFunc void dbCaRun(void);
 epicsShareFunc void dbCaPause(void);
+epicsShareFunc void dbCaShutdown(void);
 
 epicsShareFunc void dbCaAddLinkCallback(struct link *plink,
     dbCaCallback connect, dbCaCallback monitor, void *userPvt);

=== modified file 'src/ioc/db/dbScan.c'
--- src/ioc/db/dbScan.c	2013-04-23 15:38:57 +0000
+++ src/ioc/db/dbScan.c	2013-12-14 18:55:29 +0000
@@ -3,6 +3,8 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
+* Copyright (c) 2013 Helmholtz-Zentrum Berlin
+*     für Materialien und Energie GmbH.
 * EPICS BASE is distributed subject to a Software License Agreement found
 * in file LICENSE that is included with this distribution. 
 \*************************************************************************/
@@ -129,6 +131,7 @@
 static void initOnce(void);
 static void periodicTask(void *arg);
 static void initPeriodic(void);
+static void deletePeriodic(void);
 static void spawnPeriodic(int ind);
 static void initEvent(void);
 static void eventCallback(CALLBACK *pcallback);
@@ -139,10 +142,13 @@
 static void addToList(struct dbCommon *precord, scan_list *psl);
 static void deleteFromList(struct dbCommon *precord, scan_list *psl);
 

-static void scanShutdown(void *arg)
+void scanShutdown(void)
 {
     int i;
 
+    if (scanCtl == ctlExit) return;
+    scanCtl = ctlExit;
+
     interruptAccept = FALSE;
 
     for (i = 0; i < nPeriodic; i++) {
@@ -153,6 +159,18 @@
 
     scanOnce((dbCommon *)&exitOnce);
     epicsEventWait(startStopEvent);
+
+    deletePeriodic();
+
+    epicsRingPointerDelete(onceQ);
+
+    epicsEventDestroy(startStopEvent);
+    epicsEventDestroy(onceSem);
+    onceSem = startStopEvent = NULL;
+
+    free(periodicTaskId);
+    papPeriodic = NULL;
+    periodicTaskId = NULL;
 }
 
 long scanInit(void)
@@ -169,7 +187,6 @@
     for (i = 0; i < nPeriodic; i++)
         spawnPeriodic(i);
 
-    epicsAtExit(scanShutdown, NULL);
     return 0;
 }
 
@@ -672,6 +689,22 @@
     }
 }
 
+static void deletePeriodic(void)
+{
+    int i;
+
+    for (i = 0; i < nPeriodic; i++) {
+        periodic_scan_list *ppsl = papPeriodic[i];
+        ellFree(&ppsl->scan_list.list);
+        epicsEventDestroy(ppsl->loopEvent);
+        epicsMutexDestroy(ppsl->scan_list.lock);
+        free(ppsl);
+    }
+
+    free(papPeriodic);
+    papPeriodic = NULL;
+}
+
 static void spawnPeriodic(int ind)
 {
     periodic_scan_list *ppsl;
@@ -778,23 +811,25 @@
 {
     dbRecordType *pdbRecordType;
 
-    /*Look for first record*/
     for (pdbRecordType = (dbRecordType *)ellFirst(&pdbbase->recordTypeList);
          pdbRecordType;
          pdbRecordType = (dbRecordType *)ellNext(&pdbRecordType->node)) {
         dbRecordNode *pdbRecordNode;
+
         for (pdbRecordNode = (dbRecordNode *)ellFirst(&pdbRecordType->recList);
              pdbRecordNode;
              pdbRecordNode = (dbRecordNode *)ellNext(&pdbRecordNode->node)) {
             dbCommon *precord = pdbRecordNode->precord;
+
             if (!precord->name[0] ||
                 pdbRecordNode->flags & DBRN_FLAGS_ISALIAS)
                 continue;
+
             scanAdd(precord);
         }
     }
 }
-

+
 static void addToList(struct dbCommon *precord, scan_list *psl)
 {
     scan_element *pse, *ptemp;

=== modified file 'src/ioc/db/dbScan.h'
--- src/ioc/db/dbScan.h	2012-04-10 22:10:50 +0000
+++ src/ioc/db/dbScan.h	2013-12-14 18:55:29 +0000
@@ -44,6 +44,7 @@
 epicsShareFunc long scanInit(void);
 epicsShareFunc void scanRun(void);
 epicsShareFunc void scanPause(void);
+epicsShareFunc void scanShutdown(void);
 
 epicsShareFunc EVENTPVT eventNameToHandle(const char* event);
 epicsShareFunc void postEvent(EVENTPVT epvt);

=== added file 'src/ioc/db/dbUnitTest.c'
--- src/ioc/db/dbUnitTest.c	1970-01-01 00:00:00 +0000
+++ src/ioc/db/dbUnitTest.c	2013-12-14 18:55:29 +0000
@@ -0,0 +1,123 @@
+/*************************************************************************\
+* Copyright (c) 2013 Brookhaven National Laboratory.
+* Copyright (c) 2013 ITER Organization.
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+ \*************************************************************************/
+
+/*
+ *  Author: Michael Davidsaver <[e-mail]>
+ *          Ralph Lange <[e-mail]>
+ */
+
+#include <string.h>
+
+#include "epicsUnitTest.h"
+#include "osiFileName.h"
+#include "dbmf.h"
+#include "registry.h"
+#define epicsExportSharedSymbols
+#include "iocInit.h"
+#include "initHooks.h"
+#include "dbBase.h"
+#include "dbAccess.h"
+#include "dbStaticLib.h"
+
+#include "dbUnitTest.h"
+
+void testdbPrepare(void)
+{
+    /* No-op at the moment */
+}
+
+void testdbReadDatabase(const char* file,
+                        const char* path,
+                        const char* substitutions)
+{
+    if(!path)
+        path = "." OSI_PATH_LIST_SEPARATOR ".." OSI_PATH_LIST_SEPARATOR
+                "../O.Common" OSI_PATH_LIST_SEPARATOR "O.Common";
+    if(dbReadDatabase(&pdbbase, file, path, substitutions))
+        testAbort("Failed to load test database\ndbReadDatabase(%s,%s,%s)",
+                  file, path, substitutions);
+}
+
+int testiocInit(void)
+{
+    return iocBuildNoCA() || iocRun();
+}
+
+int testiocShutdown(void)
+{
+    return iocShutdown();
+}
+
+void testdbCleanup(void)
+{
+    dbFreeBase(pdbbase);
+    initHookFree();
+    registryFree();
+    pdbbase = NULL;
+    dbmfFreeChunks();
+}
+
+long testdbPutField(const char* pv, short dbrType, ...)
+{
+    long ret;
+    va_list ap;
+    va_start(ap, dbrType);
+    ret = testVdbPutField(pv, dbrType, ap);
+    va_end(ap);
+    return ret;
+}
+
+union anybuf {
+    epicsAny val;
+    char bytes[sizeof(epicsAny)];
+};
+
+long testVdbPutField(const char* pv, short dbrType, va_list ap)
+{
+    DBADDR addr;
+    union anybuf pod;
+
+    if(dbNameToAddr(pv, &addr))
+        testAbort("Missing PV %s", pv);
+
+    switch(dbrType) {
+    case DBR_STRING: {
+        const char *uarg = va_arg(ap,char*);
+        epicsOldString buffer;
+        strncpy(buffer, uarg, sizeof(buffer));
+        buffer[sizeof(buffer)-1] = '\0';
+        return dbPutField(&addr, dbrType, buffer, 1);
+    }
+
+#define OP(DBR,Type,mem) case DBR: {pod.val.mem = va_arg(ap,Type); break;}
+    OP(DBR_CHAR, int, int8);
+    OP(DBR_UCHAR, int, uInt8);
+    OP(DBR_SHORT, int, int16);
+    OP(DBR_USHORT, int, uInt16);
+    OP(DBR_LONG, int, int32);
+    OP(DBR_ULONG, unsigned int, uInt32);
+    OP(DBR_FLOAT, double, float32);
+    OP(DBR_DOUBLE, double, float64);
+    OP(DBR_ENUM, int, enum16);
+#undef OP
+    default:
+        testAbort("invalid DBR: dbPutField(%s, %d, ...)",
+                  addr.precord->name, dbrType);
+    }
+
+    return dbPutField(&addr, dbrType, pod.bytes, 1);
+}
+
+dbCommon* testGetRecord(const char* pv)
+{
+    DBADDR addr;
+
+    if(dbNameToAddr(pv, &addr))
+        testAbort("Missing PV %s", pv);
+
+    return addr.precord;
+}

=== added file 'src/ioc/db/dbUnitTest.h'
--- src/ioc/db/dbUnitTest.h	1970-01-01 00:00:00 +0000
+++ src/ioc/db/dbUnitTest.h	2013-12-14 18:55:29 +0000
@@ -0,0 +1,54 @@
+/*************************************************************************\
+* Copyright (c) 2013 Brookhaven National Laboratory.
+* Copyright (c) 2013 ITER Organization.
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+ \*************************************************************************/
+
+/*
+ *  Author: Michael Davidsaver <[e-mail]>
+ *          Ralph Lange <[e-mail]>
+ */
+
+#ifndef EPICSUNITTESTDB_H
+#define EPICSUNITTESTDB_H
+
+#include <stdarg.h>
+
+#include "epicsUnitTest.h"
+#include "dbAddr.h"
+#include "dbCommon.h"
+
+#include "shareLib.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+epicsShareFunc void testdbPrepare(void);
+epicsShareFunc void testdbReadDatabase(const char* file,
+                                       const char* path,
+                                       const char* substitutions);
+epicsShareFunc int testiocInit(void);
+epicsShareFunc int testiocShutdown(void);
+epicsShareFunc void testdbCleanup(void);
+
+/* Scalar only version.
+ *
+ * Remember to use the correct argument type!s
+ *
+ * int for DBR_UCHAR, DBR_CHAR, DBR_USHORT, DBR_SHORT, DBR_LONG
+ * unsigned int for DBR_ULONG
+ * double for DBR_FLOAT and DBR_DOUBLE
+ * const char* for DBR_STRING
+ */
+epicsShareFunc long testdbPutField(const char* pv, short dbrType, ...);
+epicsShareFunc long testVdbPutField(const char* pv, short dbrType, va_list ap);
+
+epicsShareFunc dbCommon* testGetRecord(const char* pv);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif // EPICSUNITTESTDB_H

=== modified file 'src/ioc/db/initHooks.c'
--- src/ioc/db/initHooks.c	2010-10-05 19:27:37 +0000
+++ src/ioc/db/initHooks.c	2013-12-14 18:55:29 +0000
@@ -93,6 +93,14 @@
     }
 }
 
+void initHookFree(void)
+{
+    initHookInit();
+    epicsMutexMustLock(listLock);
+    ellFree(&functionList);
+    epicsMutexUnlock(listLock);
+}
+
 /*
  * Call any time you want to print out a state name.
  */

=== modified file 'src/ioc/db/initHooks.h'
--- src/ioc/db/initHooks.h	2010-10-05 19:27:37 +0000
+++ src/ioc/db/initHooks.h	2013-12-14 18:55:29 +0000
@@ -60,6 +60,7 @@
 epicsShareFunc int initHookRegister(initHookFunction func);
 epicsShareFunc void initHookAnnounce(initHookState state);
 epicsShareFunc const char *initHookName(int state);
+epicsShareFunc void initHookFree(void);
 
 #ifdef __cplusplus
 }

=== modified file 'src/ioc/db/test/Makefile'
--- src/ioc/db/test/Makefile	2012-07-17 19:33:31 +0000
+++ src/ioc/db/test/Makefile	2013-12-14 18:55:29 +0000
@@ -16,6 +16,15 @@
 
 PROD_LIBS = xRec dbCore ca Com
 
+TESTPROD_HOST += dbShutdownTest
+TARGETS += $(COMMON_DIR)/dbShutdownTest.dbd
+dbShutdownTest_SRCS += dbShutdownTest.c
+dbShutdownTest_SRCS += dbShutdownTest_registerRecordDeviceDriver.cpp
+dbShutdownTest_LIBS += $(EPICS_BASE_IOC_LIBS)
+dbShutdownTest_DBD += base.dbd
+TESTFILES += $(COMMON_DIR)/dbShutdownTest.dbd ../sRecord.db
+TESTS += dbShutdownTest
+
 TESTPROD_HOST += callbackTest
 callbackTest_SRCS += callbackTest.c
 testHarness_SRCS += callbackTest.c

=== added file 'src/ioc/db/test/dbShutdownTest.c'
--- src/ioc/db/test/dbShutdownTest.c	1970-01-01 00:00:00 +0000
+++ src/ioc/db/test/dbShutdownTest.c	2013-12-14 18:55:29 +0000
@@ -0,0 +1,94 @@
+/*************************************************************************\
+* Copyright (c) 2013 Brookhaven National Laboratory.
+* Copyright (c) 2013 ITER Organization.
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+ \*************************************************************************/
+
+/*
+ *  Author: Michael Davidsaver <[e-mail]>
+ *          Ralph Lange <[e-mail]>
+ */
+
+#include <string.h>
+
+#include "dbUnitTest.h"
+#include "epicsThread.h"
+#include "iocInit.h"
+#include "dbBase.h"
+#include "dbAccess.h"
+#include "registry.h"
+#include "dbStaticLib.h"
+#include "osiFileName.h"
+#include "dbmf.h"
+
+#include "testMain.h"
+
+void dbShutdownTest_registerRecordDeviceDriver(struct dbBase *);
+
+static struct threadItem {
+    char *name;
+    char found;
+} commonThreads[] = {
+    { "errlog", 0 },
+    { "taskwd", 0 },
+    { "timerQueue", 0 },
+    { "cbLow", 0 },
+    { "scanOnce", 0 },
+    { NULL, 0 }
+};
+
+static
+void findCommonThread (epicsThreadId id) {
+    struct threadItem *thr;
+    char name[32];
+
+    epicsThreadGetName(id, name, 32);
+
+    for (thr = commonThreads; thr->name; thr++) {
+        if (strcasecmp(thr->name, name) == 0) {
+            thr->found = 1;
+        }
+    }
+}
+
+static
+void checkCommonThreads (void) {
+    struct threadItem *thr;
+
+    for (thr = commonThreads; thr->name; thr++) {
+        testOk(thr->found, "Thread %s is running", thr->name);
+        thr->found = 0;
+    }
+}
+
+static
+void cycle(void) {
+
+    testdbPrepare();
+
+    testdbReadDatabase("dbShutdownTest.dbd", NULL, NULL);
+
+    dbShutdownTest_registerRecordDeviceDriver(pdbbase);
+
+    testdbReadDatabase("sRecord.db", NULL, NULL);
+
+    testOk1(!testiocInit());
+
+    epicsThreadMap(findCommonThread);
+    checkCommonThreads();
+
+    testOk1(testiocShutdown()==0);
+
+    testdbCleanup();
+}
+
+MAIN(dbShutdownTest)
+{
+    testPlan(14);
+
+    cycle();
+    cycle();
+
+    return testDone();
+}

=== added file 'src/ioc/db/test/sRecord.db'
--- src/ioc/db/test/sRecord.db	1970-01-01 00:00:00 +0000
+++ src/ioc/db/test/sRecord.db	2013-12-14 18:55:29 +0000
@@ -0,0 +1,1 @@
+record(ai, "somename") {}

=== modified file 'src/ioc/misc/dbCore.dbd'
--- src/ioc/misc/dbCore.dbd	2011-02-26 05:56:51 +0000
+++ src/ioc/misc/dbCore.dbd	2013-12-14 18:55:29 +0000
@@ -5,6 +5,9 @@
 # This file provides iocsh access to variables that control some lesser-used
 # and debugging features of the IOC database code.
 
+# show epicsAtExit callbacks as they are run
+variable(atExitDebug,int)
+
 # Access security subroutines
 variable(asCaDebug,int)
 

=== modified file 'src/ioc/misc/iocInit.c'
--- src/ioc/misc/iocInit.c	2013-05-30 20:00:37 +0000
+++ src/ioc/misc/iocInit.c	2013-12-14 18:55:29 +0000
@@ -3,6 +3,8 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
+* Copyright (c) 2013 Helmholtz-Zentrum Berlin
+*     für Materialien und Energie GmbH.
 * EPICS BASE is distributed subject to a Software License Agreement found
 * in file LICENSE that is included with this distribution. 
 \*************************************************************************/
@@ -31,6 +33,7 @@
 #include "errMdef.h"
 #include "taskwd.h"
 #include "caeventmask.h"
+#include "iocsh.h"
 
 #define epicsExportSharedSymbols
 #include "alarm.h"
@@ -88,12 +91,13 @@
     return iocBuild() || iocRun();
 }
 
-int iocBuild(void)
+static int iocBuild_1(void)
 {
-    if (iocState != iocVirgin) {
-        errlogPrintf("iocBuild: IOC can only be initialized once\n");
+    if (iocState != iocVirgin && iocState != iocStopped) {
+        errlogPrintf("iocBuild: IOC can only be initialized from uninitialized or stopped state\n");
         return -1;
     }
+    errlogInit(0);
     initHookAnnounce(initHookAtIocBuild);
 
     if (!epicsThreadIsOkToBlock()) {
@@ -109,14 +113,17 @@
     initHookAnnounce(initHookAtBeginning);
 
     coreRelease();
-    /* After this point, further calls to iocInit() are disallowed.  */
     iocState = iocBuilding;
 
     taskwdInit();
     callbackInit();
     initHookAnnounce(initHookAfterCallbackInit);
 
-    dbCaLinkInit();
+    return 0;
+}
+
+static int iocBuild_2(void)
+{
     initHookAnnounce(initHookAfterCaLinkInit);
 
     initDrvSup();
@@ -147,9 +154,11 @@
 
     initialProcess();
     initHookAnnounce(initHookAfterInitialProcess);
+    return 0;
+}
 
-    /* Start CA server threads */
-    rsrv_init();
+static int iocBuild_3(void)
+{
     initHookAnnounce(initHookAfterCaServerInit);
 
     iocState = iocBuilt;
@@ -157,6 +166,39 @@
     return 0;
 }
 
+int iocBuild(void)
+{
+    int status;
+
+    status = iocBuild_1();
+    if (status) return status;
+
+    dbCaLinkInit();
+
+    status = iocBuild_2();
+    if (status) return status;
+
+    /* Start CA server threads */
+    rsrv_init();
+
+    status = iocBuild_3();
+    return status;
+}
+
+int iocBuildNoCA(void)
+{
+    int status;
+
+    status = iocBuild_1();
+    if (status) return status;
+
+    status = iocBuild_2();
+    if (status) return status;
+
+    status = iocBuild_3();
+    return status;
+}
+
 int iocRun(void)
 {
     if (iocState != iocPaused && iocState != iocBuilt) {
@@ -599,8 +641,21 @@
     }
 }
 
-static void exitDatabase(void *dummy)
+int iocShutdown(void)
 {
+    if (iocState == iocVirgin || iocState == iocStopped) return 0;
     iterateRecords(doCloseLinks, NULL);
+    scanShutdown();
+    callbackShutdown();
+    asShutdown();
+    iocshFree();
+    taskwdShutdown();
+    errlogShutdown();
     iocState = iocStopped;
+    return 0;
+}
+
+static void exitDatabase(void *dummy)
+{
+    iocShutdown();
 }

=== modified file 'src/ioc/misc/iocInit.h'
--- src/ioc/misc/iocInit.h	2009-06-10 20:19:32 +0000
+++ src/ioc/misc/iocInit.h	2013-12-14 18:55:29 +0000
@@ -19,8 +19,10 @@
 
 epicsShareFunc int iocInit(void);
 epicsShareFunc int iocBuild(void);
+epicsShareFunc int iocBuildNoCA(void);
 epicsShareFunc int iocRun(void);
 epicsShareFunc int iocPause(void);
+epicsShareFunc int iocShutdown(void);
 
 #ifdef __cplusplus
 }

=== modified file 'src/libCom/as/asLib.h'
--- src/libCom/as/asLib.h	2010-12-17 16:50:52 +0000
+++ src/libCom/as/asLib.h	2013-12-14 18:55:29 +0000
@@ -222,6 +222,7 @@
 /*following is "friend" function*/
 epicsShareFunc void * epicsShareAPI asCalloc(size_t nobj,size_t size);
 epicsShareFunc char * epicsShareAPI asStrdup(unsigned char *str);
+epicsShareFunc void asFreeAll(ASBASE *pasbase);
 #ifdef __cplusplus
 }
 #endif

=== modified file 'src/libCom/as/asLibRoutines.c'
--- src/libCom/as/asLibRoutines.c	2012-07-07 18:50:55 +0000
+++ src/libCom/as/asLibRoutines.c	2013-12-14 18:55:29 +0000
@@ -51,7 +51,6 @@
 static long asComputeAllAsgPvt(void);
 static long asComputeAsgPvt(ASG *pasg);
 static long asComputePvt(ASCLIENTPVT asClientPvt);
-static void asFreeAll(ASBASE *pasbase);
 static UAG *asUagAdd(const char *uagName);
 static long asUagAddUser(UAG *puag,const char *user);
 static HAG *asHagAdd(const char *hagName);
@@ -1015,7 +1014,7 @@
     return(0);
 }
 

-static void asFreeAll(ASBASE *pasbase)
+void asFreeAll(ASBASE *pasbase)
 {
     UAG		*puag;
     UAGNAME	*puagname;

=== modified file 'src/libCom/error/errlog.c'
--- src/libCom/error/errlog.c	2013-06-28 17:35:43 +0000
+++ src/libCom/error/errlog.c	2013-12-14 18:55:29 +0000
@@ -41,9 +41,13 @@
 /*Declare storage for errVerbose */
 epicsShareDef int errVerbose = 0;
 
-static void exitHandler(void *);
+static epicsThreadOnceId errlogOnceFlag = EPICS_THREAD_ONCE_INIT;
+
+static void errlogExitHandler(void *);
 static void errlogThread(void);
 
+static void errlogInit1(int bufsize);
+
 static char *msgbufGetFree(int noConsoleMessage);
 static void msgbufSetSize(int size); /* Send 'size' chars plus trailing '\0' */
 static char *msgbufGetSend(int *noConsoleMessage);
@@ -70,8 +74,8 @@
     epicsEventId waitForFlush; /*errlogFlush waits for this*/
     epicsEventId flush; /*errlogFlush sets errlogThread does a Try*/
     epicsMutexId flushLock;
-    epicsEventId waitForExit; /*exitHandler waits for this*/
-    int          atExit;      /*TRUE when exitHandler is active*/
+    epicsEventId waitForExit; /*errlogExitHandler waits for this*/
+    int          atExit;      /*TRUE when errlogExitHandler is active*/
     ELLLIST      listenerList;
     ELLLIST      msgQueue;
     msgNode      *pnextSend;
@@ -116,7 +120,7 @@
         return 0;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     isOkToBlock = epicsThreadIsOkToBlock();
 
     if (pvtData.atExit || (isOkToBlock && pvtData.toConsole)) {
@@ -153,7 +157,7 @@
         return 0;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     if (pvtData.atExit)
         return 0;
     isOkToBlock = epicsThreadIsOkToBlock();
@@ -191,7 +195,7 @@
         return 0;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     va_start(pvar, pFormat);
     nchar = errlogVprintfNoConsole(pFormat, pvar);
     va_end(pvar);
@@ -210,7 +214,7 @@
         return 0;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     if (pvtData.atExit)
         return 0;
 
@@ -237,7 +241,7 @@
         return 0;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     if (pvtData.sevToLog > severity)
         return 0;
 
@@ -270,7 +274,7 @@
         return 0;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     if (pvtData.atExit)
         return 0;
 
@@ -295,7 +299,7 @@
 epicsShareFunc char * epicsShareAPI errlogGetSevEnumString(
     const errlogSevEnum severity)
 {
-    errlogInit(0);
+    errlogInit1(0);
     if (severity > 3)
         return "unknown";
     return errlogSevEnumString[severity];
@@ -304,13 +308,13 @@
 epicsShareFunc void epicsShareAPI errlogSetSevToLog(
     const errlogSevEnum severity)
 {
-    errlogInit(0);
+    errlogInit1(0);
     pvtData.sevToLog = severity;
 }
 
 epicsShareFunc errlogSevEnum epicsShareAPI errlogGetSevToLog(void)
 {
-    errlogInit(0);
+    errlogInit1(0);
     return pvtData.sevToLog;
 }
 
@@ -319,7 +323,7 @@
 {
     listenerNode *plistenerNode;
 
-    errlogInit(0);
+    errlogInit1(0);
     if (pvtData.atExit)
         return;
 
@@ -338,7 +342,7 @@
     listenerNode *plistenerNode;
     int count = 0;
 
-    errlogInit(0);
+    errlogInit1(0);
     if (!pvtData.atExit)
         epicsMutexMustLock(pvtData.listenerLock);
 
@@ -367,14 +371,14 @@
 
 epicsShareFunc int epicsShareAPI eltc(int yesno)
 {
-    errlogInit(0);
+    errlogInit1(0);
     pvtData.toConsole = yesno;
     return 0;
 }
 
 epicsShareFunc int errlogSetConsole(FILE *stream)
 {
-    errlogInit(0);
+    errlogInit1(0);
     pvtData.console = stream ? stream : stderr;
     return 0;
 }
@@ -394,7 +398,7 @@
         return;
     }
 
-    errlogInit(0);
+    errlogInit1(0);
     isOkToBlock = epicsThreadIsOkToBlock();
     if (status == 0)
         status = errno;
@@ -447,8 +451,9 @@
 }
 
 
-static void exitHandler(void *pvt)
+void errlogShutdown(void)
 {
+    if (pvtData.atExit) return;
     pvtData.atExit = 1;
     epicsEventSignal(pvtData.waitForWork);
     epicsEventMustWait(pvtData.waitForExit);
@@ -461,6 +466,13 @@
     epicsMutexDestroy(pvtData.msgQueueLock);
     epicsEventDestroy(pvtData.waitForWork);
     epicsEventDestroy(pvtData.waitForExit);
+
+    epicsThreadOnceReset(&errlogOnceFlag);
+}
+
+static void errlogExitHandler(void *pvt)
+{
+    errlogShutdown();
 }
 
 struct initArgs {
@@ -505,7 +517,6 @@
 
 epicsShareFunc int epicsShareAPI errlogInit2(int bufsize, int maxMsgSize)
 {
-    static epicsThreadOnceId errlogOnceFlag = EPICS_THREAD_ONCE_INIT;
     struct initArgs config;
 
     if (pvtData.atExit)
@@ -529,9 +540,15 @@
 
 epicsShareFunc int epicsShareAPI errlogInit(int bufsize)
 {
+    pvtData.atExit = 0;
     return errlogInit2(bufsize, MAX_MESSAGE_SIZE);
 }
 
+static void errlogInit1(int bufsize)
+{
+    errlogInit2(bufsize, MAX_MESSAGE_SIZE);
+}
+
 epicsShareFunc void epicsShareAPI errlogFlush(void)
 {
     int count;
@@ -561,7 +578,7 @@
     int noConsoleMessage;
     char *pmessage;
 
-    epicsAtExit(exitHandler,0);
+    epicsAtExit(errlogExitHandler,0);
     while (TRUE) {
         epicsEventMustWait(pvtData.waitForWork);
         while ((pmessage = msgbufGetSend(&noConsoleMessage))) {

=== modified file 'src/libCom/error/errlog.h'
--- src/libCom/error/errlog.h	2013-06-28 17:35:43 +0000
+++ src/libCom/error/errlog.h	2013-12-14 18:55:29 +0000
@@ -64,6 +64,7 @@
 epicsShareFunc int errlogSetConsole(FILE *stream);
 
 epicsShareFunc int epicsShareAPI errlogInit(int bufsize);
+epicsShareFunc void epicsShareAPI errlogShutdown(void);
 epicsShareFunc int epicsShareAPI errlogInit2(int bufsize, int maxMsgSize);
 epicsShareFunc void epicsShareAPI errlogFlush(void);
 

=== modified file 'src/libCom/iocsh/iocsh.cpp'
--- src/libCom/iocsh/iocsh.cpp	2012-01-11 21:12:15 +0000
+++ src/libCom/iocsh/iocsh.cpp	2013-12-14 18:55:29 +0000
@@ -198,20 +198,22 @@
  */
 void epicsShareAPI iocshFree(void) 
 {
-    struct iocshCommand *pc, *nc;
-    struct iocshVariable *pv, *nv;
+    struct iocshCommand *pc;
+    struct iocshVariable *pv;
 
     iocshTableLock ();
     for (pc = iocshCommandHead ; pc != NULL ; ) {
-        nc = pc->next;
+        struct iocshCommand * nc = pc->next;
         free (pc);
         pc = nc;
     }
     for (pv = iocshVariableHead ; pv != NULL ; ) {
-        nv = pv->next;
+        struct iocshVariable *nv = pv->next;
         free (pv);
         pv = nv;
     }
+    iocshCommandHead = NULL;
+    iocshVariableHead = NULL;
     iocshTableUnlock ();
 }
 

=== modified file 'src/libCom/misc/epicsExit.c'
--- src/libCom/misc/epicsExit.c	2012-07-17 19:33:31 +0000
+++ src/libCom/misc/epicsExit.c	2013-12-14 18:55:29 +0000
@@ -23,7 +23,9 @@
  */
 
 #include <stdlib.h>
+#include <stdio.h>
 #include <errno.h>
+#include <string.h>
 
 #define epicsExportSharedSymbols
 #include "ellLib.h"
@@ -36,12 +38,15 @@
     ELLNODE         node;
     epicsExitFunc   func;
     void            *arg;
+    char            name[1];
 }exitNode;
 
 typedef struct exitPvt {
     ELLLIST         list;
 } exitPvt;
 
+int atExitDebug = 0;
+
 static epicsThreadOnceId exitPvtOnce = EPICS_THREAD_ONCE_INIT;
 static exitPvt * pExitPvtPerProcess = 0;
 static epicsMutexId exitPvtLock = 0;
@@ -66,15 +71,23 @@
 {
     exitPvtPerThread = epicsThreadPrivateCreate ();
     assert ( exitPvtPerThread );
-    pExitPvtPerProcess = createExitPvt ();
-    assert ( pExitPvtPerProcess );
     exitPvtLock = epicsMutexMustCreate ();
 }
 
+static void epicsExitInit(void)
+{
+    epicsThreadOnce ( & exitPvtOnce, exitPvtOnceFunc, 0 );
+}
+
 static void epicsExitCallAtExitsPvt(exitPvt *pep)
 {
     exitNode *pexitNode;
+
     while ( ( pexitNode = (exitNode *) ellLast ( & pep->list ) ) ) {
+        if (atExitDebug && pexitNode->name[0])
+            fprintf(stderr, "atExit %s(%p)\n", pexitNode->name, pexitNode->arg);
+        else if(atExitDebug)
+            fprintf(stderr, "atExit %p(%p)\n", pexitNode->func, pexitNode->arg);
         pexitNode->func ( pexitNode->arg );
         ellDelete ( & pep->list, & pexitNode->node );
         free ( pexitNode );
@@ -84,7 +97,8 @@
 epicsShareFunc void epicsExitCallAtExits(void)
 {
     exitPvt * pep = 0;
-    epicsThreadOnce ( & exitPvtOnce, exitPvtOnceFunc, 0 );
+
+    epicsExitInit ();
     epicsMutexMustLock ( exitPvtLock );
     if ( pExitPvtPerProcess ) {
         pep = pExitPvtPerProcess;
@@ -100,7 +114,8 @@
 epicsShareFunc void epicsExitCallAtThreadExits(void)
 {
     exitPvt * pep;
-    epicsThreadOnce ( & exitPvtOnce, exitPvtOnceFunc, 0 );
+
+    epicsExitInit ();
     pep = epicsThreadPrivateGet ( exitPvtPerThread );
     if ( pep ) {
         epicsExitCallAtExitsPvt ( pep );
@@ -109,14 +124,16 @@
     }
 }
 
-static int epicsAtExitPvt(exitPvt *pep, epicsExitFunc func, void *arg)
+static int epicsAtExitPvt(exitPvt *pep, epicsExitFunc func, void *arg, const char *name)
 {
     int status = -1;
-    exitNode * pExitNode
-        = calloc ( 1, sizeof( *pExitNode ) );
+    exitNode * pExitNode = calloc ( 1, sizeof( *pExitNode ) + (name?strlen(name):0) );
+
     if ( pExitNode ) {
         pExitNode->func = func;
         pExitNode->arg = arg;
+        if(name)
+            strcpy(pExitNode->name, name);
         ellAdd ( & pep->list, & pExitNode->node );
         status = 0;
     }
@@ -126,7 +143,8 @@
 epicsShareFunc int epicsAtThreadExit(epicsExitFunc func, void *arg)
 {
     exitPvt * pep;
-    epicsThreadOnce ( & exitPvtOnce, exitPvtOnceFunc, 0 );
+
+    epicsExitInit ();
     pep = epicsThreadPrivateGet ( exitPvtPerThread );
     if ( ! pep ) {
         pep = createExitPvt ();
@@ -135,16 +153,20 @@
         }
         epicsThreadPrivateSet ( exitPvtPerThread, pep );
     }
-    return epicsAtExitPvt ( pep, func, arg );
+    return epicsAtExitPvt ( pep, func, arg, NULL );
 }
 
-epicsShareFunc int epicsAtExit(epicsExitFunc func, void *arg)
+epicsShareFunc int epicsAtExit3(epicsExitFunc func, void *arg, const char* name)
 {
     int status = -1;
-    epicsThreadOnce ( & exitPvtOnce, exitPvtOnceFunc, 0 );
+
+    epicsExitInit ();
     epicsMutexMustLock ( exitPvtLock );
+    if ( !pExitPvtPerProcess ) {
+        pExitPvtPerProcess = createExitPvt ();
+    }
     if ( pExitPvtPerProcess ) {
-        status = epicsAtExitPvt ( pExitPvtPerProcess, func, arg );
+        status = epicsAtExitPvt ( pExitPvtPerProcess, func, arg, name );
     }
     epicsMutexUnlock ( exitPvtLock );
     return status;
@@ -153,6 +175,10 @@
 epicsShareFunc void epicsExit(int status)
 {
     epicsExitCallAtExits();
-    epicsThreadSleep(1.0);
+    epicsThreadSleep(0.1);
     exit(status);
 }
+
+#include "epicsExport.h"
+
+epicsExportAddress(int,atExitDebug);

=== modified file 'src/libCom/misc/epicsExit.h'
--- src/libCom/misc/epicsExit.h	2012-07-17 19:33:31 +0000
+++ src/libCom/misc/epicsExit.h	2013-12-14 18:55:29 +0000
@@ -19,7 +19,8 @@
 
 epicsShareFunc void epicsExit(int status);
 epicsShareFunc void epicsExitCallAtExits(void);
-epicsShareFunc int epicsAtExit(epicsExitFunc func, void *arg);
+epicsShareFunc int epicsAtExit3(epicsExitFunc func, void *arg, const char* name);
+#define epicsAtExit(F,A) epicsAtExit3(F,A,#F)
 
 epicsShareFunc void epicsExitCallAtThreadExits(void);
 epicsShareFunc int epicsAtThreadExit(epicsExitFunc func, void *arg);

=== modified file 'src/libCom/osi/epicsThread.h'
--- src/libCom/osi/epicsThread.h	2012-07-06 21:33:10 +0000
+++ src/libCom/osi/epicsThread.h	2013-12-14 18:55:29 +0000
@@ -57,6 +57,9 @@
 epicsShareFunc void epicsShareAPI epicsThreadOnce(
     epicsThreadOnceId *id, EPICSTHREADFUNC, void *arg);
 
+epicsShareFunc void epicsShareAPI epicsThreadOnceReset(
+    epicsThreadOnceId *id);
+
 epicsShareFunc void epicsShareAPI epicsThreadExitMain(void);
 
 epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate (

=== modified file 'src/libCom/osi/os/RTEMS/osdThread.c'
--- src/libCom/osi/os/RTEMS/osdThread.c	2012-07-31 19:04:38 +0000
+++ src/libCom/osi/os/RTEMS/osdThread.c	2013-12-14 18:55:29 +0000
@@ -505,6 +505,15 @@
     epicsMutexUnlock(onceMutex);
 }
 
+epicsShareFunc void epicsShareAPI epicsThreadOnceReset(
+    epicsThreadOnceId *id)
+{
+    if (!initialized) epicsThreadInit();
+    epicsMutexMustLock(onceMutex);
+    *id = EPICS_THREAD_ONCE_INIT;
+    epicsMutexUnlock(onceMutex);
+}
+
 /*
  * Thread private storage implementation based on the vxWorks
  * implementation by Andrew Johnson APS/ASD.

=== modified file 'src/libCom/osi/os/WIN32/osdThread.c'
--- src/libCom/osi/os/WIN32/osdThread.c	2013-06-07 23:08:38 +0000
+++ src/libCom/osi/os/WIN32/osdThread.c	2013-12-14 18:55:29 +0000
@@ -1058,6 +1058,17 @@
     LeaveCriticalSection ( & pGbl->mutex );
 }
 
+
+epicsShareFunc void epicsShareAPI epicsThreadOnceReset(
+    epicsThreadOnceId *id)
+{
+    win32ThreadGlobal * pGbl = fetchWin32ThreadGlobal ();
+    assert ( pGbl );
+    EnterCriticalSection ( & pGbl->mutex );
+    *id = EPICS_THREAD_ONCE_INIT;
+    LeaveCriticalSection ( & pGbl->mutex );
+}
+
 /*
  * epicsThreadPrivateCreate ()
  */

=== modified file 'src/libCom/osi/os/posix/osdThread.c'
--- src/libCom/osi/os/posix/osdThread.c	2012-09-20 19:55:32 +0000
+++ src/libCom/osi/os/posix/osdThread.c	2013-12-14 18:55:29 +0000
@@ -465,6 +465,25 @@
     checkStatusQuit(status,"pthread_mutex_unlock","epicsThreadOnce");
 }
 
+epicsShareFunc void epicsShareAPI epicsThreadOnceReset(
+    epicsThreadOnceId *id)
+{
+    int status;
+
+    epicsThreadInit();
+    status = mutexLock(&onceLock);
+    if(status) {
+        fprintf(stderr,"epicsThreadOnceReset: pthread_mutex_lock returned %s.\n",
+            strerror(status));
+        exit(-1);
+    }
+
+    *id = EPICS_THREAD_ONCE_INIT;
+
+    status = pthread_mutex_unlock(&onceLock);
+    checkStatusQuit(status,"pthread_mutex_unlock","epicsThreadOnceReset");
+}
+
 epicsShareFunc epicsThreadId epicsShareAPI epicsThreadCreate(const char *name,
     unsigned int priority, unsigned int stackSize,
     EPICSTHREADFUNC funptr,void *parm)

=== modified file 'src/libCom/osi/os/vxWorks/osdThread.c'
--- src/libCom/osi/os/vxWorks/osdThread.c	2013-06-07 23:08:38 +0000
+++ src/libCom/osi/os/vxWorks/osdThread.c	2013-12-14 18:55:29 +0000
@@ -167,7 +167,19 @@
     }
     semGive(epicsThreadOnceMutex);
 }
-

+
+epicsShareFunc void epicsShareAPI epicsThreadOnceReset(
+    epicsThreadOnceId *id)
+{
+    int result;
+
+    epicsThreadInit();
+    result = semTake(epicsThreadOnceMutex, WAIT_FOREVER);
+    assert(result == OK);
+    *id = EPICS_THREAD_ONCE_INIT;
+    semGive(epicsThreadOnceMutex);
+}
+
 static void createFunction(EPICSTHREADFUNC func, void *parm)
 {
     int tid = taskIdSelf();

=== modified file 'src/libCom/taskwd/taskwd.c'
--- src/libCom/taskwd/taskwd.c	2012-05-04 18:38:59 +0000
+++ src/libCom/taskwd/taskwd.c	2013-12-14 18:55:29 +0000
@@ -70,10 +70,11 @@
 
 /* Watchdog task control */
 static volatile enum {
-    twdctlInit, twdctlRun, twdctlDisable, twdctlExit
+    ctlInit, ctlRun, ctlDisable, ctlExit
 } twdCtl;
 static epicsEventId loopEvent;
 static epicsEventId exitEvent;
+static epicsThreadOnceId twdOnceFlag = EPICS_THREAD_ONCE_INIT;
 
 /* Task delay times (seconds) */
 #define TASKWD_DELAY    6.0
@@ -90,8 +91,8 @@
     struct tNode *pt;
     struct mNode *pm;
 
-    while (twdCtl != twdctlExit) {
-        if (twdCtl == twdctlRun) {
+    while (twdCtl != ctlExit) {
+        if (twdCtl == ctlRun) {
             epicsMutexMustLock(tLock);
             pt = (struct tNode *)ellFirst(&tList);
             while (pt) {
@@ -127,12 +128,26 @@
     epicsEventSignal(exitEvent);
 }
 
-
-static void twdShutdown(void *arg)
+void taskwdShutdown(void)
 {
-    twdCtl = twdctlExit;
+    if (twdCtl == ctlExit) return;
+    twdCtl = ctlExit;
+
     epicsEventSignal(loopEvent);
     epicsEventWait(exitEvent);
+
+    epicsEventDestroy(exitEvent);
+    epicsEventDestroy(loopEvent);
+    epicsMutexDestroy(fLock);
+    epicsMutexDestroy(mLock);
+    epicsMutexDestroy(tLock);
+
+    epicsThreadOnceReset(&twdOnceFlag);
+}
+
+static void twdExit(void *arg)
+{
+    taskwdShutdown();
 }
 
 static void twdInitOnce(void *arg)
@@ -143,7 +158,7 @@
     mLock = epicsMutexMustCreate();
     fLock = epicsMutexMustCreate();
 
-    twdCtl = twdctlRun;
+    twdCtl = ctlRun;
     loopEvent = epicsEventMustCreate(epicsEventEmpty);
     exitEvent = epicsEventMustCreate(epicsEventEmpty);
 
@@ -153,12 +168,11 @@
     if (tid == 0)
         cantProceed("Failed to spawn task watchdog thread\n");
 
-    epicsAtExit(twdShutdown, NULL);
+    epicsAtExit(twdExit, NULL);
 }
 
 void taskwdInit(void)
 {
-    static epicsThreadOnceId twdOnceFlag = EPICS_THREAD_ONCE_INIT;
     epicsThreadOnce(&twdOnceFlag, twdInitOnce, NULL);
 }
 

=== modified file 'src/libCom/taskwd/taskwd.h'
--- src/libCom/taskwd/taskwd.h	2010-10-05 19:27:37 +0000
+++ src/libCom/taskwd/taskwd.h	2013-12-14 18:55:29 +0000
@@ -28,6 +28,7 @@
 
 /* Initialization, optional */
 epicsShareFunc void taskwdInit(void);
+epicsShareFunc void taskwdShutdown(void);
 
 
 /* For tasks to be monitored */

=== modified file 'src/libCom/test/epicsExitTest.c'
--- src/libCom/test/epicsExitTest.c	2009-04-08 22:39:27 +0000
+++ src/libCom/test/epicsExitTest.c	2013-12-14 18:55:29 +0000
@@ -59,12 +59,20 @@
     testDiag("%s starting", pinfo->name);
     pinfo->terminate = epicsEventMustCreate(epicsEventEmpty);
     pinfo->terminated = epicsEventMustCreate(epicsEventEmpty);
-    epicsAtExit(atExit, pinfo);
-    epicsAtThreadExit(atThreadExit, pinfo);
+    testOk(!epicsAtExit(atExit, pinfo), "Registered atExit(%p)", pinfo);
+    testOk(!epicsAtThreadExit(atThreadExit, pinfo),
+        "Registered atThreadExit(%p)", pinfo);
     testDiag("%s waiting for atExit", pinfo->name);
     epicsEventMustWait(pinfo->terminate);
 }
 
+int count;
+
+static void counter(void *pvt)
+{
+    count++;
+}
+
 static void mainExit(void *pvt)
 {
     testPass("Reached mainExit");
@@ -77,16 +85,23 @@
     info *pinfoA = (info *)calloc(1, sizeof(info));
     info *pinfoB = (info *)calloc(1, sizeof(info));
 
-    testPlan(7);
-
-    epicsAtExit(mainExit, NULL);
+    testPlan(15);
+
+    testOk(!epicsAtExit(counter, NULL), "Registered counter()");
+    count = 0;
+    epicsExitCallAtExits();
+    testOk(count == 1, "counter() called once");
+    epicsExitCallAtExits();
+    testOk(count == 1, "unregistered counter() not called");
+
+    testOk(!epicsAtExit(mainExit, NULL), "Registered mainExit()");
 
     epicsThreadCreate("threadA", 50, stackSize, thread, pinfoA);
     epicsThreadSleep(0.1);
     epicsThreadCreate("threadB", 50, stackSize, thread, pinfoB);
     epicsThreadSleep(1.0);
 
-    testDiag("Calling epicsExit\n");
+    testDiag("Calling epicsExit");
     epicsExit(0);
     return 0;
 }

=== modified file 'src/libCom/test/epicsThreadOnceTest.c'
--- src/libCom/test/epicsThreadOnceTest.c	2012-07-06 22:47:56 +0000
+++ src/libCom/test/epicsThreadOnceTest.c	2013-12-14 18:55:29 +0000
@@ -21,6 +21,7 @@
 
 epicsThreadOnceId onceFlag = EPICS_THREAD_ONCE_INIT;
 epicsThreadOnceId twiceFlag = EPICS_THREAD_ONCE_INIT;
+epicsThreadOnceId resetFlag = EPICS_THREAD_ONCE_INIT;
 epicsMutexId lock;
 epicsEventId go;
 epicsEventId done;
@@ -80,7 +81,7 @@
     int i;
     epicsThreadId tid;
 
-    testPlan(3 + NUM_ONCE_THREADS);
+    testPlan(7 + NUM_ONCE_THREADS);
 
     go = epicsEventMustCreate(epicsEventEmpty);
     done = epicsEventMustCreate(epicsEventEmpty);
@@ -112,6 +113,17 @@
     } while (!epicsThreadIsSuspended(tid));
     testPass("Recursive epicsThreadOnce() detected");
 
+    initCount = 0;
+    epicsThreadOnce(&resetFlag, &onceInit, NULL);
+    testOk1(initCount==1);
+    epicsThreadOnce(&resetFlag, &onceInit, NULL);
+    testOk1(initCount==1);
+    epicsThreadOnceReset(&resetFlag);
+    epicsThreadOnce(&resetFlag, &onceInit, NULL);
+    testOk1(initCount==2);
+    epicsThreadOnce(&resetFlag, &onceInit, NULL);
+    testOk1(initCount==2);
+
     eltc(1);
     return testDone();
 }