Merge lp:~ralph-lange/epics-base/thread-hooks into lp:~epics-core/epics-base/3.15

[Ralph Lange]

Adds two interfaces for user hooks that are being called from epicsThread:

epicsThreadAddStartHook(func) and epicsThreadAddExitHook(func) each add a user function to an internal list of hooks that are being called from the thread context, immediately before or after each thread's main routine runs.
For Linux, the default start hook adds the thread's Linux LWP ID to the thread info block and sets the Linux name to match the epics thread name, and epicsShowThread() is overwritten to print this LWP ID instead of the POSIX thread ID.

epicsThreadMap(func) immediately calls the user supplied func for each currently existing thread.

Caveat: The WIN32 implementation needs the start/exit hooks calls added - I simply do not understand this implementation well enough to do that.
The vxWorks implementation of epicsThreadMap() only works for up to 2048 threads, and needs to be made dynamic.

[mdavidsaver (mdavidsaver)]

> /* As we're only ever inserting hooks at the head of the list, forward traversing is safe */
> pHook = (epicsThreadHook *) ellFirst(list);

It can be unlocked to traverse, but the mutex should be locked when fetching the head pointer.

[Andrew Johnson (anj)]

Should the AtThreadExits run before the ExitHooks? You have this in several places:
  + epicsThreadRunExitHooks(pthreadInfo);
    epicsExitCallAtThreadExits ();
My feeling is that task-specific clean-up should happen before the generic task clean-up, so things nest nicely.

I think the Win32 calls to the epicsThreadRun*Hooks() routines belong in the epicsWin32ThreadEntry() routine, and epicsThreadHooksInit() should be called towards the bottom of fetchWin32ThreadGlobal() after the atexit() call but before initCompleted gets set (there doesn't look to be anything in the mutex creation code that would cause problems there).

Then with some tests in libCom/test/epicsThreadTest.cpp and comments in the Release Notes I think this should be good to go. Eventually I'll want the API documenting in the AppDevGuide too, but not for this release.

Thanks,

- Andrew

[Ralph Lange (ralph-lange)]

> > /* As we're only ever inserting hooks at the head of the list, forward
> traversing is safe */
> > pHook = (epicsThreadHook *) ellFirst(list);
>
> It can be unlocked to traverse, but the mutex should be locked when fetching
> the head pointer.

That's because the pointer modifying operations (in elllib.c) are not atomic?
Other than that, I would assume the map function could either get or miss the element currently being inserted or added, and both would be ok.

[lp:~ralph-lange/epics-base/thread-hooks]

12306. By Ralph Lange on 2012-07-02

libCom/osi: Fix nesting for epicsThreadHooks, add WIN32 implementation, lock traversal
- call (generic) exit hook after calling the (specific) epicsExitCallAtThreadExits()
- for start hooks added as 1-2-3, run exit hooks in opposite order: 3-2-1
- add calls to hooks module to WIN32 osdThread.c
- add lock/unlock to hook list traversal

12307. By Ralph Lange on 2012-07-02

libCom/osi: Make vxWorks implementation of epicsThreadMap() safe and dynamic

12308. By Ralph Lange on 2012-07-02

libCom/test: Add test for epicsThreadHooks

12309. By Ralph Lange on 2012-07-02

documentation: Add epicsThreadHooks info to RELEASE NOTES

[Ralph Lange (ralph-lange)]

I think I pretty much worked down the list.

[Andrew Johnson (anj)]

Hi Ralph,

Thanks.

Sorry, a couple more issues, then I think I'm done:

The name 'epicsShowThreadInfo' implies this routine is part of the epicsShow subsystem. The original name 'showThreadInfo' was static and thus only used internally to the posix implementation, but if you're making it visible I think it needs to match the other epicsThread... names. Maybe name it epicsThreadShowPrivate(), which matches the Win32 implementation (although there the routine is static).

If you don't intend that routine to be called by user code the declaration for it doesn't have to appear in any header files either, it could just appear in the posix/osdThread.c file where it's used. I would also prefer that it take an epicsThreadId argument instead of the epicsThreadOSD* pointer; that should remain for internal use only (epicsThread.h has a typedef making the two identical).

- Andrew

[Andrew Johnson (anj)]

Sorry, a few more, although they're minor.

Please don't mark new API routines with epicsShareAPI; I'm trying to phase that out since almost nobody needs to use Pascal calling conventions on Windows any more. It's really only needed for routines that may be called by Visual Basic (if there are any users of that left), so it may still be necessary on CA API routines but not really on most others.

This may be more of a discussion point. Here's my output from epicsThreadShowAll:

epics> epicsThreadShowAll
            NAME EPICS ID LWP ID OSIPRI OSSPRI STATE
          _main_ 0x10b1060 0 0 0 OK
          errlog 0x113c830 20849 10 0 OK
          taskwd 0x113cc80 20850 10 0 OK
      timerQueue 0x113d5e0 20851 70 0 OK
           cbLow 0x113d8c0 20852 59 0 OK
        cbMedium 0x113db90 20853 64 0 OK
          cbHigh 0x113de60 20854 71 0 OK
        dbCaLink 0x113e300 20855 50 0 OK
        scanOnce 0x113ea10 20856 70 0 OK
         scan-10 0x114d670 20857 60 0 OK
          scan-5 0x114d8b0 20858 61 0 OK
          scan-2 0x114daf0 20859 62 0 OK
          scan-1 0x114dd30 20860 63 0 OK
        scan-0.5 0x114df70 20861 64 0 OK
        scan-0.2 0x114e1b0 20862 65 0 OK
        scan-0.1 0x114e3f0 20863 66 0 OK
         CAS-TCP 0x114ebc0 20864 18 0 OK
      CAS-beacon 0x7f043c000990 20865 17 0 OK
         CAS-UDP 0x7f0440000970 20866 16 0 OK

The _main_ thread is not started using epicsThreadCreate() and thus never runs the thread_hook, so we don't know its LWP ID. It would be nice if we could see it here. The _main_ thread gets its epicsThreadOSD structure created and added to pthreadList inside the once() routine in od/posix/osdThread.c before epicsThreadHooksInit() is run. The Init() routine cannot call any other epicsThread... routines such as epicsThreadGetIdSelf() because once() hasn't finished yet. Maybe epicsThreadHooksInit() could take main's epicsThreadId as an argument and (if non-zero) call the StartHook for it?

I think we can safely forget about calling _main_'s ExitHook, which brings up the interesting question of whether ExitHook is actually needed -- any StartHook can register an epicsAtThreadExit() routine if it needs to do clean-up. If we were to remove the ExitHooks we would never have problems with an ExitHook being called for a thread for which where there was no corresponding StartHook. It also means that the ExitHook routines would be run in the reverse order of the StartHook routines, whereas at the moment the order is the same.

- Andrew

[Ralph Lange (ralph-lange)]

Hi Andrew,

> The name 'epicsShowThreadInfo' implies this routine is part of the epicsShow
> subsystem. The original name 'showThreadInfo' was static and thus only used
> internally to the posix implementation, but if you're making it visible I
> think it needs to match the other epicsThread... names. Maybe name it
> epicsThreadShowPrivate(), which matches the Win32 implementation (although
> there the routine is static).

Will do.
On a closely related issue, I am quite unhappy with the confusing names of the public API functions

epicsThreadPrivateCreate
epicsThreadPrivateDelete
epicsThreadPrivateSet
epicsThreadPrivateGet

that are - unlike the other funtions that contain "Private" - *not* private, but public functions.
Also, they implement what is more commonly called "thread local storage".
Do you think it would be worthwhile renaming them to *Local* and deprecate the existing names, adding macros to continue supporting them?!

> If you don't intend that routine to be called by user code the declaration for
> it doesn't have to appear in any header files either, it could just appear in
> the posix/osdThread.c file where it's used.

True. Will do.

> I would also prefer that it take
> an epicsThreadId argument instead of the epicsThreadOSD* pointer; that should
> remain for internal use only (epicsThread.h has a typedef making the two
> identical).

If (after the mentioned change) the function is private, not in any header file, and any implementation would have to cast the argument to an epicsThreadOSD* pointer anyway, I don't see the point in using the opaque type.

~Ralph

[lp:~ralph-lange/epics-base/thread-hooks]

12310. By Ralph Lange on 2012-07-03

libCom/osi: Streamline epicsThreadShowInfo functions between implementations
- epicsThreadShowInfo: private function to print headers or task line,
  not in header file, using internal pointer type

[Ralph Lange (ralph-lange)]

> Please don't mark new API routines with epicsShareAPI; I'm trying to phase
> that out [...]

Will do.

> This may be more of a discussion point. Here's my output from
> epicsThreadShowAll:
>
> epics> epicsThreadShowAll
> NAME EPICS ID LWP ID OSIPRI OSSPRI STATE
> _main_ 0x10b1060 0 0 0 OK
> errlog 0x113c830 20849 10 0 OK
> [...]
>
> The _main_ thread is not started using epicsThreadCreate() and thus never runs
> the thread_hook, so we don't know its LWP ID. It would be nice if we could
> see it here.

I had considered this, then decided against, as the POSIX implementation shows the POSIX thread id as 0, and I wanted to stay minimally invasive.
If you think this is a good idea (I do), I will add that.

> I think we can safely forget about calling _main_'s ExitHook, which brings up
> the interesting question of whether ExitHook is actually needed -- any
> StartHook can register an epicsAtThreadExit() routine if it needs to do clean-
> up. If we were to remove the ExitHooks we would never have problems with an
> ExitHook being called for a thread for which where there was no corresponding
> StartHook.

Good! That was my initial approach, and I added the ExitHook routines only out of a gut feeling that I wold have to justify if I didn't.
The argument about an exit hook being called where the start hook wasn't added at thread creation time is an excellent one.

> It also means that the ExitHook routines would be run in the
> reverse order of the StartHook routines, whereas at the moment the order is
> the same.

Not true. They are run in correct (reverse) order, which is covered by the test. Never mind.

~Ralph

[lp:~ralph-lange/epics-base/thread-hooks]

12311. By Ralph Lange on 2012-07-03

libCom/osi: Don't decorate epicsThreadHook functions with epicsShareAPI

12312. By Ralph Lange on 2012-07-03

libCom/osi: Clean up epicsThreadHooks API
- remove exit hooks completely
- remove non-public functions fom header files
- add test for epicsThreadMap
- fix bugs in RTEMS and vxWorks implementation of epicsThreadMap

12313. By Ralph Lange on 2012-07-03

libCom/osi: Add default start hook for _main_ thread

[Ralph Lange (ralph-lange)]

> [...] The _main_ thread gets its epicsThreadOSD structure created and
> added to pthreadList inside the once() routine in od/posix/osdThread.c before
> epicsThreadHooksInit() is run. The Init() routine cannot call any other
> epicsThread... routines such as epicsThreadGetIdSelf() because once() hasn't
> finished yet.

I don't like all thread start hooks to inherit this limitation, so I implemented this using a second default hook, that is exclusively used for _main_.

So - if I did not miss anything - the list seems to be worked through, again.

~Ralph

[lp:~ralph-lange/epics-base/thread-hooks]

12314. By Ralph Lange on 2012-07-03

documentation: Updated RELEASE_NOTES

[Andrew Johnson (anj)]

Hi Ralph,

I made some minor changes while merging this, the most important being to properly fix the vxWorks epicsThreadInit() code so it is only run once. I also renamed a few things so the hooking API is now called epicsThreadHookAdd(), and I added the test routine to epicsRunLibComTests().

I'll do a test run on vxWorks when I'm back at work tomorrow.

Thanks,

- Andrew

review: Approve

=== modified file 'documentation/RELEASE_NOTES.html'
--- documentation/RELEASE_NOTES.html	2012-06-27 21:43:15 +0000
+++ documentation/RELEASE_NOTES.html	2012-07-03 14:42:25 +0000
@@ -15,6 +15,17 @@
 <h2 align="center">Changes between 3.14.x and 3.15.0.x</h2>
 <!-- Insert new items immediately below here ... -->
 
+<h3>New API to hook into thread creation</h3>
+
+<p>A hook API has been added allowing user-supplied functions to be called
+whenever a thread starts. The calls are made from the thread's context,
+and can be used to control additional thread properties not handled inside
+EPICS base, e.g. setting the scheduling policy or CPU affinity (on SMP
+systems).</p>
+
+<p>The API also supports a mapping operation, calling a user-supplied function
+for every thread that is currently running.</p>
+
 <h3>New scan rate units</h3>
 
 <p>Scan rates defined in the menuScan.dbd file may now be specified in seconds,

=== modified file 'src/libCom/osi/Makefile'
--- src/libCom/osi/Makefile	2012-05-03 17:19:34 +0000
+++ src/libCom/osi/Makefile	2012-07-03 14:42:25 +0000
@@ -104,6 +104,8 @@
 osdThread_CPPFLAGS += $(THREAD_CPPFLAGS_$(USE_POSIX_THREAD_PRIORITY_SCHEDULING))
 
 Com_SRCS += osdThread.c
+Com_SRCS += osdThreadExtra.c
+Com_SRCS += osdThreadHooks.c
 Com_SRCS += osdMutex.c
 Com_SRCS += osdEvent.c
 Com_SRCS += osdTime.cpp

=== modified file 'src/libCom/osi/epicsThread.h'
--- src/libCom/osi/epicsThread.h	2010-04-26 20:38:11 +0000
+++ src/libCom/osi/epicsThread.h	2012-07-03 14:42:25 +0000
@@ -3,8 +3,8 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
-* EPICS BASE Versions 3.13.7
-* and higher are distributed subject to a Software License Agreement found
+* Copyright (c) 2012 ITER Organization
+* EPICS BASE is distributed subject to a Software License Agreement found
 * in file LICENSE that is included with this distribution. 
 \*************************************************************************/
 #ifndef epicsThreadh
@@ -101,6 +101,12 @@
 epicsShareFunc void epicsShareAPI epicsThreadShow(
     epicsThreadId id,unsigned int level);
 
+/* Hooks called when a thread starts, map function called once for every thread */
+typedef void (*EPICS_THREAD_HOOK_ROUTINE)(epicsThreadId id);
+epicsShareFunc void epicsThreadHooksInit(epicsThreadId id);
+epicsShareFunc void epicsThreadAddStartHook(EPICS_THREAD_HOOK_ROUTINE hook);
+epicsShareFunc void epicsThreadMap(EPICS_THREAD_HOOK_ROUTINE func);
+
 typedef struct epicsThreadPrivateOSD * epicsThreadPrivateId;
 epicsShareFunc epicsThreadPrivateId epicsShareAPI epicsThreadPrivateCreate(void);
 epicsShareFunc void epicsShareAPI epicsThreadPrivateDelete(epicsThreadPrivateId id);

=== added file 'src/libCom/osi/os/Linux/osdThread.h'
--- src/libCom/osi/os/Linux/osdThread.h	1970-01-01 00:00:00 +0000
+++ src/libCom/osi/os/Linux/osdThread.h	2012-07-03 14:42:25 +0000
@@ -0,0 +1,46 @@
+/*************************************************************************\
+* Copyright (c) 2002 The University of Chicago, as Operator of Argonne
+*     National Laboratory.
+* Copyright (c) 2002 The Regents of the University of California, as
+*     Operator of Los Alamos National Laboratory.
+* EPICS BASE Versions 3.13.7
+* and higher are distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+\*************************************************************************/
+#ifndef osdThreadh
+#define osdThreadh
+
+#include <pthread.h>
+
+#include "shareLib.h"
+#include "ellLib.h"
+#include "epicsEvent.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+typedef struct epicsThreadOSD {
+    ELLNODE            node;
+    pthread_t          tid;
+    pid_t              lwpId;
+    pthread_attr_t     attr;
+    struct sched_param schedParam;
+    EPICSTHREADFUNC    createFunc;
+    void              *createArg;
+    epicsEventId       suspendEvent;
+    int                isSuspended;
+    int                isEpicsThread;
+    int                isFifoScheduled;
+    int                isOnThreadList;
+    unsigned int       osiPriority;
+    char               name[1];     /* actually larger */
+} epicsThreadOSD;
+
+epicsShareFunc pthread_t epicsShareAPI epicsThreadGetPosixThreadId(epicsThreadId id);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* osdThreadh */

=== added file 'src/libCom/osi/os/Linux/osdThreadExtra.c'
--- src/libCom/osi/os/Linux/osdThreadExtra.c	1970-01-01 00:00:00 +0000
+++ src/libCom/osi/os/Linux/osdThreadExtra.c	2012-07-03 14:42:25 +0000
@@ -0,0 +1,66 @@
+/*************************************************************************\
+* Copyright (c) 2002 The University of Chicago, as Operator of Argonne
+*     National Laboratory.
+* Copyright (c) 2002 The Regents of the University of California, as
+*     Operator of Los Alamos National Laboratory.
+* Copyright (c) 2012 ITER Organization
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+\*************************************************************************/
+
+/* Author:  Marty Kraimer Date:    18JAN2000 */
+
+/* This differs from the posix implementation of epicsThread by:
+ * - printing the Linux LWP ID instead of the POSIX thread ID in the show routines
+ * - installing a default thread start hook, that sets the Linux thread name to the
+ *   EPICS thread name to make it visible on OS level, and discovers the LWP ID */
+
+#include <unistd.h>
+#include <signal.h>
+#include <sys/syscall.h>
+#include <sys/types.h>
+#include <sys/prctl.h>
+
+#define epicsExportSharedSymbols
+#include "epicsStdio.h"
+#include "ellLib.h"
+#include "epicsEvent.h"
+#include "epicsThread.h"
+
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook;
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook;
+
+void epicsThreadShowInfo(epicsThreadOSD *pthreadInfo, unsigned int level)
+{
+    if(!pthreadInfo) {
+        fprintf(epicsGetStdout(),"            NAME     EPICS ID   "
+            "LWP ID   OSIPRI  OSSPRI  STATE\n");
+    } else {
+        struct sched_param param;
+        int policy;
+        int priority = 0;
+
+        if(pthreadInfo->tid) {
+            int status;
+            status = pthread_getschedparam(pthreadInfo->tid,&policy,&param);
+            if(!status) priority = param.sched_priority;
+        }
+        fprintf(epicsGetStdout(),"%16.16s %12p %8lu    %3d%8d %8.8s\n",
+             pthreadInfo->name,(void *)
+             pthreadInfo,(unsigned long)pthreadInfo->lwpId,
+             pthreadInfo->osiPriority,priority,
+             pthreadInfo->isSuspended?"SUSPEND":"OK");
+    }
+}
+
+static void thread_hook(epicsThreadOSD *pthreadInfo)
+{
+    /* Set the name of the thread's process. Limited to 16 characters. */
+    char comm[16];
+    snprintf(comm, sizeof(comm), "%s", pthreadInfo->name);
+    prctl(PR_SET_NAME, comm, 0l, 0l, 0l);
+    pthreadInfo->lwpId = syscall(SYS_gettid);
+}
+
+EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook = thread_hook;
+EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook = thread_hook;

=== modified file 'src/libCom/osi/os/RTEMS/osdThread.c'
--- src/libCom/osi/os/RTEMS/osdThread.c	2010-10-05 19:27:37 +0000
+++ src/libCom/osi/os/RTEMS/osdThread.c	2012-07-03 14:42:25 +0000
@@ -39,6 +39,8 @@
 #include "osdInterrupt.h"
 #include "epicsExit.h"
 
+epicsShareFunc void epicsThreadRunStartHooks(epicsThreadId id);
+
 /*
  * Per-task variables
  */
@@ -164,6 +166,7 @@
 {
     struct taskVar *v = (struct taskVar *)arg;
 
+    epicsThreadRunStartHooks(v->id);
     (*v->funptr)(v->parm);
     epicsExitCallAtThreadExits ();
     taskVarLock ();
@@ -235,6 +238,7 @@
         taskVarMutex = epicsMutexMustCreate ();
         rtems_task_ident (RTEMS_SELF, 0, &tid);
         setThreadInfo (tid, "_main_", NULL, NULL);
+        epicsThreadHooksInit(tid);
         initialized = 1;
         epicsThreadCreate ("ImsgDaemon", 99,
                 epicsThreadGetStackSize (epicsThreadStackSmall),
@@ -640,19 +644,17 @@
 }
 
 static void
-epicsThreadShowHeader (void)
-{
-    fprintf(epicsGetStdout(),"            PRIORITY\n");
-    fprintf(epicsGetStdout(),"    ID    EPICS RTEMS   STATE    WAIT         NAME\n");
-    fprintf(epicsGetStdout(),"+--------+-----------+--------+--------+---------------------+\n");
-}
-
-static void
 epicsThreadShowInfo (struct taskVar *v, unsigned int level)
 {
+    if (!v) {
+        fprintf(epicsGetStdout(),"            PRIORITY\n");
+        fprintf(epicsGetStdout(),"    ID    EPICS RTEMS   STATE    WAIT         NAME\n");
+        fprintf(epicsGetStdout(),"+--------+-----------+--------+--------+---------------------+\n");
+    } else {
         fprintf(epicsGetStdout(),"%9.8x", (int)v->id);
         showInternalTaskInfo (v->id);
         fprintf(epicsGetStdout()," %s\n", v->name);
+    }
 }
 
 void epicsThreadShow (epicsThreadId id, unsigned int level)
@@ -660,7 +662,7 @@
     struct taskVar *v;
 
     if (!id) {
-        epicsThreadShowHeader ();
+        epicsThreadShowInfo (NULL, level);
         return;
     }
     taskVarLock ();
@@ -674,6 +676,23 @@
     fprintf(epicsGetStdout(),"*** Thread %x does not exist.\n", (unsigned int)id);
 }
 
+void epicsThreadMap(EPICS_THREAD_HOOK_ROUTINE func)
+{
+    struct taskVar *v;
+
+    taskVarLock ();
+    /*
+     * Map tasks in the order of creation (backwards through list)
+     */
+    for (v = taskVarHead ; v != NULL && v->forw != NULL ; v = v->forw)
+        continue;
+    while (v) {
+        func ((epicsThreadId)v->id);
+        v = v->back;
+    }
+    taskVarUnlock ();
+}
+
 void epicsThreadShowAll (unsigned int level)
 {
     struct taskVar *v;

=== modified file 'src/libCom/osi/os/WIN32/osdThread.c'
--- src/libCom/osi/os/WIN32/osdThread.c	2012-02-03 00:14:01 +0000
+++ src/libCom/osi/os/WIN32/osdThread.c	2012-07-03 14:42:25 +0000
@@ -38,6 +38,8 @@
 #include "ellLib.h"
 #include "epicsExit.h"
 
+epicsShareFunc void epicsThreadRunStartHooks(epicsThreadId id);
+
 void setThreadName ( DWORD dwThreadID, LPCSTR szThreadName );
 static void threadCleanupWIN32 ( void );
 
@@ -226,6 +228,7 @@
         pWin32ThreadGlobal = 0;
         return 0;
     }
+    epicsThreadHooksInit (NULL);
 
     InterlockedExchange ( & initCompleted, 1 );
 
@@ -496,6 +499,7 @@
 
         success = TlsSetValue ( pGbl->tlsIndexThreadLibraryEPICS, pParm );
         if ( success ) {
+            epicsThreadRunStartHooks ( ( epicsThreadId ) pParm );
             /* printf ( "starting thread %d\n", pParm->id ); */
             ( *pParm->funptr ) ( pParm->parm );
             /* printf ( "terminating thread %d\n", pParm->id ); */
@@ -510,7 +514,6 @@
     }
 
     epicsExitCallAtThreadExits ();
-
     /*
      * CAUTION: !!!! the thread id might continue to be used after this thread exits !!!!
      */
@@ -947,9 +950,9 @@
 }
 
 /*
- * epicsThreadShowPrivate ()
+ * epicsThreadShowInfo ()
  */
-static void epicsThreadShowPrivate ( epicsThreadId id, unsigned level )
+static void epicsThreadShowInfo ( epicsThreadId id, unsigned level )
 {
     win32ThreadParam * pParm = ( win32ThreadParam * ) id;
 
@@ -975,6 +978,28 @@
 }
 
 /*
+ * epicsThreadMap ()
+ */
+epicsShareFunc void epicsShareAPI epicsThreadMap ( EPICS_THREAD_HOOK_ROUTINE func )
+{
+    win32ThreadGlobal * pGbl = fetchWin32ThreadGlobal ();
+    win32ThreadParam * pParm;
+
+    if ( ! pGbl ) {
+        return;
+    }
+
+    EnterCriticalSection ( & pGbl->mutex );
+
+    for ( pParm = ( win32ThreadParam * ) ellFirst ( & pGbl->threadList );
+            pParm; pParm = ( win32ThreadParam * ) ellNext ( & pParm->node ) ) {
+        func ( ( epicsThreadId ) pParm );
+    }
+
+    LeaveCriticalSection ( & pGbl->mutex );
+}
+
+/*
  * epicsThreadShowAll ()
  */
 epicsShareFunc void epicsShareAPI epicsThreadShowAll ( unsigned level )
@@ -987,11 +1012,11 @@
     }
 
     EnterCriticalSection ( & pGbl->mutex );
-    
-    epicsThreadShowPrivate ( 0, level );
-    for ( pParm = ( win32ThreadParam * ) ellFirst ( & pGbl->threadList ); 
+
+    epicsThreadShowInfo ( 0, level );
+    for ( pParm = ( win32ThreadParam * ) ellFirst ( & pGbl->threadList );
             pParm; pParm = ( win32ThreadParam * ) ellNext ( & pParm->node ) ) {
-        epicsThreadShowPrivate ( ( epicsThreadId ) pParm, level );
+        epicsThreadShowInfo ( ( epicsThreadId ) pParm, level );
     }
 
     LeaveCriticalSection ( & pGbl->mutex );
@@ -1002,8 +1027,8 @@
  */
 epicsShareFunc void epicsShareAPI epicsThreadShow ( epicsThreadId id, unsigned level )
 {
-    epicsThreadShowPrivate ( 0, level );
-    epicsThreadShowPrivate ( id, level );
+    epicsThreadShowInfo ( 0, level );
+    epicsThreadShowInfo ( id, level );
 }
 
 /*

=== added file 'src/libCom/osi/os/default/osdThreadExtra.c'
--- src/libCom/osi/os/default/osdThreadExtra.c	1970-01-01 00:00:00 +0000
+++ src/libCom/osi/os/default/osdThreadExtra.c	2012-07-03 14:42:25 +0000
@@ -0,0 +1,19 @@
+/*************************************************************************\
+* Copyright (c) 2012 ITER Organization
+*
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+\*************************************************************************/
+
+/* Author:  Ralph Lange   Date:  26 Jun 2012 */
+
+/* Null default thread hooks for all platforms that do not do anything special */
+
+#define epicsExportSharedSymbols
+#include "shareLib.h"
+
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook;
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook;
+
+EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook;
+EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook;

=== added file 'src/libCom/osi/os/default/osdThreadHooks.c'
--- src/libCom/osi/os/default/osdThreadHooks.c	1970-01-01 00:00:00 +0000
+++ src/libCom/osi/os/default/osdThreadHooks.c	2012-07-03 14:42:25 +0000
@@ -0,0 +1,74 @@
+/*************************************************************************\
+* Copyright (c) 2012 ITER Organization
+*
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+\*************************************************************************/
+
+/* Author:  Ralph Lange   Date:  28 Jun 2012 */
+
+/* Secure hooks for epicsThread */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+
+#define epicsExportSharedSymbols
+#include "ellLib.h"
+#include "epicsMutex.h"
+#include "epicsThread.h"
+
+epicsShareFunc void epicsThreadRunStartHooks(epicsThreadId id);
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook;
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook;
+
+#define checkStatusOnceReturn(status, message, method) \
+if((status)) { \
+    fprintf(stderr,"%s error %s\n",(message),strerror((status))); \
+    fprintf(stderr," %s\n",(method)); \
+    return; }
+
+typedef struct epicsThreadHook {
+    ELLNODE                   node;
+    EPICS_THREAD_HOOK_ROUTINE func;
+} epicsThreadHook;
+
+static ELLLIST startHooks = ELLLIST_INIT;
+
+/* Locking could probably be avoided, if elllist implementation was using atomic ops */
+static epicsMutexId hookLock;
+
+epicsShareFunc void epicsThreadAddStartHook(EPICS_THREAD_HOOK_ROUTINE hook)
+{
+    epicsThreadHook *pHook;
+
+    pHook = calloc(1, sizeof(epicsThreadHook));
+    if (!pHook) checkStatusOnceReturn(errno,"calloc","epicsThreadAddStartHook");
+    pHook->func = hook;
+    epicsMutexLock(hookLock);
+    ellAdd(&startHooks, &pHook->node);
+    epicsMutexUnlock(hookLock);
+}
+
+epicsShareFunc void epicsThreadHooksInit(epicsThreadId id)
+{
+    if (!hookLock) {
+        hookLock = epicsMutexMustCreate();
+        if (epicsThreadDefaultStartHook) epicsThreadAddStartHook(epicsThreadDefaultStartHook);
+    }
+    if (id && epicsThreadMainStartHook) epicsThreadMainStartHook(id);
+}
+
+epicsShareFunc void epicsThreadRunStartHooks(epicsThreadId id)
+{
+    epicsThreadHook *pHook;
+
+    epicsMutexLock(hookLock);
+    pHook = (epicsThreadHook *) ellFirst(&startHooks);
+    while (pHook) {
+        pHook->func(id);
+        pHook = (epicsThreadHook *) ellNext(&pHook->node);
+    }
+    epicsMutexUnlock(hookLock);
+}

=== modified file 'src/libCom/osi/os/posix/osdThread.c'
--- src/libCom/osi/os/posix/osdThread.c	2012-06-19 19:28:26 +0000
+++ src/libCom/osi/os/posix/osdThread.c	2012-07-03 14:42:25 +0000
@@ -3,6 +3,7 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
+* Copyright (c) 2012 ITER Organization
 * EPICS BASE is distributed subject to a Software License Agreement found
 * in file LICENSE that is included with this distribution. 
 \*************************************************************************/
@@ -34,6 +35,9 @@
 #include "epicsAssert.h"
 #include "epicsExit.h"
 
+epicsShareFunc void epicsThreadShowInfo(epicsThreadOSD *pthreadInfo, unsigned int level);
+epicsShareFunc void epicsThreadRunStartHooks(epicsThreadId id);
+
 static int mutexLock(pthread_mutex_t *id)
 {
     int status;
@@ -57,22 +61,6 @@
     int                schedPolicy;
 } commonAttr;
 
-typedef struct epicsThreadOSD {
-    ELLNODE            node;
-    pthread_t          tid;
-    pthread_attr_t     attr;
-    struct sched_param schedParam;
-    EPICSTHREADFUNC    createFunc;
-    void              *createArg;
-    epicsEventId       suspendEvent;
-    int                isSuspended;
-    int                isEpicsThread;
-    int                isFifoScheduled;
-    int                isOnThreadList;
-    unsigned int       osiPriority;
-    char               name[1];     /* actually larger */
-} epicsThreadOSD;
-
 #ifdef _POSIX_THREAD_PRIORITY_SCHEDULING
 typedef struct {
     int min_pri, max_pri;
@@ -366,6 +354,7 @@
     checkStatusQuit(status,"pthread_mutex_unlock","epicsThreadInit");
     status = atexit(epicsExitCallAtExits);
     checkStatusOnce(status,"atexit");
+    epicsThreadHooksInit(pthreadInfo);
     epicsThreadOnceCalled = 1;
 }
 

@@ -375,7 +364,7 @@
     int status;
     int oldtype;
     sigset_t blockAllSig;
- 
+
     sigfillset(&blockAllSig);
     pthread_sigmask(SIG_SETMASK,&blockAllSig,NULL);
     status = pthread_setspecific(getpthreadInfo,arg);
@@ -388,11 +377,11 @@
     pthreadInfo->isOnThreadList = 1;
     status = pthread_mutex_unlock(&listLock);
     checkStatusQuit(status,"pthread_mutex_unlock","start_routine");
+    epicsThreadRunStartHooks(pthreadInfo);
 
     (*pthreadInfo->createFunc)(pthreadInfo->createArg);
 
     epicsExitCallAtThreadExits ();
-
     free_threadInfo(pthreadInfo);
     return(0);
 }
@@ -747,27 +736,23 @@
     name[size-1] = '\0';
 }
 

-static void showThreadInfo(epicsThreadOSD *pthreadInfo,unsigned int level)
+epicsShareFunc void epicsThreadMap(EPICS_THREAD_HOOK_ROUTINE func)
 {
-    if(!pthreadInfo) {
-        fprintf(epicsGetStdout(),"            NAME     EPICS ID   "
-            "PTHREAD ID   OSIPRI  OSSPRI  STATE\n");
-    } else {
-        struct sched_param param;
-        int policy;
-        int priority = 0;
+    epicsThreadOSD *pthreadInfo;
+    int status;
 
-        if(pthreadInfo->tid) {
-            int status;
-            status = pthread_getschedparam(pthreadInfo->tid,&policy,&param);
-            if(!status) priority = param.sched_priority;
-        }
-        fprintf(epicsGetStdout(),"%16.16s %12p %12lu    %3d%8d %8.8s\n",
-             pthreadInfo->name,(void *)
-             pthreadInfo,(unsigned long)pthreadInfo->tid,
-             pthreadInfo->osiPriority,priority,
-             pthreadInfo->isSuspended?"SUSPEND":"OK");
+    epicsThreadInit();
+    status = mutexLock(&listLock);
+    checkStatus(status, "pthread_mutex_lock epicsThreadMap");
+    if (status)
+        return;
+    pthreadInfo=(epicsThreadOSD *)ellFirst(&pthreadList);
+    while (pthreadInfo) {
+        func(pthreadInfo);
+        pthreadInfo = (epicsThreadOSD *)ellNext(&pthreadInfo->node);
     }
+    status = pthread_mutex_unlock(&listLock);
+    checkStatus(status, "pthread_mutex_unlock epicsThreadMap");
 }
 
 epicsShareFunc void epicsShareAPI epicsThreadShowAll(unsigned int level)
@@ -783,7 +768,7 @@
         return;
     pthreadInfo=(epicsThreadOSD *)ellFirst(&pthreadList);
     while(pthreadInfo) {
-        showThreadInfo(pthreadInfo,level);
+        epicsThreadShowInfo(pthreadInfo,level);
         pthreadInfo=(epicsThreadOSD *)ellNext(&pthreadInfo->node);
     }
     status = pthread_mutex_unlock(&listLock);
@@ -798,7 +783,7 @@
 
     epicsThreadInit();
     if(!showThread) {
-        showThreadInfo(0,level);
+        epicsThreadShowInfo(0,level);
         return;
     }
     status = mutexLock(&listLock);
@@ -810,7 +795,7 @@
         if (((epicsThreadId)pthreadInfo == showThread)
          || ((epicsThreadId)pthreadInfo->tid == showThread)) {
             found = 1;
-            showThreadInfo(pthreadInfo,level);
+            epicsThreadShowInfo(pthreadInfo,level);
         }
         pthreadInfo=(epicsThreadOSD *)ellNext(&pthreadInfo->node);
     }
@@ -820,7 +805,6 @@
     if (!found)
         printf("Thread %#lx (%lu) not found.\n", (unsigned long)showThread, (unsigned long)showThread);
 }
-
 

 epicsShareFunc epicsThreadPrivateId epicsShareAPI epicsThreadPrivateCreate(void)
 {

=== modified file 'src/libCom/osi/os/posix/osdThread.h'
--- src/libCom/osi/os/posix/osdThread.h	2008-08-06 16:54:25 +0000
+++ src/libCom/osi/os/posix/osdThread.h	2012-07-03 14:42:25 +0000
@@ -13,12 +13,30 @@
 #include <pthread.h>
 
 #include "shareLib.h"
+#include "ellLib.h"
+#include "epicsEvent.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-epicsShareFunc pthread_t epicsShareAPI epicsThreadGetPosixThreadId ( epicsThreadId id );
+typedef struct epicsThreadOSD {
+    ELLNODE            node;
+    pthread_t          tid;
+    pthread_attr_t     attr;
+    struct sched_param schedParam;
+    EPICSTHREADFUNC    createFunc;
+    void              *createArg;
+    epicsEventId       suspendEvent;
+    int                isSuspended;
+    int                isEpicsThread;
+    int                isFifoScheduled;
+    int                isOnThreadList;
+    unsigned int       osiPriority;
+    char               name[1];     /* actually larger */
+} epicsThreadOSD;
+
+epicsShareFunc pthread_t epicsShareAPI epicsThreadGetPosixThreadId(epicsThreadId id);
 
 #ifdef __cplusplus
 }

=== added file 'src/libCom/osi/os/posix/osdThreadExtra.c'
--- src/libCom/osi/os/posix/osdThreadExtra.c	1970-01-01 00:00:00 +0000
+++ src/libCom/osi/os/posix/osdThreadExtra.c	2012-07-03 14:42:25 +0000
@@ -0,0 +1,48 @@
+/*************************************************************************\
+* Copyright (c) 2002 The University of Chicago, as Operator of Argonne
+*     National Laboratory.
+* Copyright (c) 2002 The Regents of the University of California, as
+*     Operator of Los Alamos National Laboratory.
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+\*************************************************************************/
+
+/* Author:  Marty Kraimer Date:    18JAN2000 */
+
+/* This is part of the posix implementation of epicsThread */
+
+#define epicsExportSharedSymbols
+#include "epicsStdio.h"
+#include "ellLib.h"
+#include "epicsEvent.h"
+#include "epicsThread.h"
+
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook;
+epicsShareExtern EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook;
+
+EPICS_THREAD_HOOK_ROUTINE epicsThreadDefaultStartHook;
+EPICS_THREAD_HOOK_ROUTINE epicsThreadMainStartHook;
+
+void epicsThreadShowInfo(epicsThreadOSD *pthreadInfo, unsigned int level)
+{
+    if(!pthreadInfo) {
+        fprintf(epicsGetStdout(),"            NAME     EPICS ID   "
+            "PTHREAD ID   OSIPRI  OSSPRI  STATE\n");
+    } else {
+        struct sched_param param;
+        int policy;
+        int priority = 0;
+
+        if(pthreadInfo->tid) {
+            int status;
+            status = pthread_getschedparam(pthreadInfo->tid,&policy,&param);
+            if(!status) priority = param.sched_priority;
+        }
+        fprintf(epicsGetStdout(),"%16.16s %12p %12lu    %3d%8d %8.8s\n",
+             pthreadInfo->name,(void *)
+             pthreadInfo,(unsigned long)pthreadInfo->tid,
+             pthreadInfo->osiPriority,priority,
+             pthreadInfo->isSuspended?"SUSPEND":"OK");
+    }
+}
+

=== modified file 'src/libCom/osi/os/vxWorks/osdThread.c'
--- src/libCom/osi/os/vxWorks/osdThread.c	2010-08-11 15:45:17 +0000
+++ src/libCom/osi/os/vxWorks/osdThread.c	2012-07-03 14:42:25 +0000
@@ -3,6 +3,7 @@
 *     National Laboratory.
 * Copyright (c) 2002 The Regents of the University of California, as
 *     Operator of Los Alamos National Laboratory.
+* Copyright (c) 2012 ITER Organization
 * EPICS BASE is distributed subject to a Software License Agreement found
 * in file LICENSE that is included with this distribution. 
 \*************************************************************************/
@@ -32,6 +33,8 @@
 #include "vxLib.h"
 #include "epicsExit.h"
 
+epicsShareFunc void epicsThreadRunStartHooks(epicsThreadId id);
+
 #if CPU_FAMILY == MC680X0
 #define ARCH_STACK_FACTOR 1
 #elif CPU_FAMILY == SPARC
@@ -42,6 +45,12 @@
 static const unsigned stackSizeTable[epicsThreadStackBig+1] = 
    {4000*ARCH_STACK_FACTOR, 6000*ARCH_STACK_FACTOR, 11000*ARCH_STACK_FACTOR};
 
+/* Table and lock for epicsThreadMap() */
+#define ID_LIST_CHUNK 512
+static int *taskIdList = 0;
+int taskIdListSize = 0;
+static SEM_ID epicsThreadListMutex = 0;
+
 /*The following forces atReboot to be loaded*/
 extern int atRebootExtern;
 static struct pext {
@@ -83,16 +92,25 @@
     }
 }
 
+static void mutexInit(SEM_ID lock)
+{
+    if (lock == 0) {
+        lock = semMCreate(SEM_DELETE_SAFE|SEM_INVERSION_SAFE|SEM_Q_PRIORITY);
+        assert(lock);
+    }
+}
+
 static void epicsThreadInit(void)
 {
     static int lock = 0;
 
     while(!vxTas(&lock)) taskDelay(1);
-    if(epicsThreadOnceMutex==0) {
-        epicsThreadOnceMutex = semMCreate(
-                SEM_DELETE_SAFE|SEM_INVERSION_SAFE|SEM_Q_PRIORITY);
-        assert(epicsThreadOnceMutex);
-    }
+    epicsThreadHooksInit(NULL);
+    mutexInit(epicsThreadOnceMutex);
+    mutexInit(epicsThreadListMutex);
+    taskIdList = calloc(ID_LIST_CHUNK, sizeof(int));
+    assert(taskIdList);
+    taskIdListSize = ID_LIST_CHUNK;
     lock = 0;
 }
 
@@ -157,6 +175,7 @@
     taskVarAdd(tid,(int *)(char *)&papTSD);
     /*Make sure that papTSD is still 0 after that call to taskVarAdd*/
     papTSD = 0;
+    epicsThreadRunStartHooks((epicsThreadId)tid);
     (*func)(parm);
     epicsExitCallAtThreadExits ();
     free(papTSD);
@@ -312,6 +331,29 @@
     name[size-1] = '\0';
 }
 
+epicsShareFunc void epicsThreadMap ( EPICS_THREAD_HOOK_ROUTINE func )
+{
+    int noTasks = 0;
+    int i;
+    int result;
+
+    result = semTake(epicsThreadListMutex, WAIT_FOREVER);
+    assert(result == OK);
+    while (noTasks == 0) {
+        noTasks = taskIdListGet(taskIdList, taskIdListSize);
+        if (noTasks == taskIdListSize) {
+            taskIdList = realloc(taskIdList, (taskIdListSize+ID_LIST_CHUNK)*sizeof(int));
+            assert(taskIdList);
+            taskIdListSize += ID_LIST_CHUNK;
+            noTasks = 0;
+        }
+    }
+    for (i = 0; i < noTasks; i++) {
+        func ((epicsThreadId)taskIdList[i]);
+    }
+    semGive(epicsThreadListMutex);
+}
+
 void epicsThreadShowAll(unsigned int level)
 {
     taskShow(0,2);

=== modified file 'src/libCom/test/Makefile'
--- src/libCom/test/Makefile	2012-06-22 19:06:52 +0000
+++ src/libCom/test/Makefile	2012-07-03 14:42:25 +0000
@@ -89,6 +89,11 @@
 testHarness_SRCS += epicsThreadPrivateTest.cpp
 TESTS += epicsThreadPrivateTest
 
+TESTPROD_HOST += epicsThreadHooksTest
+epicsThreadHooksTest_SRCS += epicsThreadHooksTest.c
+testHarness_SRCS += epicsThreadHooksTest.c
+TESTS += epicsThreadHooksTest
+
 TESTPROD_HOST += epicsExitTest
 epicsExitTest_SRCS += epicsExitTest.c
 testHarness_SRCS += epicsExitTest.c

=== added file 'src/libCom/test/epicsThreadHooksTest.c'
--- src/libCom/test/epicsThreadHooksTest.c	1970-01-01 00:00:00 +0000
+++ src/libCom/test/epicsThreadHooksTest.c	2012-07-03 14:42:25 +0000
@@ -0,0 +1,122 @@
+/*************************************************************************\
+* Copyright (c) 2012 ITER Organization
+*
+* EPICS BASE is distributed subject to a Software License Agreement found
+* in file LICENSE that is included with this distribution.
+\*************************************************************************/
+
+/* epicsThreadHooksTest.c */
+
+#include <stdio.h>
+#include <stdint.h>
+
+#include "epicsThread.h"
+#include "epicsExit.h"
+#include "epicsEvent.h"
+#include "errlog.h"
+#include "epicsUnitTest.h"
+#include "testMain.h"
+
+#define THREAD_NO 6
+#define HOOKS_NO 4
+
+epicsThreadPrivateId threadNo;
+static int order[THREAD_NO][HOOKS_NO];
+static int cnt[THREAD_NO];
+static int mine[THREAD_NO];
+static int called[THREAD_NO];
+static epicsThreadId tid[THREAD_NO];
+epicsEventId shutdown[THREAD_NO];
+
+static int newThreadIndex(epicsThreadId id)
+{
+    int i = 0;
+    while (tid[i] != 0 && i < THREAD_NO) i++;
+    tid[i] = id;
+    return i;
+}
+
+static int findThreadIndex(epicsThreadId id)
+{
+    int i = 0;
+    while (tid[i] !=  id && i < THREAD_NO) i++;
+    return i;
+}
+
+static void atExitHook1 (void *arg)
+{
+    int no = findThreadIndex(epicsThreadGetIdSelf());
+    order[no][3] = cnt[no]++;
+}
+
+static void atExitHook2 (void *arg)
+{
+    int no = findThreadIndex(epicsThreadGetIdSelf());
+    order[no][2] = cnt[no]++;
+}
+
+static void startHook1 (epicsThreadId id)
+{
+    int no = newThreadIndex(id);
+    order[no][0] = cnt[no]++;
+    epicsAtThreadExit(atExitHook1, NULL);
+}
+
+static void startHook2 (epicsThreadId id)
+{
+    int no = findThreadIndex(id);
+    order[no][1] = cnt[no]++;
+    epicsAtThreadExit(atExitHook2, NULL);
+}
+
+static void my_thread (void *arg)
+{
+    int no = findThreadIndex(epicsThreadGetIdSelf());
+    mine[no] = 1;
+    epicsEventMustWait((epicsEventId) arg);
+}
+
+static void mapper (epicsThreadId id)
+{
+    called[findThreadIndex(id)]++;
+}
+
+MAIN(epicsThreadHooksTest)
+{
+    int i;
+    int ok;
+
+    testPlan(THREAD_NO);
+    epicsThreadAddStartHook(startHook1);
+    epicsThreadAddStartHook(startHook2);
+
+    for (i = 0; i < THREAD_NO-1; i++) {
+        shutdown[i] = epicsEventCreate(epicsEventEmpty);
+        char name[10];
+        sprintf(name, "t%d", (int) i);
+        epicsThreadCreate(name, epicsThreadPriorityMedium, epicsThreadGetStackSize(epicsThreadStackMedium), my_thread, shutdown[i]);
+    }
+
+    epicsThreadMap(mapper);
+
+    ok = 1;
+    for (i = 0; i < THREAD_NO-1; i++) {
+        if (mine[i] && called[i] != 1) ok = 0;
+    }
+    testOk(ok, "All tasks covered once by epicsThreadMap");
+
+    for (i = 0; i < THREAD_NO-1; i++) {
+        epicsEventSignal(shutdown[i]);
+    }
+
+    epicsThreadSleep(1.0);
+
+    for (i = 0; i < THREAD_NO; i++) {
+        int j;
+        for (j = 0; j < HOOKS_NO; j++) {
+            if (mine[i] && order[i][j]!=j) ok = 0;
+        }
+        if (mine[i]) testOk(ok, "All hooks for task %d called in correct order", (int)i);
+    }
+    return testDone();
+}