Merge lp:~epics-documenters/epics-appdev/parallel-cbthreads-2 into lp:~epics-documenters/epics-appdev/3.16

[Ralph Lange]

Matching documentation for lp:~epics-core/epics-base/parallel-cbthreads-2

[lp:~epics-documenters/epics-appdev/parallel-cbthreads-2]

52. By Ralph Lange on 2013-10-23

Merge ioc-shutdown branch.

53. By Ralph Lange on 2013-10-23

generalPurposeTasks: Move callback request error codes to the "Notes:" section, small fixes.

=== modified file 'tex/generalPurposeTasks.tex'
--- tex/generalPurposeTasks.tex	2010-11-15 18:16:16 +0000
+++ tex/generalPurposeTasks.tex	2013-10-23 09:48:56 +0000
@@ -21,17 +21,21 @@
 other tasks are not delayed. In addition the message can be sent to a system wide error message file.
 
 \section{General Purpose Callback Tasks}
+\label{sec:callbackThreads}
 
 \subsection{Overview}
 
-EPICS provides three general purpose IOC callback tasks. The only difference between the tasks is their scheduling 
-priority; low, medium or high. The low priority task runs at a priority just higher than Channel Access, the medium at a 
-priority about equal to the median of the periodic scan tasks, and the high at a priority higher than the event scan task. The 
+EPICS provides three sets of general purpose IOC callback tasks. The only difference between the task sets is their scheduling 
+priority: low, medium or high. The low priority tasks runs at a priority just higher than Channel Access, the medium priority tasks at a 
+priority about equal to the median of the periodic scan tasks, and the high priority tasks at a priority higher than the event scan task. The 
 callback tasks are available for any software component that needs a task under which to run some job either immediately 
-or after some delay. Jobs can also be cancelled during their delay period. The three callback tasks register themselves with 
+or after some delay. Jobs can also be cancelled during their delay period. The callback tasks register themselves with 
 the task watchdog (described below). They are created with a generous amount of stack space and can thus be used for 
 invoking record processing. For example the I/O event scanner uses the general purpose callback tasks.
 
+The number of general purpose threads per priority level is configurable.
+On SMP systems with multi-core CPUs, the throughput can be improved and the latency (time between job scheduling and processing) can be lowered by running multiple parallel callback tasks, which the OS scheduler may assign to different CPU cores. Parallel callback tasks must be explicitly enabled (see \ref{Parallel Callback Tasks} below), as this feature is disabled by default for compatibility reasons.
+
 The following steps must be taken in order to use the general purpose callback tasks: 
 
 \begin{enumerate}
@@ -102,14 +106,13 @@
 \index{callbackRequest}
 \index{callbackRequestProcessCallback}
 \begin{verbatim}
-callbackRequest(CALLBACK *pcb);
-callbackRequestProcessCallback(CALLBACK *pcb, int prio, void *prec);
+int callbackRequest(CALLBACK *pcb);
+int callbackRequestProcessCallback(CALLBACK *pcb, int prio, void *prec);
 \end{verbatim}
 
-Both can be called from interrupt level code. The callback routine is passed a single argument, which is the same 
-argument that was passed to \verb|callbackRequest|, i.e., the address of the \verb|CALLBACK| structure. The second 
-routine is a shortcut for calling both \verb|callbackSetProcess| and \verb|callbackRequest|. The following delayed 
-versions wait for the given time before queueing the callback routine for the relevent task to execute.
+Both can be called from interrupt level code. The callback routine is passed a single argument, which is the same argument that was passed to \verb|callbackRequest|, i.e., the address of the \verb|CALLBACK| structure. The second routine is a shortcut for calling both \verb|callbackSetProcess| and \verb|callbackRequest|. Both return zero in case of success, or an error code (see below). 
+
+The following delayed versions wait for the given time before queueing the callback routine for the relevant thread set to execute.
 
 \index{callbackRequestDelayed}
 \index{callbackRequestProcessCallbackDelayed}
@@ -127,6 +130,7 @@
 The following calls are provided:
 
 \index{callbackInit}
+\index{callbackShutdown}
 \index{callbackSetCallback}
 \index{callbackSetPriority}
 \index{callbackSetUser}
@@ -141,6 +145,7 @@
 Notes:
 \begin{verbatim}
 void callbackInit(void);
+void callbackShutdown(void);
 
 void callbackSetCallback(void *pcallbackFunction,
     CALLBACK *pcallback);
@@ -149,9 +154,9 @@
 void callbackGetUser(void *user, CALLBACK *pcallback);
 void callbackSetProcess(CALLBACK *pcallback, int Priority, void *prec);
 
-void callbackRequest(CALLBACK *);
-void callbackRequestProcessCallback(CALLBACK *pCallback,
-int Priority, void *prec);
+int callbackRequest(CALLBACK *);
+int callbackRequestProcessCallback(
+    CALLBACK *pCallback, int Priority, void *prec);
 void callbackRequestDelayed(CALLBACK *pCallback, double seconds);
 void callbackRequestProcessCallbackDelayed(
     CALLBACK *pCallback, int Priority, void *prec, double seconds);
@@ -160,12 +165,13 @@
 \end{verbatim}
 
 \begin{itemize}
-\item \verb|callbackInit| is performed automatically at IOC initialization, thus user code never calls this function.
+\item \verb|callbackInit| and \verb|callbackShutdown| are performed automatically at IOC initialization or shutdown,
+thus user code never calls these functions.
 
 \item \verb|callbackSetCallback|, \verb|callbackSetPriority|, \verb|callbackSetUser|, and \verb|callbackGetUser| are 
 actually macros.
 
-\item \verb|Both callbackRequest| and \verb|callbackRequestProcessCallback| may be called from interrupt context.
+\item \verb|Both callbackRequest| and \verb|callbackRequestProcessCallback| may be called from interrupt context. Both return zero for success, or one of the following error codes: \verb|S_db_notInit| for a NULL callback pointer, \verb|S_db_badChoice| for an illegal priority value, or \verb|S_db_bufFull| when the associated queue is full.
 
 \item The delayed versions of the \verb|callbackRequest| routines wait the given time before queueing the callback.
 
@@ -250,6 +256,23 @@
 int callbackSetQueueSize(int size)
 \end{verbatim}
 
+\subsection{Parallel Callback Tasks}
+\label{Parallel Callback Tasks}
+
+To enable multiple parallel callback tasks, and set the number of tasks to be started for each priority level, call
+\verb|callbackParallelThreads| before \verb|iocInit| in the startup file. The syntax is:
+
+\index{callbackParallelThreads}
+\begin{verbatim}
+int callbackParallelThreads(int count, const char *prio)
+\end{verbatim}
+
+The count argument is the number of tasks to start, with 0 indicating to use the default (number of CPUs), and negative numbers indicating to use the number of CPUs minus the specified amount.
+
+The prio argument specifies the priority level, with "" (empty string), "*", or NULL indicating to apply the definition to all priority levels.
+
+The default value is stored in the variable \verb|callbackParallelThreadsDefault| (initialized to the number of CPUs), which can be changed using the iocShell's \verb|var| command.
+
 \section{Task Watchdog}
 \label{Task Watchdog}
 \index{Task Watchdog}

=== modified file 'tex/libCom.tex'
--- tex/libCom.tex	2012-01-17 18:04:20 +0000
+++ tex/libCom.tex	2013-10-23 09:48:56 +0000
@@ -541,13 +541,24 @@
 \index{ellFree2}
 \index{ellFree}
 \index{ellVerify}
+
+
 \section{epicsRingBytes}
 
-\verb|epicsRingBytes.h| contains
+\index{epicsRingBytes}
+\index{epicsRingBytes.h}
+\verb|epicsRingBytes.h| describes a C facility for a commonly used type of ring buffer.
+
+\subsection{C interface}
+
+EpicsRingBytes provides methods for creating and using ring buffers (first in first out circular buffers) that store bytes.
+The unlocked variant is designed so that one writer thread and one reader thread can access the ring simultaneously without requiring mutual exclusion.
+The locked variant uses an epicsSpinLock, and works with any numbers of writer and reader threads.
 
 \index{epicsRingBytes.h}
 \begin{verbatim}
 epicsRingBytesId epicsRingBytesCreate(int nbytes);
+epicsRingBytesId epicsRingBytesLockedCreate(int nbytes);
 void epicsRingBytesDelete(epicsRingBytesId id);
 int epicsRingBytesGet(epicsRingBytesId id, char *value,int nbytes);
 int epicsRingBytesPut(epicsRingBytesId id, char *value,int nbytes);
@@ -561,6 +572,7 @@
 
 \index{epicsRingBytesId}
 \index{epicsRingBytesCreate}
+\index{epicsRingBytesLockedCreate}
 \index{epicsRingBytesDelete}
 \index{epicsRingBytesGet}
 \index{epicsRingBytesPut}
@@ -575,6 +587,7 @@
 \textbf{Method} & \textbf{Meaning}\\
 \hline
 epicsRingBytesCreate() & Create a new ring buffer of size nbytes. The returned epicsRingBytesId is passed to the other ring methods.\\
+epicsRingBytesLockedCreate() & Same as epicsRingBytesCreate, but create the spin lock secured variant of the ring buffer.\\
 epicsRingBytesDelete() & Delete the ring buffer and free any associated memory.\\
 epicsRingBytesGet() & Move up to nbytes from the ring buffer to value. The number of bytes actually moved is returned.\\
 epicsRingBytesPut() & Move nbytes from value to the ring buffer if there is enough free space available to hold them. The number of bytes actually moved is returned, which will be zero if insufficient space exists.\\
@@ -600,6 +613,7 @@
 
 \end{itemize}
 
+
 \section{epicsRingPointer}
 
 \index{epicsRingPointer}
@@ -609,13 +623,14 @@
 \subsection{C++ Interface}
 
 EpicsRingPointer provides methods for creating and using ring buffers (first in first out circular buffers) that store pointers.
-It is designed so that a writer thread and reader thread can access the ring simultaneously without requiring mutual exclusion.
+The unlocked variant is designed so that one writer thread and one reader thread can access the ring simultaneously without requiring mutual exclusion.
+The locked variant uses an epicsSpinLock, and works with any numbers of writer and reader threads.
 
 \begin{verbatim}
 template <class T>
 class epicsRingPointer {
 public:
-    epicsRingPointer(int size);
+    epicsRingPointer(int size, bool locked);
     ~epicsRingPointer();
     bool push(T *p);
     T* pop();
@@ -654,21 +669,19 @@
 \begin{longtable}{p{1.27778in}p{5.0in}}
 \textbf{Method} & \textbf{Meaning}\\
 \hline
-epicsRingPointer() & Constructor. The size is the maximum number of elements (pointers) that can be stored in the ring.\\
+epicsRingPointer() & Constructor. The size is the maximum number of elements (pointers) that can be stored in the ring. If locked is true, the spin lock secured variant is created.\\
 \~{}epicsRingPointer() & Destructor.\\
-push() & Push a new entry on the ring. It returns (false,true) is (failure, success). Failure means the ring was full. If a single writer is present it does not have to use a lock while performing the push. If multiple writers are present they must use a common lock while issuing the push. \\
-pop() & Take a element off the ring. It returns 0 (null) if the ring was empty. If a single reader is present it does not have to lock while issuing the pop. If multiple readers are present they must use a common lock while issuing the pop.\\
-flush() & Remove all elements from the ring. If this operation is performed then all access to the ring should be locked.\\
+push() & Push a new entry on the ring. It returns (false,true) is (failure, success). Failure means the ring was full.\\
+pop() & Take a element off the ring. It returns 0 (null) if the ring was empty.\\
+flush() & Remove all elements from the ring. If this operation is performed on a ring buffer of the unsecured variant, all access to the ring should be locked.\\
 getFree() & Return the amount of empty space in the ring, i.e. how many additional elements it can hold.\\
 getUsed() & Return the number of elements stored on the ring\\
 getSize() & Return the size of the ring, i.e. the value of size specified when the ring was created.\\
 isEmpty() & Returns true if the ring is empty, else false.\\
 isFull() & Returns true if the ring is full, else false.
 \end{longtable}
-
 \end{center}
 
-
 \subsection{C interface}
 
 \index{ringPointerId}
@@ -685,6 +698,7 @@
 \begin{verbatim}
 typedef void *epicsRingPointerId;
     epicsRingPointerId epicsRingPointerCreate(int size);
+    epicsRingPointerId epicsRingPointerLockedCreate(int size);
     void epicsRingPointerDelete(epicsRingPointerId id);
     /*epicsRingPointerPop returns 0 if the ring was empty */
     void * epicsRingPointerPop(epicsRingPointerId id) ;
@@ -700,6 +714,9 @@
 
 Each C function corresponds to one of the C++ methods.
 
+epicsRingPointerCreate() creates the unsecured variant, epicsRingPointerLockedCreate() creates the spin lock secured variant of the ring buffer.
+
+
 \section{epicsTimer}
 
 \index{epicsTimer}
@@ -1567,6 +1584,26 @@
 call to the runTestFunc() routine.
 The last test program or the harness program itself must finish by calling epicsExit() which triggers the test summary mechanism to generate its result outputs (from an epicsAtExit callback routine).
 
+Some tests require the context of an IOC to be run. This conflicts with the idea of running multiple tests within a test harness, as iocInit() is only allowed to be called once, and some parts of the full IOC (e.g., the rsrv CA server) can not be shut down cleanly.
+The function iocBuildNoCA() allows to start an IOC without its Channel Access parts, so that it can be shutdown quite cleany using iocShutdown(). This feature is only intended to be used from test programs. Do not use it on productional IOCs. After building the IOC using iocBuildNoCA() or iocBuild(), it has to be started by calling iocRun(). The suggested call sequence in a test program that needs to run the IOC without Channel Access is:
+
+\begin{verbatim}
+#include "iocInit.h"
+
+MAIN(iocTest)
+{
+    iocBuildNoCA() || iocRun();
+
+[... test code ...]
+
+    iocShutdown();
+    dbFreeBase(pdbbase);
+    registryFree();
+    pdbbase = NULL;
+    return testDone();
+}
+\end{verbatim}
+
 To make it easier to create a single test program that can be built for both the embedded and workstation operating system harnesses, the header file \verb|testMain.h| provides a convenience macro MAIN() that adjusts the name of the test program according to the platform it is running on: main() on workstations and a regular function name on embedded systems.
 
 \index{testMain.h}

=== modified file 'tex/libComOsi.tex'
--- tex/libComOsi.tex	2013-05-17 14:29:34 +0000
+++ tex/libComOsi.tex	2013-10-23 09:48:56 +0000
@@ -841,6 +841,49 @@
 
 A posix version is implemented via pthreads.
 
+
+\section{epicsSpin}
+
+\index{epicsSpin}
+\index{epicsSpin.h}
+\verb|epicsSpin.h| contains definitions for a spin lock semaphore.
+
+\subsection{C Interface}
+
+\begin{verbatim}
+typedef struct epicsSpin *epicsSpinId;
+
+epicsSpinId epicsSpinCreate();
+void epicsSpinDestroy(epicsSpinId);
+
+void epicsSpinLock(epicsSpinId);
+int epicsSpinTryLock(epicsSpinId);
+void epicsSpinUnlock(epicsSpinId);
+\end{verbatim}
+
+\index{epicsSpinId}
+\index{epicsSpinCreate}
+\index{epicsSpinDestroy}
+\index{epicsSpinLock}
+\index{epicsSpinTryLock}
+\index{epicsSpinUnlock}
+\begin{center}
+\begin{longtable}{p{1.38889in}p{5.0in}}
+\textbf{Method} & \textbf{Meaning}\\
+\hline
+epicsSpinCreate & Create a spin lock, allocate required resources, and initialize it to an unlocked state.\\
+epicsSpinDestroy() & Remove the spin lock and any resources it uses. Any further use of the spin lock may result in unknown (most certainly bad) behavior. The results are also undefined if epicsSpinDestroy is used when a thread holds the lock.\\
+epicsSpinLock() & Wait (by spinning, i.e. busy waiting) until the resource is free. After a successful lock, no additional, i.e. recursive, locking attempts may be issued.\\
+epicsSpinTryLock() & Similar to lock except that, if the resource is owned by another thread, the call completes immediately. The return value is 0 if the resource is owned by the caller.\\
+epicsSpinUnlock() & Release the spin lock resource which was locked by epicsSpinLock or epicsSpinTryLock. The results are undefined if the lock is not held by the calling thread, or if epicsSpinUnlock is used on an uninitialized spin lock.\\
+\end{longtable}
+\end{center}
+
+\subsection{Implementation Notes}
+
+The default implementation uses POSIX spin locks on POSIX compliant systems, and epicsMutex for non-POSIX environments.
+
+
 \section{epicsStdlib}
 
 \index{epicsStdlib}
@@ -864,7 +907,6 @@
 They are provided because some architectures have implementations of scanf which do not accept NAN or INFINITY.
 
 
-
 \section{epicsStdio}
 
 \index{epicsStdio}
@@ -1018,6 +1060,7 @@
 double epicsThreadSleepQuantum(void);
 epicsThreadId epicsThreadGetIdSelf(void);
 epicsThreadId epicsThreadGetId(const char *name);
+int epicsThreadGetCPUs(void);
 
 const char * epicsThreadGetNameSelf(void);
 void epicsThreadGetName(epicsThreadId id, char *name, size_t size);
@@ -1071,6 +1114,7 @@
 \index{epicsThreadSleepQuantum}
 \index{epicsThreadGetIdSelf}
 \index{epicsThreadGetId}
+\index{epicsThreadGetCPUs}
 \index{epicsThreadGetNameSelf}
 \index{epicsThreadGetName}
 \index{epicsThreadIsOkToBlock}
@@ -1103,6 +1147,7 @@
 epicsThreadSleepQuantum & This function returns the minimum slumber interval obtainable with epicsThreadSleep() in seconds. On most OS there is a system scheduler interrupt interval which determines the value of this parameter. Knowledge of this parameter is used by the various components of EPICS to improve scheduling of software tasks intime when the reduction of average time scheduling errors is important.If this parameter is unknown or is unpredictable for a particular OS then it is safe to return zero.\\
 epicsThreadGetIdSelf & Get the threadId of the calling thread.\\
 epicsThreadGetId & Get the threadId of the specified thread. A return of 0 means that no thread was found with the specified name.\\
+epicsThreadGetCPUs & Get the number of CPUs (logical cores) available to the IOC. On systems that use Hyper-Threading, this may be up to twice the number of physical cores.\\
 epicsThreadGetNameSelf & Get the name of the calling thread.\\
 epicsThreadGetName & Get the name of the specified thread. The value is copied to a caller specified buffer so that if the thread terminates the caller is not left with a pointer to something that may no longer exist.\\
 epicsThreadIsOkToBlock & Is it OK for a thread to block? This can be called by support code that does not know if it is called in a thread that should not block. For example the errlog system calls this to decide when messages should be displayed on the console.\\

=== modified file 'tex/scanning.tex'
--- tex/scanning.tex	2010-11-15 18:16:16 +0000
+++ tex/scanning.tex	2013-10-23 09:48:56 +0000
@@ -123,6 +123,7 @@
 
 /*definitions for I/O Interrupt Scanning */
 typedef struct io_scan_list *IOSCANPVT;
+typedef void (*io_scan_complete)(void *, IOSCANPVT, int);
 
 long scanInit(void);
 void scanRun(void);
@@ -140,7 +141,8 @@
 int scanpiol(void);   /* print io_event list*/
 
 void scanIoInit(IOSCANPVT *);
-void scanIoRequest(IOSCANPVT);
+unsigned int scanIoRequest(IOSCANPVT);
+void scanIoSetComplete(IOSCANPVT, io_scan_complete, void*);
 \end{verbatim}
 
 \index{SCAN\_1ST\_PERIODIC}The first set of definitions defines the various scan types. The next definition \verb|IOSCANPVT| is used when interfacing with 
@@ -210,7 +212,7 @@
 \begin{enumerate}
 \item Include \verb|<dbScan.h>|
 
-\item For each separate event source the following must be done:
+\item For each separate I/O event source the following must be done:
 
 \begin{enumerate}
 
@@ -242,12 +244,25 @@
 \item Whenever an I/O event is detected call \verb|scanIoRequest|, e.g.
 
 \begin{verbatim}
-scanIoRequest(ioscanpvt)
-\end{verbatim}
-
-This routine can be called from interrupt level. The request is actually directed to one of the standard callback 
-tasks. The actual one is determined by the \verb|PRIO| field of \verb|dbCommon|. 
-
+scanIoRequest(ioscanpvt);
+\end{verbatim}
+
+This routine can be called from interrupt level. The request is queued and will be handled by one of the standard callback threads. There are three sets of callback threads feeding from three queues, one for each priority level (see \ref{sec:callbackThreads}), and the \verb|PRIO| field of the record determines which one is being used. 
+
+\verb|scanIoRequest| returns a bit mask that indicates the priority levels for which record processing requests have been queued.
+
+Device and driver support that wants to implement flow control can set up a completion callback by calling \verb|scanIoSetComplete|, e.g.
+
+\begin{verbatim}
+static void myCallback(void *arg, IOSCANPVT pvt, int prio)
+{
+[...]
+}
+
+scanIoSetComplete(ioscanpvt, myCallback, (void*)arg);
+\end{verbatim}
+
+The completion callback will be called from the callback thread context, once per priority used (bits set in the return value of \verb|scanIoRequest|), after the list of records with that priority level has been processed. Note that for records with asynchronous device support, record processing might not have completed when the callback is issued.
 \end{enumerate}
 
 The following code fragment shows an event record device support module that supports I/O event scanning: