Merge lp:~epics-documenters/epics-appdev/parallel-cbthreads into lp:~epics-documenters/epics-appdev/3.16

[Ralph Lange]

Documentation for parallel callback threads on multi-core (SMP) systems.
Includes spin locks and spin lock secured ring buffers.

=== modified file 'tex/generalPurposeTasks.tex'
--- tex/generalPurposeTasks.tex	2010-11-15 18:16:16 +0000
+++ tex/generalPurposeTasks.tex	2013-05-17 16:29:29 +0000
@@ -24,14 +24,17 @@
 
 \subsection{Overview}
 
-EPICS provides three general purpose IOC callback tasks. The only difference between the tasks is their scheduling 
-priority; low, medium or high. The low priority task runs at a priority just higher than Channel Access, the medium at a 
-priority about equal to the median of the periodic scan tasks, and the high at a priority higher than the event scan task. The 
+EPICS provides three sets of general purpose IOC callback tasks. The only difference between the task sets is their scheduling 
+priority: low, medium or high. The low priority tasks runs at a priority just higher than Channel Access, the medium priority tasks at a 
+priority about equal to the median of the periodic scan tasks, and the high priority tasks at a priority higher than the event scan task. The 
 callback tasks are available for any software component that needs a task under which to run some job either immediately 
-or after some delay. Jobs can also be cancelled during their delay period. The three callback tasks register themselves with 
+or after some delay. Jobs can also be cancelled during their delay period. The callback tasks register themselves with 
 the task watchdog (described below). They are created with a generous amount of stack space and can thus be used for 
 invoking record processing. For example the I/O event scanner uses the general purpose callback tasks.
 
+The number of general purpose threads per priority level is configurable.
+On SMP systems with multi-core CPUs, the throughput can be improved and the latency (time between job scheduling and processing) can be lowered by running multiple parallel callback tasks, which the OS scheduler may assign to different CPU cores. Parallel callback tasks must be explicitly enabled (see \ref{Parallel Callback Tasks} below), as this feature is disabled by default for compatibility reasons.
+
 The following steps must be taken in order to use the general purpose callback tasks: 
 
 \begin{enumerate}
@@ -250,6 +253,23 @@
 int callbackSetQueueSize(int size)
 \end{verbatim}
 
+\subsection{Parallel Callback Tasks}
+\label{Parallel Callback Tasks}
+
+To enable multiple parallel callback tasks, and set the number of tasks to be started for each priority level, call
+\verb|callbackParallelThreads| before \verb|iocInit| in the startup file. The syntax is:
+
+\index{callbackParallelThreads}
+\begin{verbatim}
+int callbackParallelThreads(int count, const char *prio)
+\end{verbatim}
+
+The count argument is the number of tasks to start, with 0 indicating to use the default (number of CPUs), and negative numbers indicating to use the number of CPUs minus the specified amount.
+
+The prio argument specifies the priority level, with "" (empty string), "*", or NULL indicating to apply the definition to all priority levels.
+
+The default value is stored in the variable \verb|callbackParallelThreadsDefault| (initialized to the number of CPUs), which can be changed using the iocShell's \verb|var| command.
+
 \section{Task Watchdog}
 \label{Task Watchdog}
 \index{Task Watchdog}

=== modified file 'tex/libCom.tex'
--- tex/libCom.tex	2012-01-17 18:04:20 +0000
+++ tex/libCom.tex	2013-05-17 16:29:29 +0000
@@ -541,13 +541,24 @@
 \index{ellFree2}
 \index{ellFree}
 \index{ellVerify}
+
+
 \section{epicsRingBytes}
 
-\verb|epicsRingBytes.h| contains
+\index{epicsRingBytes}
+\index{epicsRingBytes.h}
+\verb|epicsRingBytes.h| describes a C facility for a commonly used type of ring buffer.
+
+\subsection{C interface}
+
+EpicsRingBytes provides methods for creating and using ring buffers (first in first out circular buffers) that store bytes.
+The unlocked variant is designed so that one writer thread and one reader thread can access the ring simultaneously without requiring mutual exclusion.
+The locked variant uses an epicsSpinLock, and works with any numbers of writer and reader threads.
 
 \index{epicsRingBytes.h}
 \begin{verbatim}
 epicsRingBytesId epicsRingBytesCreate(int nbytes);
+epicsRingBytesId epicsRingBytesLockedCreate(int nbytes);
 void epicsRingBytesDelete(epicsRingBytesId id);
 int epicsRingBytesGet(epicsRingBytesId id, char *value,int nbytes);
 int epicsRingBytesPut(epicsRingBytesId id, char *value,int nbytes);
@@ -561,6 +572,7 @@
 
 \index{epicsRingBytesId}
 \index{epicsRingBytesCreate}
+\index{epicsRingBytesLockedCreate}
 \index{epicsRingBytesDelete}
 \index{epicsRingBytesGet}
 \index{epicsRingBytesPut}
@@ -575,6 +587,7 @@
 \textbf{Method} & \textbf{Meaning}\\
 \hline
 epicsRingBytesCreate() & Create a new ring buffer of size nbytes. The returned epicsRingBytesId is passed to the other ring methods.\\
+epicsRingBytesLockedCreate() & Same as epicsRingBytesCreate, but create the spin lock secured variant of the ring buffer.\\
 epicsRingBytesDelete() & Delete the ring buffer and free any associated memory.\\
 epicsRingBytesGet() & Move up to nbytes from the ring buffer to value. The number of bytes actually moved is returned.\\
 epicsRingBytesPut() & Move nbytes from value to the ring buffer if there is enough free space available to hold them. The number of bytes actually moved is returned, which will be zero if insufficient space exists.\\
@@ -600,6 +613,7 @@
 
 \end{itemize}
 
+
 \section{epicsRingPointer}
 
 \index{epicsRingPointer}
@@ -609,13 +623,14 @@
 \subsection{C++ Interface}
 
 EpicsRingPointer provides methods for creating and using ring buffers (first in first out circular buffers) that store pointers.
-It is designed so that a writer thread and reader thread can access the ring simultaneously without requiring mutual exclusion.
+The unlocked variant is designed so that one writer thread and one reader thread can access the ring simultaneously without requiring mutual exclusion.
+The locked variant uses an epicsSpinLock, and works with any numbers of writer and reader threads.
 
 \begin{verbatim}
 template <class T>
 class epicsRingPointer {
 public:
-    epicsRingPointer(int size);
+    epicsRingPointer(int size, bool locked);
     ~epicsRingPointer();
     bool push(T *p);
     T* pop();
@@ -654,21 +669,19 @@
 \begin{longtable}{p{1.27778in}p{5.0in}}
 \textbf{Method} & \textbf{Meaning}\\
 \hline
-epicsRingPointer() & Constructor. The size is the maximum number of elements (pointers) that can be stored in the ring.\\
+epicsRingPointer() & Constructor. The size is the maximum number of elements (pointers) that can be stored in the ring. If locked is true, the spin lock secured variant is created.\\
 \~{}epicsRingPointer() & Destructor.\\
-push() & Push a new entry on the ring. It returns (false,true) is (failure, success). Failure means the ring was full. If a single writer is present it does not have to use a lock while performing the push. If multiple writers are present they must use a common lock while issuing the push. \\
-pop() & Take a element off the ring. It returns 0 (null) if the ring was empty. If a single reader is present it does not have to lock while issuing the pop. If multiple readers are present they must use a common lock while issuing the pop.\\
-flush() & Remove all elements from the ring. If this operation is performed then all access to the ring should be locked.\\
+push() & Push a new entry on the ring. It returns (false,true) is (failure, success). Failure means the ring was full.\\
+pop() & Take a element off the ring. It returns 0 (null) if the ring was empty.\\
+flush() & Remove all elements from the ring. If this operation is performed on a ring buffer of the unsecured variant, all access to the ring should be locked.\\
 getFree() & Return the amount of empty space in the ring, i.e. how many additional elements it can hold.\\
 getUsed() & Return the number of elements stored on the ring\\
 getSize() & Return the size of the ring, i.e. the value of size specified when the ring was created.\\
 isEmpty() & Returns true if the ring is empty, else false.\\
 isFull() & Returns true if the ring is full, else false.
 \end{longtable}
-
 \end{center}
 
-
 \subsection{C interface}
 
 \index{ringPointerId}
@@ -685,6 +698,7 @@
 \begin{verbatim}
 typedef void *epicsRingPointerId;
     epicsRingPointerId epicsRingPointerCreate(int size);
+    epicsRingPointerId epicsRingPointerLockedCreate(int size);
     void epicsRingPointerDelete(epicsRingPointerId id);
     /*epicsRingPointerPop returns 0 if the ring was empty */
     void * epicsRingPointerPop(epicsRingPointerId id) ;
@@ -700,6 +714,9 @@
 
 Each C function corresponds to one of the C++ methods.
 
+epicsRingPointerCreate() creates the unsecured variant, epicsRingPointerLockedCreate() creates the spin lock secured variant of the ring buffer.
+
+
 \section{epicsTimer}
 
 \index{epicsTimer}

=== modified file 'tex/libComOsi.tex'
--- tex/libComOsi.tex	2013-05-17 16:29:29 +0000
+++ tex/libComOsi.tex	2013-05-17 16:29:29 +0000
@@ -841,6 +841,49 @@
 
 A posix version is implemented via pthreads.
 
+
+\section{epicsSpin}
+
+\index{epicsSpin}
+\index{epicsSpin.h}
+\verb|epicsSpin.h| contains definitions for a spin lock semaphore.
+
+\subsection{C Interface}
+
+\begin{verbatim}
+typedef struct epicsSpin *epicsSpinId;
+
+epicsSpinId epicsSpinCreate();
+void epicsSpinDestroy(epicsSpinId);
+
+void epicsSpinLock(epicsSpinId);
+int epicsSpinTryLock(epicsSpinId);
+void epicsSpinUnlock(epicsSpinId);
+\end{verbatim}
+
+\index{epicsSpinId}
+\index{epicsSpinCreate}
+\index{epicsSpinDestroy}
+\index{epicsSpinLock}
+\index{epicsSpinTryLock}
+\index{epicsSpinUnlock}
+\begin{center}
+\begin{longtable}{p{1.38889in}p{5.0in}}
+\textbf{Method} & \textbf{Meaning}\\
+\hline
+epicsSpinCreate & Create a spin lock, allocate required resources, and initialize it to an unlocked state.\\
+epicsSpinDestroy() & Remove the spin lock and any resources it uses. Any further use of the spin lock may result in unknown (most certainly bad) behavior. The results are also undefined if epicsSpinDestroy is used when a thread holds the lock.\\
+epicsSpinLock() & Wait (by spinning, i.e. busy waiting) until the resource is free. After a successful lock, no additional, i.e. recursive, locking attempts may be issued.\\
+epicsSpinTryLock() & Similar to lock except that, if the resource is owned by another thread, the call completes immediately. The return value is 0 if the resource is owned by the caller.\\
+epicsSpinUnlock() & Release the spin lock resource which was locked by epicsSpinLock or epicsSpinTryLock. The results are undefined if the lock is not held by the calling thread, or if epicsSpinUnlock is used on an uninitialized spin lock.\\
+\end{longtable}
+\end{center}
+
+\subsection{Implementation Notes}
+
+The default implementation uses POSIX spin locks on POSIX compliant systems, and epicsMutex for non-POSIX environments.
+
+
 \section{epicsStdlib}
 
 \index{epicsStdlib}
@@ -864,7 +907,6 @@
 They are provided because some architectures have implementations of scanf which do not accept NAN or INFINITY.
 
 
-
 \section{epicsStdio}
 
 \index{epicsStdio}