\documentstyle[11pt,a4wide]{article}
\hyphenation{euro-gam}
\def\EG{EuroGam}
\def\VW{VxWorks}
\def\T#1{\hbox{\tt #1}}
\def\N#1{\hbox{\rm #1}}
\def\nt#1{\hbox{\sf #1}}
\def\lb{\hbox{\it [}}
\def\rb{\hbox{\it ]}}
\def\signed#1#2{{\unskip\nobreak\hfil\penalty50
  \hskip2em\hbox{}\nobreak\hfil#1\quad#2
  \parfillskip=0pt \finalhyphendemerits=0 \par}}       % TeXBook, p106

\title{Resource Sharing in EuroGam}
\author{David Brightly}
\date{Edition 1.0\\November 1991}
\begin{document}
\begin{titlepage}
% a front or cover page for EG docs suitable for inclusion inside
% a titlepage environment in a LaTeX document
% for use with Plain TeX uncomment the \nopagenumbers and the \eject

\hoffset=.5in
\hsize=5.25in
\vsize=10.25in
%---gives left and right margins of 1.5in
\font\small=cmssbx10 scaled \magstep2
\font\medium=cmssbx10 scaled \magstep3
\font\large=cmssbx10 scaled \magstep4
%\nopagenumbers
\hrule height 0pt
\parindent=0pt
%\parskip=0pt
\hskip 3.9in
\large
EDOC093\par
\vskip .5in
\large
EUROGAM PROJECT\par
\vskip 1.5in
\hrule height 2pt
\vskip 20pt
\large
NSF DATA ACQUISITION SYSTEM\par
\vskip .5in
\baselineskip 25pt
Resource Sharing in Eurogam\par
\vskip 20pt
\hrule height 2pt
\vskip 1in
\medium
Edition 1.0\par
\vskip 5pt
November 1991\par
\vfill
\medium
NSF Software Systems Group\par
\vskip 5pt
UK Science and Engineering Research Council\par
\vskip 5pt
Daresbury Laboratory\par
\vskip .5in
%\eject
\end{titlepage}
\maketitle
% \noindent [This document is not yet complete but there is ample material
% to convey the main ideas.
% People might like to take an early look at it in its present form and
% prepare comments while I'm away over the next two weeks.
% I haven't written up the XDR bindings for all the procedures yet, and
% the section on implementation recommendations needs expansion.
% Author's remarks are in square brackets.
% \signed{D.B.}{Thursday 16 August 1990]}

\section{Introduction}     %----------------------------
This document explains the machinery supporting the sharing of
data acquisition resources in the \EG\ system.
The main aim of the system is to enable multiple workstations
to share in the set up and monitoring of the data acquisition
hardware in a controlled but flexible way.
A secondary aim is to permit the operation of multiple \EG\ 
hardware systems or partial systems within a common computing
environment consisting of a central file server and discless
client workstations.

\section{Resources}        %-----------------------------
For the purpose of this discussion a {\em resource\/} is
an RPC server process that provides access functions to \EG\
hardware components.
As a general rule, the only way users can affect the data
acquisition hardware is by making RPCs to these server processes.
Most servers provide an {\em abstract interface\/} that hides
the messier details of the underlying hardware.
For example, each \EG\ VXI crate contains a {\em register server}
that works in terms of named abstract objects called {\em registers\/}.
Usually a  register object represents a bit field within some actual
hardware register on some data acquisition card within the crate.
Similarly, the \EG\ histogrammer card is supported by a {\em spectrum
server\/} process that controls the card and manages histogramming
memory by operations on abstract {\em spectrum\/} objects.

Each data acquisition crate can contain one or more servers
(though usually at most one of each type) and a \EG\ system 
consists of one or more such crates.
These crates can be complemented by other computers running RPC servers.
For example, the \EG\ system requires a data
{\em storage server\/} that manages the flow of event-by-event data
onto tape and one or more additional spectrum servers tailored 
towards the management of histograms held in disc files.

A resource thus has the following attributes:
\begin{itemize}
\item a {\em name\/} which identifies it;
\item a {\em host\/} (represented by an IP address) in which it runs;
\item an {\em RPC protocol\/} which it recognises;
\item a UDP {\em port\/} number on which it listens;
\end{itemize}
Given this information a program running in a workstation (usually
referred to as a {\em client\/} program) can contact a resource
and request it to perform operations on the objects it manages.

\section{Capabilities}
All \EG\ RPC is based on the {\em connectionless\/} UDP transport protocol
so an explicit protection mechanism is required in order to safeguard
against accidental interference with one another's equipment by
unrelated users. 
In an essentially {\em single-user\/} environment  this would not
be necessary.
In \EG\ this protection is achieved through {\em capabilities\/}.
These are access tokens which resource servers use to identify and
authenticate users. 
Each RPC must carry a valid capability else it will be rejected by
the server. 
Capabilities are awarded in response to {\bf ClaimResource} RPCs and
at any time may be voluntarily given up by a {\bf FreeResource} RPC.
Resources may be {\em exclusive\/} or {\em shared\/}.
An exclusive resource has no more than a  single capability awarded
at any moment, and hence supports just one ``user'' at a time.
All resource servers running in \EG\ data acquisition crates are
exclusive resources.
A shared resource can have several capabilities outstanding at any
time and thus supports multiple ``users'', each identified by a
distinct capability.
The \EG\ disc-file spectrum server represents a shared resource.
Its users must provide a valid unix login id and password as authentication
in order to obtain a capability.
Thereafter the server can identify the user issuing each RPC by means of the
capability contained therein.

\section{Experiments}
A resource user at a workstation can store his capabilities in a disc file 
and thus make them visible to other workstation users, who can then 
embed them in their own RPC calls. 
Servers will be unable to distinguish  two sources of RPC bearing the
same capability.
Hence two distinct workstation users submitting RPCs with the same
capabilities will appear to a resource server as if from a single source.
This provides the basic mechanism for multiple user workstations to
share access to a common set of resources.


To avoid the proliferation of capabilities in files scattered all over the
filestore (or users scribbling them down on scraps of paper!) the \EG\
resource sharing software  formalises the concept of a {\em capability file\/},
and expects such files to be kept in certain directories only.
A capability file is an array of \T{struct}s of the following form:
\begin{quote}
\begin{verbatim}
struct resourceentry {                         /* records a resource */
   char resname [16];                            /* name of resource */
   char reshost [16];           /* hostname of system its running in */
   int  resport;                           /* UDP port it listens on */
   char resrole [16];                                  /* its "role" */
   char restype [16];         * ie, which RPC protocol it recognises */
   char resapparatus [16];      /* name of "apparatus" it belongs to */
   int  resIPaddr;                                         /* unused */
   CAP  rescapability;          /* 8 byte opaque capability, or null */
   int  resflags;                  /* status flags for this resource */
   };
\end{verbatim}
\end{quote}
recording the attributes  of a set of resource servers and possibly 
a set of corresponding capabilities.

Shared use of a set of data acquisition resources by a group of
workstation users constitutes an {\em experiment\/}, and each experiment
is effectively identified by its capability file.
Any workstation user who has read and write access to such a capability
file can access the resources detailed therein.
Since users collaborating on a \EG\ experiment will need access to further
files  defining other aspects of the experiment the resource sharing
software observes a naming convention for experiment-related files.
Each experiment is given a name (at the users' discretion) and represented
by the directory
\begin{quote}
{\tt /eg/experiments/}{\em experiment}
\end{quote}
All ancillary files required by the \EG\ software system for an experiment
are found beneath this directory.
In particular, an experiment's capability file is named
\begin{quote}
{\tt /eg/experiments/}{\em experiment}{\tt /capabilities}
\end{quote}
Placing experiment related files beneath the fixed directory
{\tt /eg/experiments} also helps any VxWorks application software which
needs access (via NFS) to files created by workstation users---compiled
Event Builder and Sorter programs, for example.

Workstation application programs can assume the existence of an environment
variable, \T{EXPERIMENTDIR}, which contains the full pathname of the
user's current experiment directory, as in
\begin{quote}
{\tt setenv EXPERIMENTDIR /eg/experiments/{\em experiment}}
\end{quote}
In any session involving a \EG\ experiment this variable must be set before
any attempt is made to access \EG\ hardware resources.
To do this we envisage a user interface feature which offers the
possibility of selecting an existing experiment directory or
creating a new one.
Typically the first option would be chosen when a user group wish to
bring a second or subsequent workstation to bear on their experiment.
The second option would be chosen at the start of the setting up 
process for a brand new experiment. 
The user-interface feature solicits a name for the new experiment,
checks that the name is not currently in use, and then creates the experiment
directory and populates it with an empty capability file and 
whatever other ancillary files are defined by \EG\ application software.
At the end of an experiment a further user-interface feature releases all
the experiment's resources and optionally destroys the experiment directory.
All exclusive resources {\em must\/} be released at the end of an experiment 
else the following user group  will not be able to claim them for their
own experiment.
An experiment directory  (possibly even including a capability file
recording active capabilities) can continue to exist after all workstation
sessions  accessing the experiment have ended, and will survive a workstation or
file server boot.
This enables a user-group to regard an ``experiment'' as extending over
several episodes of scheduled beam-time.
However, retaining active capabilities beyond the end of the current
beam-time episode is not recommended since this can inconvenience other
users. 
The system administrator must always be empowered to release such claimed resources
either by seizing the capability file or, more violently, by restarting  the 
held resource server processes.


\section{EGlib}
Clearly, any workstation application program for controlling a \EG\ 
experiment must exploit the information in some capability file.
The preferred way of doing this is through a library of functions, EGlib,
built around the capability file concept. 
EGlib provides functions for  accessing all the object types supported by
\EG\ resource  servers.
For example, the function
\begin{quote}
  {\bf EGreadRegister ({\em name}) $\rightarrow$ {\em value}}
\end{quote}
returns the current {\em value} of the register called {\em name}.
The EGlib  functions expect a capability file to be memory-mapped into
the calling program's virtual address space so that it
appears as an array of {\tt struct resourceentry}s. 
{\bf EGreadRegister}  finds the {\tt resourceentry} representing the
server supporting register {\em name} by means of a naming convention for 
\EG\ objects and resources.
It can then generate an RPC to this server using the host and port addressing
information held in the {\tt struct}.
EGlib has further functions for creating and deleting capability files and
for memory-mapping a selected capability file.

\section{Naming objects}
Each object supported by a \EG\ resource server has a full name in 
the form 
\begin{quote}
{\tt /{\em resource\/}/{\em name}}
\end{quote}
where {\em resource\/} denotes the name of the server resource
supporting the object and {\em name\/} is the resource's own name
for the object.
Within a resource names must be unique, but different resources
can support objects of the same name.
Objects are also distinguished by their {\em type} (e.g, 
{\bf register}, or {\bf spectrum}), so two objects of different
type can have the same full name, though this may give rise to
confusion in the minds of users!
For example, the register object {\tt Ge23.CFD.Thresh} supported
by the VXI register server resource {\tt vxi1} has the full name
{\tt /vxi1/Ge23.CFD.Thresh}.
To locate a server for a given object an EGlib function just scans
the capability file looking for a {\tt resname} field 
matching {\em resource}.
Alternatively, objects can be located by means of {\em aliases} in 
the form
\begin{quote}
{\tt /{\em role\/}/{\em name}}
\end{quote}
where {\em role} must match one or more {\tt resrole} fields.
In the case of matching by role, different resource
servers can support the same {\em role} but should not support
the same object {\em name\/}s.
The {\em role\/} concept can be used to permit access to (say)
register objects without knowing which register server supports them.
For example, a register with the name  {\tt Ge23.CFD.Thresh}
could be found in one of at least two \EG\ VXI crates. 
If both register server resources in these crates support the role
(say) {\tt frontend} then the register can be located by using
the alias {\tt /frontend/Ge23.CFD.Thresh}.

The algorithm used in EGlib for locating an object with full name
{\tt /{\em resource}/{\em name\/}} and type {\em type\/} is thus 
quite simple: 
RPC in sequence to all resources found in the
capability file whose {\tt resname} or {\tt resrole} fields match
the {\em resource} component of the object name 
and whose {\tt restype} matches {\em type\/} until one is found
that recognises the {\em name} component. 
In the case of functions that operate by pattern matching on
{\em name\/},
({\bf EGwriteRegisters}, for example) the scan continues until all
resources whose {\tt resname} or {\tt resrole} fields match {\em name}
have been RPCd.


\section{Apparatuses}
A viable \EG\ system consists of an inter-related set of co-operating
hardware and software components. 
This is formalised in the EGlib concept of an {\em apparatus\/}---a
named set of resource servers comprising part or all of a \EG\
configuration.
Editable ASCII descriptions of possible configurations exist in the
form of so-called {\em apparatus files\/}. 
These are in a simple keyword-value property list format that can
be readily compiled into {\tt resourceentry} structs.
For example, a simple \EG\ configuration consisting of two crates
each running register servers and one running a spectrum server
might appear thus:
\begin{quote}
\begin{verbatim}
((name vxi1)
 (host nnvd)
 (role frontend)
 (type register)
 (ipaddress 123456)
 (port 10220))
((name vxi2)
 (host nnvk)
 (role frontend)
 (type register)
 (ipaddress 123456)
 (port 10220))
((name vxi2)
 (host nnvk)
 (role hist)
 (type spectrum)
 (ipaddress 123456)
 (port 10230))
\end{verbatim}
\end{quote}
(Please note that the {\em ipaddress\/} field has to be present 
even though the value is ignored---this will be corrected!)

EGlib provides functions for constructing capability file 
{\tt resourceentry} structs
by compiling an apparatus file ({\bf EGattachApparatus}), and
for claiming all the resources in a capability file belonging to
a given apparatus ({\bf EGclaimApparatus}).
There are also converse functions for freeing all resources belonging
to an apparatus and for destroying the corresponding structs.
More than one apparatus may be attached or claimed at any one time.

\section{Worked Example}
The following C fragment shows the code needed by a typical
workstation  program that accesses a spectrum. 
The program's first argument is the full name of the spectrum
to be read (assumed 1d), and it follows the conventions on the use of
the EXPERIMENTDIR environment variable and the naming of capability
files.
All error handling is omitted for clarity.
\begin{quote}
\begin{verbatim}
char specname[128];
int counts[4096];
char exptdir[128];
int i;
strcpy (specname, argv[1]);
strcpy (exptdir, getenv ("EXPERIMENTDIR"));
i=EGsetExperimentPath (exptdir);
i=EGconnectExperiment ("capabilities");
i=EGsetSpectrumPath ("");
i=EGread1dSpectrum (specname,0,4096,counts,5)
\end{verbatim}
\end{quote}
The program might be invoked with the command
\begin{quote}
\begin{verbatim}
demo-prog /hist/ge23.4mev.q0
\end{verbatim}
\end{quote}
The next example shows how capability files are constructed, 
assuming the existence of a suitable experiment directory,
with a sub-directory called {\tt apparatus} containing 
apparatus files:
\begin{quote}
\begin{verbatim}
char exptdir[128];
char apparatusdir[128];
char value [128];
int i;
strcpy (exptdir, getenv ("EXPERIMENTDIR"));
i=EGsetExperimentPath (exptdir);
i=EGcreateExperiment ("capabilities");
strcpy (apparatusdir, exptdir);
strcat (apparatusdir, "/");
strcat (apparatusdir, "apparatus");
i=EGsetApparatusPath (apparatusdir);
i=EGattachApparatus ("demo-apparatus");
i=EGclaimApparatus ("demo-apparatus");
i=EGsetRegisterPath ("/frontend");
i=EGreadRegister ("Ge23.CFD.Thresh", value);
\end{verbatim}
\end{quote}


\begin{thebibliography}{99}
\bibitem{EGSS}{\em Eurogam Project: System Specification}, Pucknell et al,
   Edition 1.0, May 1990.
\bibitem{EGlib}{\em EGlib (3L)} manual pages
\bibitem{FRS} {\em \EG\ Frontend Crate Server}, EDOC024, D.Brightly,
   September 1990
\bibitem{CD}{\em Distributed Systems---Concepts and Design}, Coulouris and
   Dollimore, ISBN~0--201--18059--6
\end{thebibliography}


\end{document}