EDOC085

EUROGAM PROJECT

NSF DATA ACQUISITION SYSTEM

VXI/VME integrated data acquisition and control system for the EUROGAM array

Real Time ’91 - Julich June 25 1991

NSF Software Systems Group UK Science and Engineering Research Council Daresbury Laboratory

VXI/VME integrated data acquisition and control system for the EUROGAM array

M.M.Aleonard (CENBG, IN2P3, 33175 Gradignan, France) V.F.E.Pucknell (NSF, Daresbury Laboratory, Warrington, WA44AD, UK)

Abstract

The EUROGAM array data acquisition and control system is distributed over several VXI and VME crates connected by an ethernet local area network. The real time software kernel used is VxWorks and graphical user interfaces are implemented in UNIX based workstations. Remote Procedure Call techniques are used to implement communications between client workstations and the server data acquisition resources.

1 Introduction

Eurogam is an array of suppressed Germanium detectors used to detect gamma-rays with high resolving power and total efficiency. It is being built by a French-UK collaboration involving IN2P3 in France and SERC in UK. The principal laboratories involved are

• CENBG (Bordeaux)

• CRN (Strasbourg)

• IPN and CSNSM (Orsay)

• NSF Daresbury Laboratory

• Liverpool University

Eurogam is a successor to existing devices in Europe such as Tessa3 and Polytessa (UK), Chateau de Cristal (France) and Nordball (Denmark).

For the first phase which will be operational at the Daresbury Laboratory Nuclear Structure Facility for 12 months starting in January 1992 the array will consist of 45 Germanium detectors each of which is surrounded by 10 BGO crystals to form the suppression shield. For the second phase starting in Spring 1993 it is hoped to be able to use the new Vivitron beam at CRN Strasbourg by which time the array should be completed to 70 detectors.

The data acquisition and control system for Eurogam is implemented within a distributed computing environment. Such an environment consists of a number of workstations plus file servers, print servers, tape servers ,etc., all connected via a local area network (typically ethernet). This concept is extended by connecting the data acquisition resources as additional servers onto the local area network. Integrated electronics using the VXI standard and parallel processing of data within VME crates are used for data acquisition. Each VME and VXI crate has a crate controller which is a cpu board having an ethernet interface. This cpu runs the server software and provides access to the crates from any source on the local area network (normally a UNIX workstation) which provides for overall control and monitoring of experiments.

2 Hardware Components

2.1 Front-end Electronics

Since the full Eurogam array consists of 70 GE detectors and 700 BGO suppression shields a departure from conventional electronics using equipment housed in NIM modules is required.

The data acquisition system needs to provide:

• a high degree of integration for the electronics components in order to minimise interconnections (and cables) and thus ensure higher reliability for the array;

• software control of all adjustable parameters;

• several data buses to allow fast response to control requests plus high data transfer rates;

• modularity of hardware to allow a continuous upgrading of the equipment as required.

Using the VXI standard (VME eXtensions for Instrumentation) electronics modules have been designed which will accept the analog signals output from the detectors, digitize the data and output to the data readout bus (DT32 bus).

• GE card—will handle readout from 6 GE detectors

• BGO card—will handle readout from the BGO suppression shields associated with 6 GE detectors (i.e. 60 BGO crystals).

• Master trigger module—handles selection of raw data and initiation of readout.

• Readout Controller—one per VXI crate. Initiates readout via the VME bus from each instrumentation card and outputs to the DT32 bus.

• Resource Manager—provides VXI slot 0 functions

Software can adjust all variable parameters of the hardware (trigger conditions, threshold controls ,etc.,) via the VME backplane.

Each VXI crate can hold all the electronics for 30 channels and hence for the first phase only 2 VXI crates are required for data capture and readout. For the second phase a third VXI crate will be added.

Each VXI crate requires a resource manager (RM) in slot 0. A VXI RM card has been produced which incorporates a standard VME processor card (MVME147 or similar) having an ethernet interface. This card then acts as both the VXI crate master and as a crate controller accessible over the ethernet LAN which can be programmed as desired.

Data from each VXI crate is passed by the Readout Controllers at rates of up to 4 Mbytes/sec onto the DT32 bus (a 32 bit bus using ECL logic).

2.2 Histogrammer

After all VXI crates the DT32 bus passes through a histogrammer unit which is a VME module acting as a spy onto the data. The unit is software programmable to select any specific parameters from any choosen detectors and build histograms via direct memory increment into global dual port memory. The histograms can be examined using any of the workstations to check the quality of the data and monitor detector performance. Initially there will be 32 Mbytes of histogramming memory but this can be increased by adding more memory cards into the VME crate.

2.3 Event Builder

The DT32 bus terminates in a VME crate containing the Event Builder. Input data are received from the DT32 bus via an interface module into one or more CES HSM8170 VME modules controlled by a CES FIC 8232 cpu. A number of processor modules (MVME 165) operating in parallel will process the incoming raw data to reduce the data volume and generate blocks of formatted events. The output data will be passed on via a dedicated fibre optic link using a CES FIC 8232 and CES ODL 8142 capable of data rates up to 6 Mbytes/sec.

For Phase 1 it will be required to process about 1 Mbyte/sec of raw data. However this construction is very flexible and by addition of further hardware modules operating in parallel the data rate through the Event Builder can be increased to handle the full data rate from the DT32 bus.

2.4 Data Router

Data from the optical data link is received by another CES FIC 8232/CES ODL 8142 pair. It may then be routed to either the data sorter or the tape server or both.

The equipment before the data router is located close to the detectors while the Data Router and latter components are located in the control room. At Daresbury these are some distance apart (100 m) and electrically isolated from each other. The optical link provides an excellent means of passing data from one area to the other.

2.5 Data Sorter

This occupies the same VME crate as the Data Router. It uses software running in the crate controller (MVME 147) as a supervisor and a number of processors running in parallel to provide online analysis of the data. Multi-dimensional histograms are built in global memory. Initially 64 Mbytes of memory will be available but this can simply be increased by adding further memory modules. By the use of a suitable number of cpus all data generated by the Event Builder can be processed.

2.6 Tape Server

The Tape Server supports a minimum of four tape streams using Exabyte 8500 drives. Event by event data received from the Data Router is written to one or more of these tapes. For Phase 1 a total of between 0.5 and 1 Mbyte/sec of data will be required to be written to tape. Duplicates of all tapes will be required since post experiment analysis will be divided between the collaborators in the experiment.

2.7 Other Components

In addition to the above modules used to collect, process, sort and store the data other systems provide and control the environment in which the above operate.

2.7.1 Autofill

Another VME crate which provides for the filling and monitoring of the liquid nitrogen in the detectors.

2.7.2 High Voltage control

Uses LeCroy high voltage units interfaced via a VME module to provide the necessary bias to the GE detectors and BGO crystals.

2.7.3 Environment Monitoring Monitors temperature, electrical supply, fans ,etc., in the electronics racks. 2.7.4 UNIX workstations Provide the user interface to the system.

3 Software Components

The user interface to the data acquisition system is via UNIX workstations while all VME cpus run the VxWorks real-time kernel.

3.1 VxWorks

The VxWorks kernel was choosen for the real time components because

• it is designed specifically for real-time control and data acquisition functions within a multi-tasking environment;

• it gives very fast handling of interrupts, task scheduling ,etc.;

• it is UNIX source compatible. All software generation and compilation is performed using UNIX workstations;

• it has full support for BSD networking allowing complete integration into a UNIX workstation environment;

• source level debugging is available from UNIX workstations using DBX.

3.2 Communications

Each VXI and VME crate contains a cpu (MVME 147) having an ethernet interface which is a node on the local area network and which also acts as a gateway for other CPU cards in the same crate. The use of UNIX and VxWorks allows for common network software in all components of the system

• Internet networking (IP)

• UDP/IP (User Datagram Protocol/Internet Protocol)

• RPC (Remote Procedure Calls)

• XDR (eXternal Data Representation)

• NFS (Network File System).

All communications between applications specific to the Data Acquisition System are based on the Client/Server model using Remote Procedure Calls. Using the network software RPC requests can be made between clients and servers across the ethernet, between VME cpus within a crate across the VME backplane, and within a single CPU or workstation. This allows the software components to communication in a way which is independant of their location.

The standard RPC library supplied by Sun Microsystems has been extended to be more flexible and suitable for use in the data acquisition environment. An application

• may issue a number of RPC requests in parallel to different servers in order to improve performance during setup;

• may adjust the RPC timeouts to be suitable for the specific application;

• must at all times be responsive to the user interface.

Fixed UDP/IP ports are used by the servers in order to avoid the delays which the use of dynamic ports and the Portmapper would involve.

Currently ethernet is used as the network medium. This would be inadequate for the main event-by-event data path and hence non standard solutions are used here. In the future FDDI may provide a suitable standard solution.

3.3 The Servers

The Eurogam data acquisition system has a small number of server programs which provide access to all the data acquisition resources.

• register server

• spectrum server

• tape server

• message/error logger

All access is via the RPC service from client workstations. For each server an RPC program protocol is defined which is designed to provide access to the server facilities and to conform with the RPC methodology.

3.3.1 Register Server

The register server is a general purpose communications core which can be customized for specific data acquisition applications. It is used for communication with • VXI crate setup and control application

• Histogrammer setup and control application

• High Voltage control system

• Nitrogen Autofill control system

• Environment monitoring

• Event Builder

• Data Sorter

Initially each server has a number of application specific procedures which are loaded when the system is booted. Registers are then defined dynamically by the clients using the server protocol and specifying the local procedures in the server to be invoked when the register is subsequently accessed.

The register server allows named items of arbitrary data to be passed between the user interface programs and the server applications. The register name and the server command primitive select the local procedure to be called.

The data can be a single 32 bit word to be written to a VXI register or may be up to 8 Kbytes when defining conditions for the data sorter.

Register Server primitives are

• read

• write

• initialise

• read attribute

• write attribute

The register attributes provide additional information as needed by the application procedures to execute the read, write and initialise primitives. For example, a simple case would be a VXI register and the register attributes would include the VME address offset used to access the register on the VXI card.

The server allows many registers to be accessed by a single RPC request by the use of wild-card techniques applied to the names. The available register names can be obtained by clients from the server thus allowing a number of user interfaces in workstations to access the server with all coordination provided by the server.

3.3.2 Spectrum Server

Provides a uniform program interface to both on-line spectra in the histogrammer and data sorter and to off-line spectra held on disc. Supports various high level primitives to allow data to be accessed while minimizing network traffic.

3.3.3 Tape Server

Provides access to the tape storage devices. Supports data striping allowing a high rate data stream to be written to a number of tape drives accessed in parallel and supports data duplication allowing a data stream to be duplicated onto a number of tapes.

3.3.4 Message/error logger

Accepts messages from any component of the system for logging and will pass on to any client which has expressed an interest in the particular message class.

3.4 Event Builder & Data Sorter

In addition to the servers these both run an experiment specific application code which performs the required transformation and analysis of the data. Computer languages have been defined to allow the requirements for these programs to be expressed in a physics-orientated language. Using LEX and YACC compilers have been produced for both languages which converts the user input into C which is then compiled on the UNIX workstation and loaded into the VxWorks processors using NFS.

3.5 Visualisation

Unix workstations are used to setup, control and monitor data acquisition. They are also used for off-line analysis of the data. A window based system is used based on X11 and Open Windows and using the interactive tool Dev Guide to design the screen layouts.

4 Acknowledgements

This work is supported by IN2P3 and SERC funding. The authors wish to acknowledge the contributions from the many hardware and software engineers in France and the UK who are collaborating on the project and discussions with the physicists who will eventually use the array.