Data Acquisition for ExoGam

Status following the meeting held 24th April 1998 at IOP, London

 

V.Pucknell 28th April 1998

 

Software set-up and control of the VXI electronics

 

The software and Graphical User Interface (GUI) developed for Eurogam and Euroball be used for ExoGam. This is a highly modular software package, which can easily be extended to support the new VXI modules. With the present implementation addition of control software and the corresponding GUI for a new VXI module would take at most a week (possibly less) to implement an initial version. This can be done in parallel with the design of the electronics using the module specification and the EVRI memory map.

Experience has shown that a number of factors cause small changes to be necessary to the software during the testing phase of the prototype module. These are caused by misunderstands of the specification (and errors in the specification and EVRI memory map documentation) and corrections needed as the characteristics of the real hardware are determined. Once the first production module is commissioned final adjustments to the default settings and calibration information can be made.

The possibility of making the configuration of the software for new VXI modules more dynamic has been investigated and found to be perfectly feasible particularly during the development phase. This was the original aim for Eurogam I, but was not carried out because of concerns on the performance of doing the whole configuration dynamically. However the software used for CAMAC module configuration is totally dynamic and performance is fully acceptable. While CAMAC modules do tend to be a lot simpler than VXI modules adding a new CAMAC ADC module to the software system can take as little as an hour.

An "engineering" option has been developed which will now be added to all VXI module GUI interfaces. This permits dynamic setup and interactive modification of the software configuration for VXI modules. Direct unrestricted access is also provided to all components of the hardware. The new option can be used in parallel with the standard "user" GUI interface and further development will allow the production system (which requires very easy to use and fast to configure software) to be built from the configuration files created and modified by the "engineering" option.

It has been proposed that all detectors which do not have dedicated VXI electronics should be interfaced via the GANIL xDC6414 VXI module which exists in a number of forms (ADC, qdc and tdc). A detailed specification for this module exists which has been used as the basis for implementation of the necessary control software using the new software "engineering" option.

While the module was designed to be compatible with the Eurogam/Euroball VXI standard this needs to be proven. It has been arranged that the module will be tested in conjunction with a Struck 8032 RM and Struck 8080 VRE at Daresbury on May 5th/8th. This will also be a good test of the effectiveness of the new options in the VXI setup and development software.

Software support for other data sources such as CAMAC and the Bordeaux FVI (Fera to VXI Interface) which may be used by specific ancillary detector systems is a standard option within the MIDAS package and will continue to be available.

 

Software Kernel

For Eurogam/Euroball, and all other instances of the MIDAS software package to control VXI modules, the GUI communicates with software running within a Motorola MVME147 single board VME processor module located in the VXI crate Resource Manager and using VxWorks as the software kernel. While this configuration is perfectly satisfactory for the task the MVME147 module is now over 10 years old and perhaps now is the time to look for an update.

The Motorola PowerPC processor range would seem to be a good choice. For embedding within the Resource Manager the entry level Motorola board the MVME2301 (200 MHz PowerPC 603e and 16 Mbyte DRAM) and the equivalent CES board the RIO2 8062GA (200 MHz PowerPC 603e and 32 Mbyte DRAM) have been considered. Other members of the families are suitable for all other VME based processor applications needed by ExoGam.

After a very careful consideration of the advantages and disadvantages of the options it has been decided that ExoGam will adopt the CES RIO2 8062 family.

The RIO2 8062 family is supported by VxWorks, which has been used so far by Eurogam and Euroball, and so it is an option that we continue to use that. However we would need a new architecture software license from Wind River for the PowerPC processor so from a cost viewpoint we are able to consider alternatives.

LynxOS has been considered since much of the Eurogam VME software has already been ported to it. There is experience of LynxOS at GANIL (but not of VxWorks) and Miniball would also prefer LynxOS rather than VxWorks for the same reason. LynxOS is the preferred software kernel for the CES PowerPC processor board family.

It has therefore been agreed that the VXI control software will be ported to LynxOS and that LynxOS will be used as the real-time kernel for all VME processors used within ExoGam.

Since it has been decided to switch from the MVME147 then changes will be needed to the RM front panel. Now that the processor to be used has been decided (CES8062GA) Daresbury will purchase an ExoGam RM and processor and agree the front panel changes with the supplier. This should be done before any further ExoGam RMs are purchased unless it is accepted that the front panel may need replacing when the changes to accommodate the processor are known.

Daresbury will port the VXI control software from VxWorks to LynxOS using the hardware purchased above. This should not be a major task but there are a couple of technical issues to be resolved. It is estimated that the job will not take more than 4 months. An interim version of the software could be made available before this time which would not contain restrictions of importance during the initial stages of testing.

The software porting required is only of the software which is loaded into the VXI RM. There should not be any requirements to make significant changes to the software which generates the GUI. Small changes may be needed to circumvent feature differences between VxWorks and LynxOS.

Unless there are delays in obtaining funding to purchase the necessary hardware and software (or delays in delivery) then the VXI software will be available by late summer (using CES8062GA and LynxOS). The additional software to support the new VXI modules will be provided as the necessary information is made available and tested in parallel with testing of the prototype VXI modules.

Task

VXI control software using PowerPC processor and LynxOS

Responsible

Daresbury - V.Pucknell and S.Letts

Timescale

First version 3/4 months from delivery of hardware and LynxOS from CES (target late summer 98)

 

Singles histograms

ExoGam will provide two types of singles histograms

  1. The "pseudo" singles as produced for Eurogam and Euroball by hardware which spies on the DT32 bus readout. This histograms only data which is a part of a multi-parameter event. It is also a very valuable diagnostic tool since it is possible to histogram DT32 bus control data that can in certain cases detect very intermittent data errors. The histograms are held in RAM within a VME memory module which the Histogram control unit accesses for increment operations via a dedicated VSB backplane. VME bus is only used for access to the RAM in order to fetch counts for display purposed (and the histogram save function). The software required is exactly the same as for Eurogam/Euroball. It will be required to move the software to LynxOS but some of this work has already been done and some is included in the port of the VXI software.
  2. The "true" singles which is a new development for ExoGam. Patrick Coleman-Smith presented the proposals for the hardware for this during the meeting of 15th Dec 1997. The required data to generate the histograms is passed from the VXI modules to DSPs on PMC daughter boards within a PowerPC processor. The histograms are held in RAM within the processor module and no access to the VME bus is required. Several such processor boards can be used to make a totally scalable system. A development of the Spectrum Access Server software used for the "pseudo" singles described above will be provided. This software in conjunction with the GUI provides a common access method to all histograms created by the Data Acquisition system.

Both types of singles histogramming system can be accommodated within a single VME crate.

Task

Histogrammer control software (pseudo singles)

Responsible

Daresbury - S.Letts

Timescale

First version 3/4 months from delivery of hardware and LynxOS from CES (target late summer 98 - see VXI)

Task

Histogrammer control software (true singles)

Responsible

Daresbury - S.Letts and V.Pucknell

Timescale

First version autumn 98

Task

True singles (DSP software)

Responsible

Daresbury - P.Coleman-Smith

Timescale

First version end 98

 

 

VXI data readout

Readout of data from the various data sources within a VXI crate is via the VXI backplane and a Struck 8080 VXI module (VXI Readout Engine – VRE). This method has been used very successfully for Euroball. The software running in the onboard DSP within the VRE module will be unchanged from Euroball. Set-up, control and monitoring of the VRE module is covered by the VXI software mentioned above. The VRE module GUI in conjunction with features within the onboard firmware supports extensive test and diagnostic facilities, which allow, for example, display of the internal data buffers and generation of user defined test data.

There will be a number of VXI crates each containing a VRE module and linked by a DT32 bus readout chain to a DT32 bus master controller the D2VB (DT32 bus to VME bus). (DT32 bus is rather like a 32-bit version of FERA bus). The D2VB was used for Eurogam II and is used for readout in a number of smaller VXI systems. The D2VB is a VME module, which is in charge of the DT32 bus and delivers data blocks into its onboard 2Mbyte SRAM. The data RAM is divided by the control software into a number of buffers (for example 128 buffers each of size 16 Kbytes). The D2VB also contains a control store, which is a list of buffers within the data ram, which it can use. As the current buffer becomes full the D2VB firmware will switch within 1 microsec and at an event boundary to the next free buffer taken from the list within the control store. If there are no free buffers in the list then the D2VB stops data readout.

The data readout VME crate will contain a control processor, which could be the same as chosen for the VXI RMs (see above). A simple software task runs in the control processor to control the D2VB. As the data buffers are filled by the D2VB the control processor must handle (for example copy to a tape storage server) and return to the D2VB free buffer list. As long as the software can handle data buffers faster than they arrive the D2VB will never stop readout. The 2 Mbytes of internal data ram within the D2VB is adequate for most short-term (a few seconds) fluctuations in data rate.

The D2VB control GUI contains diagnostics to monitor and test the D2VB module and to display the contents of both the data ram and list ram. The data ram displays can be formatted to show the data as events and interpret each data word in terms of detector and ADC.

Software to control the D2VB exists and will be easy to move to LynxOS since it is not strongly dependent on kernel features.

Event Builder

The end of the hardware readout path is the D2VB module, which resides within a VME crate and using a dual ported RAM makes blocks of raw data available via the VME bus. The Event Builder in the ExoGam case is not a builder of events at all since this is done within the VXI electronics and the DT32 bus. However it does have to perform the following tasks:

    1. data readout
    2. raw data distribution
    3. data checking
    4. data formatting
    5. formatted data distribution

data readout

This has been mentioned previously. The existing D2VB control software can be used as a starting point.

Task

Control of D2VB and data readout

Responsible

Liverpool - J.Cresswell

Timescale

First version late summer 98

 

data checking

The raw data blocks received via the D2VB are checked to ensure that they conform to the expected format. The raw data format generated within the VXI electronics modules and the VRE contains a number of fields which are intended specifically to allow checks to be made to ensure that the VXI electronics is functioning correctly. Typical of the checks, which can be made, is:

    1. simple format consistency tests
    2. checks that the contribution to the event from each VXI crate corresponds to the same event (same master trigger pulse).
    3. checks that the data addresses within the event are valid and unique
    4. checks that expected data is not missing and that there is no unexpected data

Statistics and rates of a number of event types including all abnormal events must be accumulated and made available for data quality monitoring.

This software does not exist for ExoGam and requires to be implemented.

Task

Raw data checking

Responsible

Liverpool - J.Cresswell

Timescale

First version autumn 98

 

data formatting

The data-checking phase will pass on all good events to be formatted into blocks to be written to tape. Since ExoGam will be operational at the same time as Euroball IV it is intended that the same format will be used.

Formatting data in this manner (rather than writing raw VXI data to tape) can if an efficient format is used reduce the data volume by a factor of 50%.

This software also does not exist for ExoGam and requires to be implemented.

Task

Data formatting

Responsible

Ganil - F.Saillant & B.Raine

Timescale

First version end Nov 98

 

data distribution

The event builder passes both the formatted data blocks and optionally the raw data blocks (for further online diagnostic purposes when needed) to the data network. The data network should a broadcast network – which is data on the network can be read by all stations on the network. Such networks are FDDI and Ethernet. The data should be transmitted using a network protocol that is compatible with data broadcasting. This means that the UDP protocol should be used and not the TCP protocol. Thus as an example; using UDP protocol to send the data blocks over a FDDI network requires that the data be transmitted once regardless of the number of destinations.

There are two modes of data distribution:

a) controlled - This is the normal mode of data distribution

b) uncontrolled - Used only when there is no requirement for a)

controlled data distribution

This is the normal mode of data distribution for formatted data. The primary requirement is that the formatted data generated by the Event Builder is passed to the tape server to be written to tape. A simple but highly efficient dialogue between the event builder and the tape server controls the flow of data at a rate which is determined by the ability of the tape server to dispatch the data to the tape drives. If this dialogue takes into account the capabilities of the network interface hardware then loose of data and the need to retransmit should never occur. As in any controlled data flow there can be only one master and in our case it is the tape server. In addition to the tape server the broadcast data is available for capture by online data analysis programs and as required by online diagnostic programs in a manner similar to the Uncontrolled data distribution mode.

Uncontrolled data distribution

This is the normal mode of data distribution for raw data and can be used for formatted data when there is no requirement to write the data to tape. The data blocks are passed to the network interface, as they become available without regard to the result and capacity of the network. Since it is not important in this mode that 100% of the data is received by a specific destination there is no problem in doing this. The broadcast data is available for capture by online data analysis programs and as required by online diagnostic programs.

Software for the data distribution task exists from Eurogam using an Interphase FDDI VME controller. As with the data readout task this existing software could be used as a starting point for any new implementation.

Software which can handle the data readout and data distribution tasks already exists and so a simple system reading data from the DT32 bus and broadcasting as raw VXI data for the tape server (and any other spy destinations) can be provided immediately if required. This software could be used as a starting point for a new implementation and combined with software to handle the data check and data format tasks.

Task

Data distribution

Responsible

Ganil - F.Saillant & B.Raine

Timescale

First version end Nov 98

 

 

Task

Event Builder (co-ordination)

Responsible

Ganil - F.Saillant with J.Cresswell/V.Pucknell (advisors)

Timescale

First version end 98

 

 

Online data diagnostics

Online data diagnostics functions use the capability of access to raw VXI data blocks broadcast on the data network. These diagnostics would be in addition to those provided by the data check functions built into the Event builder. Experience with Euroball has shown that understanding of very intermittent errors in the raw data stream can be very difficult using the event builder. The ability to write very specific code to detect very specific errors without effecting the standard Event Builder would be a valuable diagnostic tool. Such diagnostics can be run if necessary during normal experimental runs with minimal effect on the experiment and the data throughput available to it.

The data diagnostics would be run in hardware normally used for Online Data Analysis.

Task

Online data diagnostics

Responsible

Daresbury - P.Coleman-Smith

Timescale

First version late autumn 98

 

 

Online Data Analysis

Online data analysis functions use the capability of slave access to the broadcast network to access the formatted data generated by the Event Builder. Since access is provided to the formatted data the analysis programs can be essentially the same as those used for offline analysis of data tapes. Any number of different analysis programs can in principle be run simultaneously the only limitation being the availability of processors with access to the data network. Library routines/procedures will be provided which give access to the data blocks equivalent to those typically required for access to data blocks from an event-by-event data tape.

A standard package will be provided which uses the MIDAS software for its GUI. This is currently widely used for the analysis of data from Euroball and several other sources. The current implementation is available for Sun Microsystems Solaris 2.5 for Sparc and Debian Linux.

Other packages already available at GANIL will also provided.

Task

Data analysis

Responsible

Liverpool - J.Sampson, I.Hibbert & J.Cresswell

Ganil (?)

Timescale

First version Jan 1999

 

 

Tape Server

The tape server is the master system that controls the broadcast of data from the event builder when running in the "controlled data distribution" mode. When writing experimental event-by-event data to tape this will always be the mode used.

Experience has shown that the tape server should be a dedicated system and that it is highly undesirable for other functions to share the same system. Since the most unreliable part of the whole data acquisition system can be the tape drives (failure of mechanical components) immediate access to repair or replace is essential for the most efficient use of ExoGam.

The tape server and the tape drives are controlled and monitored from a number of windows in the MIDAS GUI. The communications protocols and a guide for implementation are available in the documents edoc304 and edoc305.

For the main operational phase of ExoGam a new implementation confirming to these documents should be made. As an interim measure until the final system is available the VME based tape server from Eurogam can be used taking advantage of existing hardware.

The tape server should be implemented as far as possible in a manner which is not system dependent and all system dependencies should be carefully encapsulated so that versions for various operating systems can be produced. Experience with a Unix based tape server shows that use of the standard Unix device driver /dev/rmt is totally inadequate and research is needed to find a suitable alternative.

For an interim data acquisition system to be available by Jan 1999 the Eurogam tape server can be used. This can be provided using existing hardware. A new implementation (using current generation hardware) could be produced for the start of full experiments, which is expected by mid 1999.

As an alternative (or additionally) it has been proposed that the tape server functions be provided by the existing GANIL data acquisition VMS system. It would be necessary to implement the MIDAS tape server interface within the VMS system. Investigation of the possibility of implementing this interface within VMS, which is dependent on a suitable RPC implementation, for VMS has demonstrated that it is feasible.

Hardware for the tape server will not be required until a choice of implementation has been made. Tape drives will be required to be purchased for the interim system and moved to the final system when ready.

The user community will be asked for preferences as to the tape drives to be provided. The options appear to be DLT drives (which were introduced for Euroball) or Exabyte 8mm drives. A final decision on which to purchase needs to be made by mid autumn if it is intended to have an interim data acquisition system operational by Jan 1999.

Task

Tape server using Ganil VMS system

Responsible

Ganil - B.Piquet + V.Pucknell (advisor)

Timescale

Demonstrate compatibility with MIDAS GUI by Sept 1998

Full implementation by end 1998

 

 

Liquid Nitrogen Auto Fill system

The ExoGam LN2 AutoFill system will functionally be a copy of the Eurogam I/II system. It will however be implemented within an industrial PC in order to reduce the cost. York University will manufacture the hardware and Liverpool University will provide the control system. In addition to local control restricted remote control will be possible via a MIDAS GUI. A full specification document for the software system has been produced (edoc401).

Task

Liquid N2 autofill system

Responsible

Liverpool - I.Hibbert

Timescale

End 1998

 

 

High Voltage control system

The ExoGam High Voltage system will functionally be a copy of the Eurogam/Euroball system. It will however be implemented using an industrial PC in order to reduce the cost. In addition to local control restricted remote control will be possible via a MIDAS GUI.

Task

High Voltage control system

Responsible

Liverpool - J.Sampson

Timescale

End 1998

 

Control and Monitoring stations (GUI)

The user interface to the ExoGam data acquisition system will be provided by the MIDAS GUI. This is used by a number of data acquisition systems including Euroball and also installed at a number of laboratories developing hardware to be used in conjunction with these data acquisition systems. It also provides the user interface to data analysis systems for use with Eurogam/Euroball and other data.

The core of the system; that is the data acquisition control and monitoring functions and the base data analysis functions; is now available for a number of platforms including:

Sun Microsystems Unix for Sparc – Solaris 2.5 or later

Microsoft Windows 95

Microsoft Windows NT 4.0

Linux – Debian, Red Hat and Caldera are known to work; probably others

Sun Microsystems Unix for x86 – Solaris 2.5/2.6

DEC Alpha + Digital Unix (OSF1)

Windows98 and NT 5.0 will be supported as released.

A significant fraction of the MIDAS package is common to all installations and thus the continuing development of the package will be automatically available to ExoGam. There will be certain developments specific to ExoGam mainly for support of the new VXI modules as mentioned previously. As produced these will be supplied to GANIL and to the laboratories developing the new hardware both to enable testing of the software in parallel with testing of the hardware and also to aid checking of the hardware functionality itself.

Task

MIDAS GUI (core system support)

Responsible

Daresbury - S.Letts & V.Pucknell

Timescale

Ongoing development as required

 

 

Summary

The requirement is to have a complete first version of the data acquisition system for ExoGam available from the start of 1999 when 6 detectors will be available to build a small array to use the first beam from SPIRAL.

This document assigns timescales which meet this requirement. It has been necessary to assume that suitable VXI electronics will be available by late 1998. This is the preferred solution since it is closest to the final ExoGam electronics, however, there is currently no commitment from any source that suitable VXI electronics will in fact be available.

Any other electronics can probably be handled within the overall structure but in this case the timescales and implementation plan would have to be reconsidered.