Blog | Accelerators, Big science, Technology

A Generic Control-System Stack on MicroTCA for Data Acquisition

Jure Krašna

Among humankind’s most pivotal projects are Big Physics experiments investigating the foundations of fusion, fundamental particles, origins of the Universe, the structure of materials, biochemical reactions, and much more.
On the other hand, MicroTCA (MTCA) is an open-standard architecture for switched fabric computer systems. Its .4 standard module (MicroTCA.4) increases PCB-space and modularity and is geared towards use in large-scale scientific facilities. With its I/O backplane architecture, MicroTCA is becoming one of the de-facto COTS (commercial off-the-shelf systems) standards in Big Physics, having been already wholly accepted by the high-energy physics community.

Cosylab_MicroTCA

At Cosylab, we specialise in integrating MicroTCA systems, such as high-speed data acquisition devices for our customers.
We have simplified this process of combining MicroTCA with high-speed data acquisition by developing a generic control system stack. The latter allows us to support many MicroTCA hardware board types with minimal firmware and software modifications.

Our firmware supports generic data acquisition up to 32-bit sample width and also generic data generation. We implemented the firmware modules to support MRF timing modules and allow the board to act as an MRF timing receiver. On the software side, we leveraged the control software stack in Nominal Device Support (NDS), which means that we offer support for EPICS and TANGO control systems out of the box.

A data acquisition design, independent from specific frameworks

Cosylab has worked on multiple high-performance data acquisition (DAQ) solutions throughout the years. Some DAQ solutions, such as beam current monitors, beam profile monitors and LLRF, had overlapping core functionality. Because of this, we became intrigued whether we could develop a generic control system stack that would cover this fundamental and super-imposed functionality.

The main objective was to design a system that would improve code reusability and reduce time to support new hardware without sacrificing the ability to extend functionality for custom use-cases. Additionally, the requirement was to cover as many control system frameworks as possible and make the system agnostic to specific frameworks. One of the desired outcomes was to reduce the expertise needed to support existing and custom use cases and create a central, standardised, and mature codebase from which all engineers could learn.

A Hardware design based on MicroTCA

Looking at the generic form factors currently available on the market that support data acquisition use cases, especially for high-end applications, we can safely say that MicroTCA has the most vendors supporting it. It is also growing in popularity on the user side. It is used at various labs worldwide for the already mentioned use cases (BCMs, BPMs, LLRF), and engineers commonly select it as the go-to platform for DAQ applications.

The reasoning for selecting MicroTCA are high performance, customizability options through rear transition module expansions, and advanced features, such as IPMI support and others. By working with multiple MicroTCA vendors — for example, Cosylab and Teledyne co-developing data acquisition drivers for ITER — we have gained a lot of experience and insight into the platform.

When selecting a hardware platform for our own product, a Dose delivery system used in medical particle accelerators for cancer treatment, the decision to go towards MicroTCA was straightforward.

Figure 1: Vadatech AMC523 module

Cosylab: Vadatech AMC523 module

After careful consideration of which hardware was most appropriate for our general data acquisition system (gDAQ for short), we decided to use AMC523 from Vadatech as well as the MRT523 rear transition module and MZ523B mezzanine (see Fig. 1). The boards support 12 analog input channels with variable gain, two analog output channels and can sample at 125 megasamples per second.

Hardware-agnostic software: Nominal Device Support

We developed the application driver using Nominal Device Support (NDS3), a public, open-source library developed at Cosylab to simplify the integration of DAQ hardware. The library is designed to be control system framework agnostic but is targeted primarily towards EPICS and TANGO. We built the first iteration with a focus on EPICS (see Fig. 2). We created all GUI components using Control System Studio Phoebus (see Fig. 3).

Figure 2: System architecture of the EPICS application

To write a hardware-independent and generalised system, we moved most hardware-specific parameters to a JSON configuration file used by all components in the stack to adapt to a specific hardware board. We also made the firmware configurable by pushing the configuration data, which is also stored in the JSON file, through the userspace library.

Figure 3: System GUI created with the Control System Studio Phoebus

Cosylab: System GUI created with the Control System Studio Phoebus

We designed the kernel driver to include as little functionality as possible to reduce the amount of code running in the kernel, decreasing the likelihood of causing a kernel crash due to bugs or errors.

The userspace library has a generalised but still rich and user-friendly C++ API which allows direct integrations, avoiding Nominal Device Support altogether. We wrote the userspace library with extensibility in mind by exposing the firmware register set. The latter makes support for new firmware logic straightforward.

Firmware for a diversity of data acquisition features

Nominal Device Support provides a general abstraction of the device and its signals. While it supports both EPICS and TANGO out of the box, you can extend it to support other frameworks as long as the tree-like description of the device and its channels and state machine logic is versatile enough for the new interface.

The main feature of Nominal Device Support that made this generic stack possible is that NDS generates the EPICS database and TANGO device servers programmatically, enabling us to use the JSON configuration as input during generation.

We designed the firmware to support a wide variety of DAQ features that other vendors include on their firmware modules:

  • Configurable data width up to 32-bit;
  • Configurable up to 32 data acquisition and 32 signal generation channels;
  • Configurable buffer allocation for each channel separately;
  • Separate enable/disable for each channel;
  • Multiple trigger sources including software;
  • Arm and automatic re-arm;
  • Trigger mode: Front trigger, end trigger (prebuffering);
  • Multiple data generation sources: from RAM, internal data stream (NCO), digitally processed ADC data;
  • Data streaming options: on trigger, periodic;

We are also confident that firmware can be ported to other hardware platforms with minimal changes as long as a similar FPGA chip is available as the one used on the AMC523 — the Kintex-7.

Timing Support

We developed an FPGA core with Micro Research Finland (MRF) receiver functionality to support timing functionality. You can include this core on data acquisition boards that provide an SFP or other compatible connector and allow the timing to be connected directly to the board. We also support, out of the box, the more typical use case of having a separate MRF receiver and triggering data acquisition over the backplane or through a front-panel connector.

Conclusion

Our decision to develop a general data acquisition system based on the most successful form-factors on the market – MicroTCA – and to use an excellent publicly available open-source library – the Nominal Device Support – was a winning move.

The latter combination has already proved its versatility and dependability in Cosylab’s Dose delivery system for medical particle accelerators. The gDAQ system is nearing the end of its prototyping and testing phase for external customers and will soon be available for our clients’ production environments.

Cosylab_Jure_Krasna

Jure Krasna is a Technical Lead with ten years of experience in control system design and implementation. Jure specialises in EPICS control systems and data acquisition systems with a focus on MicroTCA. He is also a team lead with a growing team that is passionate about control system development. In his free time, he enjoys baking and mountain biking.

The Republic of Slovenia and the European Union co-financed the investment under the European Regional Development Fund. More information can be found at www.eu-skladi.si. The co-financing of the operation amounts to 488.554,48 EUR.

Others Also Read

Back to Blog