Control Sheet No. 18
- MYRRHA: The New SCK•CEN Multi-purpose Irradiation Facility
- Daniel Kahneman: Beware the ‘inside view’
- Timing is Everything!
- Cosylab joins the DISCS Collaboration
- Handling multiple devices callbacks in a single FESA deploy unit
- Conference Announcement - PCaPAC 2014 Call for Abstracts
By : Robert Modic (Cosylab), Maud Baylac (LPSC), Roberto Salemme (SCK-CEN), Dirk Vandeplassche (SCK-CEN)
MYRRHA (Multi-purpose Hybrid Research Reactor for High-tech Applications), the new multi-purpose irradiation facility in Mol, Belgium has been designed to be an accelerator driven system (ADS) . SCK•CEN, the Belgian Nuclear Research Centre has designed MYRRHA to be a replacement for their BR2 reactor, which has been operational since 1962.
The goals for MYRRHA are varied:
- demonstrate the ADS concept for transmuting long-lived radioactive waste,
- provide a flexible spectrum irradiation facility in support of fission (GEN-IV) and fusion technologies ,
- production of neutron irradiated silicon used in renewable energy technologies,
- radioisotope production for nuclear medicine,
- fundamental scientific research.
The ADS gives the possibility of efficiently transmuting long-lived radioactive waste into much shorter-lived deposits with an overall positive energy balance. A 600 MeV proton accelerator is its first part. The high energy protons are directed at a spallation target where neutrons are produced and fed into a sub-critical reactor core.
An essential aspect of accelerators suitable for ADS applications is their beam availability. For MYRRHA, it must be an order of magnitude better than the current best systems. High availability is achieved by fault tolerance and redundancy of the accelerator. Three factors play a key role here: use of components in a high MTBF (mean-time-between-failures) regime, parallel and serial redundancy of components, ability to repair failing elements. In terms of the accelerator control system (CS), EPICS and Linux are chosen as proven technologies. High availability will be achieved through making critical parts of the CS redundant. Subsystems may be designed for redundancy: if a failure is detected in a subsystem, pre-defined scenarios will kick-in in order to bypass the failing element. A system model or a “virtual accelerator” can be implemented to predict effects of parameter changes, determine configurations of set points for optimal performance or re-configuration in case of a sub-system failure. Implementation of predictive diagnostics should interpret large amounts of data created by the archiving service. Prediction of failure allows for a controlled shutdown as opposed to an abrupt stop.
The MYRRHA ADS will be operational at full power around 2025. In 2013/14 the accelerator injector test-stand is being developed within SCK•CEN in close collaboration with LPSC Grenoble. This small-scale prototype will serve as the proof of concept for technologies that will be used later with the full-size ADS. The prototype consists of an ECR-type ion source and a Low Energy Beam Transport (LEBT) line. This section hosts several beam diagnostic elements and a beam chopper. Cosylab is responsible for the EPICS control system and device integration. We will also contribute our expertise in the specification of e.g. timing, vacuum and cooling controls, interfaces of diagnostics and motion devices. Currently, the requirements document is being finalized and selected devices have already been integrated. Major integration effort should follow after equipment is procured on both sides.
Figure 1: Model of ion source and LEBT line. (Courtesy of LPSC)
Figure 2: Test system layout at LPSC Grenoble.
Figure 3: Full scale MYRRHA with linac, experimental section (ISOL) and nuclear reactor.
About the Authors
Robert Modic is a senior developer at Cosylab. He has vast experience in industrial automation, machine vision, adaptive robotics where his involvement ranged from application development, sales and marketing to project and product management. Currently Robert is the project manager for the control system of the MYRRHA accelerator driven system.
Maud Baylac obtained a PhD from the Université C. Bernard (Lyon, France) in 2000 for her work on Compton-based electron polarimetry at CEA-Saclay. She has worked at the Jefferson Laboratory and the Laboratoire de Physique Subatomique et de Cosmologie where she has been leading the accelerator group since 2006. She is involved in the development of the injector for the future linear accelerator of the MYRRHA project.
Roberto Salemme is Accelerator System Engineer at SCK•CEN. His background is in nuclear engineering with further specialization in particle accelerators. Before joining SCK•CEN in 2013, he collaborated at UA9 experiment at CERN-SPS. Currently Roberto is involved in the R&D and design development of the MYRRHA accelerator and is technical leader of the RFQ@UCL R&D program, a test platform of the future MYRRHA injector.
Dirk Vandeplassche obtained a PhD in Nuclear Physics at the Katholieke Universiteit Leuven. He has worked at CERN as part of the Operations Group of the SPS, LEP and LEAR machines. He continued onto Ion Beam Applications, working as an accelerator physicist for several cyclotrons, especially for the 235 MeV cyclotron for Proton Therapy. In 2008 he joined SCK•CEN in the framework of the MYRRHA ADS project.
By : Frank Amand (COSYLAB)
We stumbled across this excellent business article from 2002 Nobel Memorial Prize in Economic Sciences winner Daniel Kahneman, an excerpt from his book, Thinking, Fast and Slow.
The lessons learned in this article, we consider very relevant to our domain of big science control system projects.
Kahneman recounts his experience with the writing of a high-school textbook as an example of a big project with many contributors and stakeholders. And he tells us how this project was a painful example of how “inside view”-based project estimates were terribly misleading, even if made by-the-book, when all the while they had a life-saving “outside-view” reality check at their disposal… but ignored or, somehow suppressed it.
All of us know how hard good effort estimates are: even when big projects are well underway, it remains difficult to estimate and predict with any accuracy in what time (and budget) the overall project will get finished.
With hindsight, in a post-mortem, we can more easily see that all too many work estimates were too low and that precious time was lost in problems that were not foreseen... but could they be?
The main point Kahneman makes is that we tend to rely on “the inside view” when doing estimates: we rely on information that is already available within (the context of) the project at hand. There seem to be two problems with this:
- Extrapolation: The present data is not representative for the whole project, especially for what still has to come.
- Foreseeing “unknown unknowns”: There are many ways for any plan to fail, and although most of them are too improbable to be anticipated, the likelihood that something will go wrong in a big project is high.
Kahneman argues that an “outside view”, using data from other, similar projects can help alleviate these underestimation risks. He sees two parts:
- Historic data from same-reference-class projects provide a reasonable basis for a baseline prediction.
- This prediction provides an anchor for further (project specific) adjustments.
The Baseline Prediction Technique is defined as the prediction you make about a case if you know nothing except the category to which it belongs. Indeed this is something our experts at Cosylab do employ during early phase project studies. For this they have a large amount of data from reference projects and sub-projects at their disposal
Take for example a large control system integration job, with 100-something device types to be integrated. Categorizing these devices into a limited set of reference classes and assigning them historic work efforts, leads to a very good indication of the total budget. We are applying both of Kahneman’s principles here: use of baseline prediction, and adaptation to the specific project (i.e. we are not simply taking a total time, calculated by averaging total times of complete projects!).
Let us conclude with this recommendation: in the heat of the battle of your project, step out for a moment and ask your best friends in the community with experience in similar large projects a few simple questions. How long it took them (to do this and that), what kind of unforeseen things happened, how your current situation seems to compare to theirs? Their honest answers may not be what you wished to hear, but they can give you the fresh “outside view” that can prove to “save your day”.
About the Author
Frank Amand, Belgian, joined Cosylab in 2011. Previous work experience includes 12 years with Royal Philips in Belgium and the Netherlands in a variety of software engineering related roles. His technical expertise lays in the domain of human-computer interaction, GUI design and usability. He is currently Cosylab’s Head of Marketing.
By : Niklas Claesson (Cosylab), Rok Tavčar (Cosylab), Jože Dedič (Cosylab)
The timing system is an integral part of the complete control system and affects every part of the accelerator facility. Fundamentally, it is responsible for providing the correct current time and the ability to execute synchronous actions, both with a high accuracy.
Figure 1: Simple schematic of event-based and time synchronization timing systems.
At present, it is possible to divide timing system technologies into two camps: event-based and time synchronization. Event-based timing systems are fully deterministic so when a signal is sent, you know exactly when it will be received, processed and acted upon. Time synchronization systems, on the other hand, operate by agreeing on the absolute time using mechanisms like PTP. Events are then executed at the agreed times. As expected, the choice of technology has major implications on the final timing system architecture.
The event-based approach has naturally evolved from years of experience with accelerators and consists of mature products. In essence, event-based timing systems are designed so that every device receives the same message at the same time (assuming equal fiber lengths, or differences in length can be easily compensated) with exceptionally good determinism. Event-based systems tend to be simple and robust (with uni-directional communication). Raw data is transmitted which contributes to the high accuracy (on the order of ps). The optical splitters in the system can be replaced with concentrators to allow bi-directional communication, but this complicates the system and causes extra delays.
The most popular event-based solution is that offered by Micro-Research Finland (MRF) . MRF has been making timing systems since the 1990s and has a range of tried and tested commercial products that are used in various accelerator facilities internationally. Cosylab has been involved in new product development together with MRF for MedAustron and is currently working on an event-based approach that includes customizing an MRF based timing system for ESS
The Shanghai Institute of Applied Physics (SINAP)  has also developed an event-based timing system solution, similar to MRF, and has successfully applied it in several facilities, e.g. SSRF, PAL-II and SuperKEKB. While the SINAP system is not yet a fully supported commercial system, it will be used in future projects like CSNS, China ADS, Shanghai Proton Therapy Facility and SIRIUS.
Time Synchronization Systems
The premise behind time synchronization systems is that every device knows the time with a high accuracy (on the order of ns) and therefore is able to act more independently according to pre-distributed schedules. Time synchronization systems allow bi-directional communication over Ethernet and are able to determine the total time of travel for the packet (send and receive) as well as the local-receiver time. Requests are sent out for actions that must happen at a specified time in the future. Unlike with event-based systems, time synchronization systems can be easily scaled up. Unfortunately flexibility has its price, and these systems offer an inherently slower master to slave response.
A time synchronization system that is gaining popularity is White Rabbit (WR) . White Rabbit was born at CERN and is currently the technology of choice at GSI-FAIR . It is being developed as an open hardware platform in a collaboration between universities, institutes and companies, including Cosylab and it has been implemented in various small scale projects. Recent efforts have been made towards making the precision of WR comparable to that of MRF. However, a scale comparison with the final number of nodes needs to be performed as well as an analysis of precision drift over temperature. Furthermore, White Rabbit has not been implemented in a comparable accelerator set-up to e.g. the MRF products.
Commercial PTP implementations also exist, like the PCI-1588 from National Instruments (implemented at ITER), which are natively supported all across NI platforms. Timing performance is not as good as with White Rabbit, because White Rabbit relies on custom Ethernet switches, which implement; synchronous Ethernet, hardware supported PTP (hardware time-stamping) and package prioritizations.
Cosylab and Timing Systems
Cosylab has been integrating timing systems since 2005 and over this time we have gained extensive experience. We know that the timing system is tightly integrated into an accelerator facility and that in order to choose the best timing system technology for a specific installation it is critical to understand the characteristics of the accelerator, the research requirements of the scientists ultimately using the facility and the performance limitations of the timing system technologies.
We also understand that even after you have chosen e.g. MRF over White Rabbit as the foundation of your timing system, significant integration effort might be required to customize it to a specific installation. For example, even an off-the-shelf MRF timing solution will still require time to integrate it into an installation with relatively simple timing needs e.g. a light source. In general, facilities with more complex timing requirements will require significant customization, even with off-the-shelf products.
Cosylab has also established partnerships with hardware suppliers (e.g. Cosylab has been in an official partnership with MRF since 2011 as a reseller and system integrator for their range of timing system products), we are collaborating with SINAP and we are involved in development of timing systems (e.g. White Rabbit).
Currently, all high-end timing systems that Cosylab delivers are based on FPGAs and optical fiber communication, as they cater to the needs of current accelerator facilities. However, the next generation of light sources will, however, require a leap in technology. If you are interested in collaborating on the next generation of timing systems then contact us!
| || |
- MRF (http://www.mrf.fi/)
- SINAP (http://english.sinap.cas.cn/)
- White Rabbit (http://www.ohwr.org/projects/white-rabbit).
- R. Huhmann, R. C. Baer, D. Hans Beck, J. Fitzek, G. Froehlich, L. Hechler, U. Krause & M. Thieme, “The FAIR Control System - System Architecture and First Implementations,” in ICALEPCS 2013, San Francisco, 2013.
About the Authors
Niklas Claesson, Swedish, joined Cosylab in March 2013 to do his Master thesis on timing systems. Since then, he has been mostly working on the ESS project as a hardware and software engineer when he isn’t on his snowboard. Every once in a while he also enjoys a good single malt whisky.
Rok Tavčar, a graduate of the Faculty of Electrical Engineering at the University of Ljubljana, joined Cosylab in 2009. He has worked on projects (SNS, MedAustron, SINAP, FAIR) involving hardware and software design for timing systems. He enjoys loud music and any sport involving water.
Jože Dedič joined Cosylab in 2006 after his PhD in FPGA HW/SW co-design and started as an electronics developer. He gradually built an FPGA team for real-time solutions (e.g. SNS and MedAustron timing systems). Jože is currently leader of the MedAustron project and Cosylab’s COO. In his free time he either seeks good wind and flat water to do kite-surfing or curved steep roads for motorcycling or mountain-biking.
By : Miha Vitorovič (Cosylab) and Klemen Žagar (Cosylab)
During design and construction of an accelerator, a heterogeneous set of engineering disciplines, methods and tools are used. Data is stored in various databases and files, whose formats are tailored to the discipline. This hinders design as data is replicated in multiple data stores and needs to be frequently reconciled. Furthermore, during the operation and maintenance phases, exploitation of data is difficult since the tools needed to access it are not commonplace or an authoritative version of data cannot be clearly identified.
Distributed Information Services for Control Systems (DISCS)
The DISCS collaboration [1, 2] aims to construct a framework for building high-level applications for commission, operation, and maintenance of an accelerator. The idea is for DISCS to provide out-of-the-box running tools, which can be easily customised and extended if necessary.
Current members of the collaboration are the Facility for Rare Isotope Beams (Michigan, USA) , Brookhaven National Lab (New York, USA) , the European Spallation Source (Lund, Sweden) , the Institute of High Energy Physics (Beijing, China)  and Cosylab.
DISCS comprises of a set of collaborating services and applications, and manages data such as machine configuration, lattice, measurements, alignment, cables, machine state, inventory, operations, calibration, design parameters and security. Each component of the system has a database, an API, and a set of applications, and the services are accessed through RESTful and EPICS V4  (pvData and pvAccess) interfaces.
DISCS is “working together” with EPICS V4 and Control System Studio (CS-Studio)  to provide a better overall solution to controlling experimental physics projects.
DISCS’ database services development methodology is built on the understanding that requirements are going to continue to evolve and that developers are distributed among different laboratories with different technology platforms. With this in mind, the DISCS Architecture has three layers: Data, Service, and Application (Figure 1).
Figure 1: The three layers of the DISCS Architecture: Data, Service, and Application
The Data Layer contains all the data sources: managed, unmanaged, structured, and unstructured.
The Service Layer is made up of services. In the DISCS case, a service can be thought of as a software process that implements controls or physics related logic, and provides high-level data structures to the user through EPICS V4 or REST protocols.
The Application Layer consists of the software tools or components that present the information to the user. The Application Layer accesses the databases through the Service Layer.
At the heart of DISCS are the subsystems referred to as modules. Each subsystem is defined according to user requirements, functionality, and cohesion among data sources. A module consists of a database, one or more services, an API for the service, and zero or more applications to manage the data/service. Modules interact with each other through service APIs.
A Module Team (managed by a Module Team Lead) is responsible for the development of all deliverables related to the module (e.g. service, API, schema, applications, documentation etc.). DISCS is also overseen by a Collaboration Board composed of stakeholders with a vested interest in the success of the project. The Collaboration Board is responsible for governance and architecture of the system and also approves modules before they can get included as a part of DISCS.
More information on each DISCS module can be found on the DISCS Portal .
Cosylab is contributing to the development of various aspects of some of the DISCS modules.
The Configuration Module (Proteus Configuration) manages all physical and logical information about the accelerator and its component systems, including magnets, RFQs, RF cavities, detectors, solenoids, power supplies, valves, pumps etc. The Configuration Module will capture among other information device attributes, design parameters, schematics, mechanical layouts, electrical schedules, EM model information, and geographical location. The Configuration Module will include devices from both accelerator and experimental systems.
Currently work is required in developing a web based UI for data manipulation, extending the database schema and integration of the Configuration Module with other modules. Cosylab will work together with FRIB on these tasks.
The Logbook Module (oLog) is responsible for logging anything and everything that happens on the accelerator. oLog has two interfaces: web and CS-Studio. Cosylab is working on the development of the web interface .
The Naming System (Proteus: Naming) manages the naming conventions for the facility. Currently the module provides functionality only for managing the naming convention, but there is no functionality to ensure that names comply with the convention or a store of all current names. Cosylab is extending this module to be the central data store for all names used in the facility. This will provide the functionality to enforce compliance with the naming convention rules and helps to ensure that all names in the facility are unique.
The Security Module addresses security architecture, mechanisms and implementation to secure devices, applications, services, and databases using authentication, authorization, roles and other mechanisms. The Security Module will be used by all other modules. ESS is driving the development of the Security Module, with assistance from Cosylab and one of the goals is to elegantly apply authentication and authorisation at the IOC level. The main intent of the IOC security is not to prevent malicious behaviour, but to prevent accidental changes that may damage the system.
- V. Vuppala, L.B. Dalesio, D. Dohan, G. Shen, K. Shroff, M. Vitorovič, K. Žagar, K. Rathsman, G. Trahern, D. Liu, C.P. Chu, S. Peng, H. Lv, C. Wang, Z. Zhao: Distributed Information Services for Control Systems, ICALEPCS 2013, San Francisco, USA
- DISCS Portal (http://openepics.sourceforge.net/)
- Facility for Rare Isotope Beams (http://frib.msu.edu/)
- Brookhaven National Lab (http://www.bnl.gov/)
- European Spallation Source (http://europeanspallationsource.se/)
- Institute of High Energy Physics (http://english.ihep.cas.cn/)
- oLog Article Control Sheet 2013/09 (http://cosylab.com/resources/our_newsletter_control_sheet/2013092613404114/1/#Olog)
- EPICS V4 (http://epics-pvdata.sourceforge.net/index.html)
- Control System Studio (http://css.desy.de/content/index_eng.html)
About the Authors
Miha Vitorovič is a Senior Software Developer at Cosylab. He has a background in Computer Science and is currently Project Manager for the Data Management system for ESS, that is responsible for delivery of the Machine configuration, naming, cabling and lattice applications. In his free time, Miha enjoys spending time with his family, hiking and enjoying the mountains.
Klemen Žagar, Slovenian, joined Cosylab in 1999 and started as a software developer. From there he continued as a software and systems architect and is now Chief Technology Officer. His professional interests are distributed control systems, real-time and networking. In his spare time, he enjoys hiking, cycling and running.
By : Tadej Humar (Cosylab)
As a part of the FAIR project  Cosylab had to integrate a spectrometer into FESA (Front-end Software Architecture)  which is a control system framework developed and used in CERN and GSI [3, 4].
The FESA class implements all functionalities needed to control a device and to present its information to the user. It is split into two main parts. The real time part is made of RTActions that execute when triggered by an event (timing, timer, custom made ...) and custom events sources that allow the programmer to create their own way of firing these events. Each custom event source has a Wait function that needs to be blocked until an event needs to be fired. The main tasks of RTActions are to notify (trigger) the properties that send data to the user and in the case of a fast device to read its data and store it in private variables (FESA data fields) to be accessed later by properties. The second part of the class is made of so called server part that has properties and commands together representing the device interface. Properties can be executed by user on request or when RTAction notifies it and client is subscribed. Filtering and parsing of the data can be done in this part and usually configuring/controlling the device is done as well. The class runs inside a deploy unit that creates as many device instances as specified at startup.
Cosylab’s task was to handle several (up to 10) spectrometers with a single deploy unit and not use any polling mechanisms.
The spectrometer’s driver sends data through three different kinds of callbacks (new status, new configuration and new measurement). The first thing that came to mind was to use custom event sources but there is a catch. There is only one instance of a custom event source (status event, config event and measurement event) per class in FESA so that means the code in each event source will need to be able to handle as many callbacks as there are device instances defined at runtime and to keep track of which device sent it.
Before implementing the class, we consulted with Alexander Schwinn from GSI and his suggestion was to spawn a thread for each device instance inside the custom event source. Then, the thread posts any data it receives from a callback to the queue (private member of custom event source class) and sends a signal to the Wait function. The Wait function of the custom event source is blocked by a conditional variable so there is no need to constantly check if the queue is empty, thus satisfying the “no polling” requirement. Data is sent as payload in the event. This idea formed the basis of the solution.
It didn’t make sense to spend a lot of CPU cycles within the custom event source nor within RTAction as these reside inside the real time part of the class and should be executed quickly. So, a wrapper for the driver was written to first handle all new data (e.g. store new value, memory reallocation if needed, and archiving to files) and then signal that new data arrived. Another benefit of this architecture is that the FESA class will support different types of spectrometers without any changes to the custom event sources and with minimal additions to RTActions (configuration and status differs) since all logic dealing with the data is located inside the wrapper.
Only the implementation of handling new measurement data will be described. Status and configuration are handled similarly. After the wrapper/drivers and connections have been initialized, we can access the wrapper for each device instance from anywhere within the FESA class only by sending the device name as parameter.
The wrapper has a function that will block until new data is received and processed. The data is stored in a collection which can be accessed later through the wrapper.
Inside the custom event source constructor there is a for-loop that creates a new thread for each device instance. There are two parameters sent to every thread: the device name that the thread is assigned to and the pointer to the custom event source so the thread can add the device name to its queue once new data is available.
The thread uses the device name to get the correct spectrometer wrapper handle and calls the wrapper’s waiting function to block until new data is available. When the wrapper waiting function unblocks, the thread will add its name to the custom event source’s queue and signal the Wait function inside it.
When the Wait function inside the custom event source is unblocked, it checks the queue, extracts (pops) the name and sends it as the payload in a new data event. The Wait function will continue to extract names and fire events until the queue is empty. Then it will block waiting for a new signal. This is necessary since new callbacks could happen (names added to queue) while the Wait function was creating and firing the event.
Inside the FESA class, an RTAction handling new data notifications is executed on demand when a new data event is fired. It extracts the device name from the payload, checks if this device instance exists and gets its pointer. Depending on the scan status (trend, real-time monitoring, etc.) appropriate properties are notified.
The class still needs to go through another stage of testing with program profiling to identify possible optimizations.
Figure 1: Spectrometer (Residual gas analyzer) communicating data through driver 
Figure 2: Flowchart of custom event source
About the Author
Tadej Humar, Slovenian, joined Cosylab in April 2012 as a student and was employed a year later. He has mostly worked on device integration in FESA and TANGO. Until recently he also worked part time as a sound engineer. Although he is not a beer fanatic he never says no to a Belgian beer Friday night.
Is Cosylab taking over National Instruments? Probably not yet, but we can start with NI people getting used to CSL T-shirts in any case. (Sebastian Koziatek, National Instruments’ Applications and Systems Engineering Manager for Eastern Europe.)
10th International Workshop on Personal Computers and Particle Accelerator Controls (PCaPAC 2014)
14th - 17th October 2014, Karlsruhe, Germany
Call for Abstracts
The ANKA Synchrotron Radiation Facility at the Karlsruhe Institute of Technology (KIT) invites you to the PCaPAC 2014, the 10th International Workshop on Personal Computers and Particle Accelerator Controls.
Original papers are invited.
More information at : http://www.anka.kit.edu/2810.php
Abstract Deadline: 1 April 2014
back to previous content