Control Sheet No. 9
In this issue:
- What to Do About Incomplete Requirements? How to avoid the stall of a project before it has even started.
- Control Systems For New Large Experiments
- Power Converter Controller for MedAustron: high precision steering for cancer killing ion beams
How to avoid the stall of a project before it has even started.
It has happened finally – your new large accelerator project has been approved. After years of waiting, with the initial design papers long obsolete, the government has secured the funds. Now is the time to party! Actually, it's the only good time to party, because it will never be as good as it is now. Once the project has started in earnest, the problems will be arriving too. And since the end of the project is never as clear, if there is an end at all, there will never be a more clearly defined moment in the lifetime of the project. Apart, maybe, from the day the minister visits the facility and presses the red button.
We, the control people, are usually still quite relaxed in the beginning phase of the project. The control system is never on the critical path, so there is no need to start early and, furthermore, we don’t have requirements yet, have we? Ah, requirements, the magic word that makes the blood of some people boil and other freeze to death. The word of both the brave and bold, who use it to demand (“give me your requirements”), and the shy and humble who use it for excuses (“we haven’t the requirements yet”).
So, what should we as controls people do? Should we rejoice with the others and start working on prototypes, technology evaluations, etc., or should we return to our dull daily business and wait for requirements to slowly trickle in? Well, it depends solely on which of the above two options we really want to happen. Requirements can be essentially a good excuse to do any of them. Unfortunately, they can be a good excuse also for others.
It is a fact that the control system group rarely gets proper and clear requirements. And when the other groups finally come knocking on our door, they behave as customers - they want already the final product, as if they were shopping products and not buying development. Knowing that this is a fact of life, what can the control group do to preempt such embarrassing situations?
The obvious solution is to just keep asking for requirements from early on. However, as it happens only too often, the obvious solution is not the best. Even in the Middle Ages, when they could still employ torture, it usually got them the wrong answers. With that option gone, there is little leverage that we have over other groups. And if we keep insisting and threatening, we just make everybody hate us, with no other tangible success.
Based on my experience, it is my opinion, although I am aware that many people may not share this opinion, to start developing the new control system even with a limited or nonexistent set of requirements. There are several good reasons:
- No complex development ever follows the simple “waterfall” scenario with the phases like requirements, design and implementation following linearly one after the other only once the previous is finished. In reality, some requirements are dropped during the way, new come in, and what’s more, development cycles often follow a spiral and pass several times through all the phases. So, why not start the development with what we have at the moment? Maybe make a “straw man” design  or reference implementation and use the newly gained experience in the next round of the spiral?
- Developing a vertical prototype early on in the project creates immense motivational force to the control team. Technologically, it creates a sandbox to develop and test new concepts and technologies and psychologically, it creates awareness in the team and gives the junior members of the team a chance to profile themselves.
- If the list of requirements is long, people often forget that requirements have different weight, different importance, and that the importance often depends not on technical considerations but on political power of the person that has provided them. Most of the politically sponsored requirements don’t have a “price tag” attached to them – they’re just somebody’s fancy. By having done a prototype, even against different requirements, we’re in a much better position to estimate the cost of implementing incoming requirements and to argue against exaggerations; or ask for a larger budget and more people.
- Seeing a physical thing makes people think much more: a customer of ours just recently gave a new board that we have developed to a group in his project that he thought they will use. They borrowed the board for one day and only after careful inspection found out that they want a different connector. Surely they could have specified this earlier? Fortunately, this was a prototype and it is no big deal to add another connector.
- The other use of a vertical prototype is that people can actually use it already for their work, e.g. a power supply control prototype can be used in magnetic measurements, beam instrumentation people can use the controls framework to develop their own software, and so on. Thus we make sure that people get used to the new control system early on, even though it doesn’t exactly match all future requirements.
- And last but not least, there is the fun factor: wouldn’t you rather play with (call it investigate) new technologies and develop prototypes? Just don’t tell it to your boss: giving him all the other previous reason should be convincing enough.
But then you may prefer the safety of your daily chores. In that case, simply reply: “I’d like to start the new development, but we just don’t have the requirements yet”.
During the years control systems have evolved, together with information technology, computer science and electrical engineering. In the starting days, little equipment was available off-the-shelf. A handful of physics labs with difficult requirements were just not commercially interesting. At that time engineers were also scientists. Today, big number of experimental projects and the advance of computing allowed widespread standardization of components. Nevertheless, new experiments are still hiding more technical challenges and questions.
Standardization in light sources
Plenty of light sources were built in the last decades and they have a lot in common with respect to the control system. Control system packages have matured through collaboration and the increasing market has attracted industry. This has reduced control system budget from 10% to 5% of the machine’s budget. The challenge today is to implement a control system with state-of-the-art technology on a smaller budget, shorter time-frame without sacrificing quality.
Pushing the limits of control system components
Machine Protection System
Reference implementations of a flexible machine protection system (MPS) do not exist yet. Based on collected requirements from several projects new features of MPS are required. One such is the reconfiguration of the MPS depending on the working mode of the machine. Integration with the timing and control system is highly desired, allowing quick reconfigurations and post-mortem analysis. The new design allows responses in the range of microseconds making the speed of light the biggest constraint.
Hard Real-Time Feedback System
Implementations of a hard real-time feedback system already exist but are largely based on proprietary technologies and specialized solutions. We measured the deterministic performance of Gigabit Ethernet as a task for ITER. Our results showed that we can already achieve a very good latency of 0.5ms for data rates that are typical for accelerators. With 10 Gigabit or even faster Ethernet, we are confident that Ethernet is a very good choice for development efforts.
New complex machines require new complex timing systems. New features like virtual accelerators, timing super-cycles and event acknowledgements are introduced. The existing (off-the-shelf) timing solution cannot provide all the needed functionality, but they can be used as the basic component, the transmission layer. Still, a lot of customization work is necessary. You can purchase the transport layer, whereas the application layer is machine specific and needs to be implemented for every project individually.
Complex components and integration
We have established that there are definitely trends towards standardization of control system components. But, components are getting more complex and they require more time and effort for integration. Choices need to be made early in the project which is risky if not all aspects are considered.
Basic control system package
Traditionally, the first choice is about the control system package itself. In fact, we believe that people decide for a control system package in a similar way as when they are buying a car: we decide based on emotions and later we rationalize this discussion with architecture description and features. Luckily, most of control system packages are mature and modern technology will enable you to finish your project whatever your choice might be.
Integrating other packages
Many facilities use more than just one control system. Another set of examples is the machine physics world. Usage of many different packages is to be avoided, if possible. For the remaining case, we believe that responsibilities and requirements for the interfaces must be well defined. Documentation and maintenance of the systems must be considered. Technical problems come second to interpersonal relationships. It is usually best to adopt best practices developed and lessons learned by a previous similar experiment.
Distributed development and ‘in-kind’ projects
New large experiments, such as ITER, FAIR or ESS are often started as international projects with many in-kind contributions. This requires very rigid standardization. Every year, the ITER controls group publishes the Plant Control Design Handbook (PCDH), which describes all the standards, and releases the Core System software, the set of standard, ITER approved community tools and software drivers. The standardization continues with the main architecture, hardware platform and IO boards, operating system and software packages. Project life-cycle, naming convention and test plans are also specified.
Focus on development process
Building a complex system from many components is an engineering task but control system development has an even more complicated cycle: write specifications, architecture, design, prototyping, define test procedures, implementation, write documentation, testing, debugging and acceptance at customer.
Projects are increasingly aware of the development processes so more things require focus. The signal list is a golden list that represents the contract between different subsystems and developers. It heavily relies on a good naming convention. Logistics of installation and error handling should be considered, time needs to be planned for testing and debugging. Overall system responsibility should be kept in-house when control system parts are outsourced.
Standardization is the key trend emerging with new and complex projects. Today, integration replaces development as the biggest aspect of a controls project. How will all the components fall into the main architecture, what will be the interfaces and how the requirements will be addressed, are the main questions. Because of increasing organizational risks the focus needs to be shifted to more strict definition and implementation of development processes and rigorous standardization.
In short, control system development is becoming more and more an engineering discipline and less like a science.
A new synchrotron is being built in Wiener Neustadt, Austria by EBG MedAustron GmbH. It will use protons and carbon ions for cancer treatment and for medical and non-medical research. Trial operation is estimated to begin in 2013.
Cosylab has been awarded a framework contract to provide development services and skilled manpower to the MedAustron team for control system design and development. This development project is carried out at CERN (Geneva, Switzerland). Currently, our joint effort is focused on the accelerator main timing system, power converter controllers, front-end embedded software and integration of numerous industry-supplied subsystems with the supervisory control system.
The particle accelerator comprises approximately 300 magnets used for focusing and steering particle beams around the ring. Beams can be generated from various ion-sources and are accelerated to injection level via a pulsed LINAC. Acceleration to energy levels needed for irradiation is performed in a synchrotron. When a beam has reached the required energy level, it is extracted from the synchrotron into the high-energy beam transfer line from where it is delivered to individual clinical irradiation or experimental rooms.
Magnets in the synchrotron ring are powered by purpose-built, voltage-source type power converters. They have to deliver precisely configured amounts of power at defined moments in time and must be operated synchronously within a time frame of 1µs. The distributed system in charge of operating all magnets highly synchronously is the Power Converter Controller!
Each individual Power Converter Controller (PCC) connects to about 30 power converters and tells them how much power to output. A supervisory control system configures the PCCs to ”play” pre-defined waveforms to power converters. All PCCs are closely coupled with the accelerator main timing system which precisely sets the pace of PCC operation and assures system-wide synchronicity.
Awesome! is it already finished?
As of December 2010, a first series has been produced to evaluate its robustness in CERN’s existing accelerator test-beds. Basic software support to carry out functional tests and to evaluate the boards’ maturity has been developed. Together, they demonstrate the entire vertical column of the system – support software, FlexRIO cards with adapter modules and the fiber link to FEDs. The system is able to successfully demonstrate deterministic transmission of high and low priority data over the fiber link. If you disguise yourself as a power converter and connect to the FED via a terminal emulator, you see power setpoints playing in tune with the timing system.
We probably don’t have to mention that this year’s Christmas lights at Cosylab are blinking in perfect synchronicity. :)