Control Sheet No. 8
In this issue:
- Are companies more expensive than in-house development?
- Ethernet as a real-time network
- CosyBricks FPGA - getting a fresh HW team ready for action
I was a researcher working in accelerator labs for some 20 years and I had a very bad opinion about companies, because they were out there to make profits, while we were working for free, or so we thought. Now, I am an entrepreneur for nearly 10 years and I see the other side. The truth is, there is no good or bad side, and there are just two different cultures. Maybe the best illustration of the misconception most of the people working in the labs have about companies is the following quote from a friend of mine that works in a famous accelerator laboratory: “Labs pay only salaries, while companies pay salaries plus profit, therefore companies must be more expensive.”
Actually, this argument assumes that all people are equally efficient, which turns out to be completely wrong. Our experience is that we are measureable 2-4 times more effective in the work we do, as compared to labs. I can only speak for Cosylab, but I am convinced that the situation is similar for other companies in our field. Now how can this be? Surely the people in labs are smarter than we are? The point is, being smart is only one aspect of effectiveness. Actually, we really try hard to get the smartest people; therefore our recruiting process is much more stringent and meticulous than in labs. But the real challenge in larger projects, which a single person cannot handle alone, is how to make teamwork effective. At Cosylab, we have used our 10 years of experiences in hundreds of projects to optimize our processes and quality assurance to a level that is nearly impossible to match for labs. And why should they? They are researchers, in a way optimized for creativity, while we must be result-oriented. After all, knowing that you won’t get paid if the control system doesn’t work does help to overcome the last mile, even when you don’t feel like working.
So even if we charge twice our salaries, we are still cheaper than labs. And this is exactly why our offer is so cost-effective and attractive to labs, while we can still make a living. Apart from the fact that we don’t have any fancy offices and no perks for the management. So we don’t even behave or act like those “evil” profit-oriented companies.
Some control applications require a control loop that is fast, distributed and hard real-time. Among them are fast orbit feedback in synchrotron light sources , vertical plasma stabilization in tokamaks  and adaptive optics in large optical telescopes .
By fast we mean that the update rate is in the order of several kHz. Distributed implies that the sensors, control algorithms and actuators are not all residing on the same computer, but rather on multiple interconnected computers (e.g., due to the size of the system under control, or capabilities of individual computers). Hard real-time constraint requires that there is an upper bound on the sensor-to-actuator latency – a necessity for stability of a control loop.
One way to meet the requirements is to develop a custom solution. With somewhat more relaxed requirements, common off-the-shelf (but proprietary) technologies such as reflective memory can be used (e.g., ).
Historically, Ethernet was deemed as unsuitable for real-time network applications, particularly due to its inherently non-deterministic Carrier Sense Multiple Access With Collision Detection (CSMA/CD) media access protocol. However, in switched full-duplex networks, the probability of collisions is fully eliminated. Instead, non-deterministic queuing delay can occur on switches, but it is possible to use traffic prioritization via the IEEE 802.1p standard to bound it as well.
Ethernet has several advantages. Most importantly, it is a mainstream technology with a large market share and install base. Therefore, the risk of its obsolescence is, at present, small. Also, there is a large number of competing providers of Ethernet services and products in the marketplace. Furthermore, Ethernet is an evolving technology: from 10Mbs 20 years ago, 10Gbps bandwidths are now commonplace, with 40Gbps and 100Gbps standards just ratified.
So, why not use Ethernet as a real-time network? Experiments  show that the main source of non-determinism is the operating system and its implementation of the network stack. With proper choice and configuration of software, hard real-time constraints can be met! For example, when we have used Xenomai real-time patch for the Linux kernel and RTnet UDP stack implementation, we were able to achieve a jitter of end-to-end (sensor-controller-actuator) latency below 10 microseconds even for very large data packets (see Figure 1), which is sufficient for many real-time control applications.
Figure 1 – latency (left axis) and jitter (right axis) of transmitting a UDP datagram of various sizes from a sensor node, via a control algorithm node, to an actuator node.
 J. Rowland et al.: Status of the Diamond Fast Orbit Feedback System, ICALEPCS’07
 F. Sartori et al: The PCU JET Plasma Vertical Stabilization Control System, 7th Technical Meeting on Control, Data Acquisition and Remote Participation for Fusion Research, Aix-en-Provence, France, 2009
 M. Dimmler et al: E-ELT Primary Mirror Control System, SPIE Astronomical Telescopes and Instrumentation 2008
 K. H. Kim et al: The Integrated Control System for KSTAR, ICALEPCS’05
 A. Barbalace et al: Performance Comparison of VxWorks, Linux, RTAI, and Xenomai in a Hard Real-Time Application, IEEE Transactions on Nuclear Science, 2008
 K. Zagar et al: Evaluation of high-performance network technologies for ITER, Fusion Engineering and Design, 2010
In the past few years, as Cosylab's HW team achieved many great successes, its responsibilities and work opportunities have grown a lot. This also means that the team has to grow – but what do you do with a bunch of HW and FPGA rookies (I can use that word 'cause I was one of them)? In order to bring a fresh HW-team member up to speed with Cosylab's standards regarding coding, implementation and documentation, along with knowledge about basic concepts met regularly in hardware development, the CosyBricks FPGA internal project was born. Its goal was to develop a versatile platform with a low-end FPGA, along with a set of VHDL designs for hardware modules (ADC, DAC, GPIO, etc.) which are commonly used in HW projects. The modular design and a uniform interface between modules reminded us of the famous Danish company's brick toys which we went crazy for when we were kids – hence the project's name. The HW team would gain a set of commonly used building blocks, which could be put together into a working system upon customer demand very quickly.
The effort was organized in the following manner. Each member had to iterate through requirements with the customer (boss), define the architecture of his module and write specifications. He had to stay within budget, adhere to coding standards and deliver a working module accompanied by high quality documentation (document and wiki page). Each module was developed as a final task for the Cosylab FPGA Academy, which every new Cosylab HW team wannabe™ has to go through. The developed platform with its modules was also meant for use in Cosylab's EPICS and LabView Academies.
The CosyBricks base board houses a Spartan 3A FPGA. It has two onboard clocks (10MHz and 16MHz), JTAG connector for programming FPGA and FLASH, 8 connectors (Brick I/F) for connecting modules and an additional 40pin I/O connector. Each module, developed in VHDL, comes with its own I/O board which connects to the base platform via CosyBricks I/F. The developed modules include:
- analog to digital converter – BrickADC,
- digital to analog converter – BrickDAC,
- simple motion controller (single axis control) – BrickSMC,
- general purpose Input/Output – BrickGPIO,
- high-speed Interface – Brick HiSpeedIF.
The base board can be accessed via USB to serial converter (FTDI232), RS232 serial interface and a RJ45 connector for Serializer-Deserializer communication with another board for high speed data transfer (up to 660MHz).
Since the successful finish of the project, new modules have filled our toy box further, e.g. a fiber communication module with clock and data recovery. Through academies EPICS support for BrickADC, BrickDAC and BrickSMC has been developed. The project has achieved its purpose: to quickly bring a team of rookies to production level skills. CosyBricks FPGA was also a project management exercise for the (fresh) project manager and everybody involved have gained valuable experience in project dynamics from start to finish. Its members are now fully integrated into Cosylab's HW development projects, many of which rely on CosyBricks FPGA as the rapid prototyping platform of choice - we like to think of CosyBricks as programmable particles which greatly accelerate our development processes ;)
All documentation, schematics, VHDL code and EPICS support sources are freely available upon request via email. If you are interested in a custom solution based on the CosyBricks FPGA family of modules, feel free to contact us!back to previous content