Beneš Network Based Efficient Data Concentrator for Triggerless Data Acquisition Systems

1. Introduction

Most detectors in high-energy physics (HEP) experiments deliver massive data streams in multiple channels. Reception of this data and its delivery to the analyzing computers is the task of the readout chains. In the case of experiments using a trigger, the data must be processed locally to elaborate the level one (L1) trigger decision. Preparing it for concentration may be a side effect of this process. The data is zero suppressed – invalid or empty data words are removed from the stream. Finally, the data is buffered in memory, which may offer the data width conversion. It may be written with single data words and read with multiple words in parallel, as required by the DAQ interface.

In the triggerless readout, the situation is different. The readout system does not need to perform complex local processing of data. Extraction of interesting events is done in further stages of the DAQ (“event builder” and “event filter” [1] or “event selector” [2]). The responsibility of the readout system is different in this configuration. It should almost transparently deliver the detector data to DAQ. For triggerless DAQ, a very popular architecture is the one where the DAQ computers are located near the detectors, and data concentrating boards are the PCIe cards. The PCI Express blocks in FPGA require specific data bus width in the AXI interface. Table 1 shows the available AXI data widths depending on the speed of the link, the width of the PCIe lane, and AXI clock frequency. For PCIe 8xGen3, it is necessary to work with 256-bits wide data.

The detector data generated by the particle detection is usually short. For example, for the STS-XYTER (also known as SMX) [3]) the data is 24-bit long. In the concentrated stream, the data must be accompanied by metadata describing its origin, resulting in a final size of 32 bits. Hence, the data concentrator must efficiently pack 32-bit detector data into 256-bit PCIe data.

The problem may be generalized as described in the next section.

1.1. Formulation of the Problem

The system receives the data words from N inputs at frequency $f_{in}$, and puts them into the records able to store M words, which are read at $f_{out}$ frequency. The system has sufficient bandwidth. The following condition is met:

$$N·f_{in}\le M·f_{out}$$

The intensity of the data stream delivered by the inputs may be different and may vary in time. It means that some links may deliver invalid (or empty) data in a particular period of the input clock. Those invalid data should be skipped not creating “holes” in the output stream. For event reconstruction in the triggerless DAQ, the data must be assigned to a particular time period1. Therefore, an essential requirement is that the concentrator disturbs the time-ordering of the input data as little as possible.

In the next chapter, the existing solutions to the concentration problem are presented, and their disadvantages are discussed.

2. Existing Solutions for Concentrators

Unfortunately, finding published information about the data concentration methods used in existing FPGA-based data concentrators is difficult. The presented review is based on seldom publications and presentations, publicly available source code analysis, or the authors’ experience.

Most existing solutions may be grouped into two categories described in the following sections.

2.1. High Speed Polling

The trivial solution is based on browsing all inputs at the frequency $f_{scan}=N·f_{in}$ (see Figure 1).

If valid data is found, it is copied to the asymmetric FIFO queue with an input width of 1 word and an output width of M words. That warrants that no valid data is skipped, and the output words are filled with valid data only. The only problem with that solution is that the necessary $f_{scan}$ may be too high to be acceptable for FIFO in FPGA. Therefore this method may be used only in case of low input clock frequency or concentrating data from a small number of inputs.

This solution is used for concentrating the data from STS-XYTER2 [3] front-end ASICs transmitted through GBT Links [4] in the readout chain of the STS detector in the CBM experiment [5]. In that readout, each GBT Link transports data from 14 E-Links working with a 320 Mb/s rate. The 8b/10b encoded hit data occupy 30 bits in the E-Link. Therefore, the hit data rate in the individual E-Link is not higher than $\frac{320\mathrm{Mb}/\mathrm{s}}{30\mathrm{bits}}\approx 10.67\mathrm{MHz}$. That is a low frequency. Because $14·10.67\mathrm{MHz}<160\mathrm{MHz}$, the data can be safely concentrated by consecutive scanning all 14 E-Link outputs at 160 MHz.

Another example is the firmware for the CRU board [6] used by the ALICE experiment at LHC at CERN. It uses a round-robin scanning of the output of FEE links (see [6], Figure 8). Then only the valid data are packed into the 256-bit wide FIFO, delivering the data to PCIe. The authors do not describe at which frequency the inputs are scanned.

2.2. Width Conversion in Input Channels

A high-speed scanning may be avoided if width conversion is performed in the input channels. In this solution, the small FIFOs with one-word wide input and M-words wide output are placed in each input channel, as shown in Figure 2. This solution does not require using a very high clock frequency. However, it has other disadvantages. If the link occupancy significantly differs between inputs, the concentration may significantly change the time-ordering of data. The data from low-rate links may get significantly delayed until M data words are collected. That problem may be solved by introducing the timeout, after which the non-empty FIFO outputs its content even if it contains less than M words. However, that modification results in inserting “holes” into the concentrated stream. Another disadvantage is the necessity to use a separate width-converting FIFO in each input channel. Those FIFOs may have limited depth, enabling implementation based on distributed RAM, but they may still increase resource consumption. Finally, that solution cannot use a simple counter-driven multiplexer periodically browsing the data. That solution requires a more complex data-driven multiplexer, which automatically selects the first input providing the complete data record after the previously serviced one (i.e., it implements the round-robin policy).

This approach seems to be used in the firmware for the FELIX board in the ATLAS experiment readout at CERN [7]. Unfortunately, the operation of the data concentrator in the FELIX firmware is not described in detail in any paper. However, the sources of that firmware are publicly available, enabling analysis of the concentrator code. The concentration is done in the CRToHostdm module [8] containing the asymmetric FIFO responsible for concatenating a few words and width conversion. Outputs of FIFOs from multiple channels are scanned in the CRToHost module [9].

2.3. Need for Another Concentration Method

None of the above methods matches all the requirements described in Section 1.1. Therefore, a new method is proposed in the next section.

3. Proposed Solution—Concentration with the Direct Routing of Data

It is possible to avoid the disadvantages of the previously described solutions by directly routing data from inputs to the proper position of the output record. Such a solution is shown in Figure 3. The key functionality needed in this method is a capability to write the data from each individual input to a selected position in the output record. A dedicated controller calculates the desired location of data from each input. The controller must keep track of the occupancy of the output data. Additionally, it receives the “valid” flags from the input words. The controller starts with an output record occupancy equal to zero. If, for example, it receives valid words in three inputs, it routes them to positions 0, 1, and 2 in the output record and changes the occupancy to 3. The next valid word will be routed to position 4, and so on. When the output record is filled, the output strobe is generated, the collected words are transferred to the output FIFO, and the occupancy is set to 0.

There is, however, a problem if the concentrator receives more valid data than needed to fill the output record. Those superfluous words must be stored somewhere. For that purpose, an “auxiliary record” register is introduced. The controller generates a write strobe for both registers. The output strobe causes the transfer of the output record to FIFO and, at the same time, of the auxiliary record to the output record. The modified concentrator is shown in Figure 4.

The presented concentrator should work correctly, but its implementation in FPGA is inefficient. Implementing M multiplexers routing the data words consumes many resources and generates long critical paths in the FPGA. Therefore yet another modification is needed. The multiplexers must be replaced with more efficient blocks for routing the data.

3.1. Concentrator Based on Beneš Network

A similar problem had to be solved in telecommunication networks for routing connections. Networks enabling arbitrary data permutation between their inputs and outputs are known as Beneš networks and have been described in [10]. An example of such network routing 8 inputs to 8 outputs is shown in Figure 5. The Beneš network uses simple switches with two inputs and two outputs, transmitting the data transparently or swapping them. They may be efficiently implemented in FPGA. The lengths of all data paths are the same, so this network can be efficiently pipelined, which results in a short critical path. The general scheme of the concentrator based on the Beneš network is shown in Figure 6.

The problem with the Beneš network is that its complexity quickly grows when the number of inputs and outputs increases. For example, the 4x4 Beneš network requires 6 switches in 3 layers, the 8x8 Beneš network – 20 switches in 5 layers, the 16x16 Beneš network – 56 switches in 7 layers. Generally, for $2^N$ inputs and outputs, the network requires $2^{N-1}·(2·N-1)$ switches in $2·N-1$ layers.

Additionally, finding the configuration of switches that provides the required data routing is a complex task [11]. For small networks, it is possible to use a “brute force” approach to check all possible configurations and create a table with configurations needed for all possible routings. For an 8x8 network, it is necessary to analyze $2^{20}=1048576$ possibilities, and find the right configuration for $8!=40320$ possible permutations. For a 16x16 network, the number of possible switch configurations is $2^{56}\approx 7.2·{10}^{16}$, and the number of possible permutations is $16!\approx 2.1·{10}^{13}$. Therefore, neither analysis of all possible configurations nor storing the right configuration for each possible permutation is viable. Therefore, an 8x8 network is used as a basis for the concentrator with the structure shown in Figure 7.

A simple C utility was written to investigate the switch settings providing different data permutations, as shown in Listing 1. It generates a simple file with lines containing the value of the switch configuration word and the data permutation it generates. This generated file is then read by the Python utility, which creates a dictionary where the key is the particular permutation, and the value is the smallest value of the switch configuration word that provides it. Of course, not all possible permutations are needed in the concentrator. What is needed is routing the valid input words to the auxiliary or output record, starting from the first free output position, and preserving their order. Therefore, the Python utility iterates over all possible occupancies of the output record, and all possible combinations of the input data valid flags and finds the corresponding switch configuration. The analysis of the generated dictionary resulted in an interesting finding. The switch configuration values generating all needed permutations are always below 0x400. It means that only the lowest 11 bits are used. The switches in layers 3 to 5 are never used. That means that the Beneš network may be significantly reduced for that particular application using a limited set of permutations, as shown in Figure 8. Of course, that also resulted in a reduction of the C utility (see Listing 2).

3.2. Calculation of the Future Occupancy

The concentrator controller is also responsible for the calculation of the future occupancy of the output record. That value is needed in the next clock period. Therefore, its calculation is separated from finding the switch configuration. The switch configuration values may be stored in a BRAM-based look-up table. Adding pipeline registers before the Beneš network may compensate for the resulting delay. That’s not possible in case of future occupancy. That must be calculated in combinational logic. The implementation used in the project is shown in Listing 3.

Listing 1. C model of the 8x8 Beneš network

Listing 2. C model of the reduced 8x8 Beneš network

Listing 3. VHDL implementation of the combinational function calculating the future output record occupancy from the current one and the vector of valid input flags.

4. Practical Implementations of the Concentrator

Beneš-network-based concentrator appeared to be useful in different data acquisition systems currently developped. The design may be adjusted to particular needs, as shown in this section.

The solution based on 8x8 Beneš networks is needed for the GERI board [12] based on Trenz TEC0330 PCIe card [13]. This board, when supplemented with an FMC card with 8 SFP+ cages (e.g., [14]), enables concentration of data from 8 GBT Links to the DMA system [15] connected to the 8xGen 3 PCIe bus. The DMA system uses 256-bit data which may be treated as a record containing eight 32-bit words. Thence, the solution described in the previous section may be directly applied.

If the GERI board is connected to the TFC system [16], one SFP+ cage is used for the TFC communication. In that case, a smaller 7x8 Beneš network is needed. It may be obtained from an 8x8 network. The 7th input should be connected to both inputs in the last switch in layer 0. That eliminates a need to control that switch. Its control input may be connected to a constant value. As a result, the number of switches to be controlled is reduced from 12 to 11. That configuration is shown in Figure 9.

Special solutions are needed when it is necessary to concentrate data from more than eight inputs. Such a situation occurred in designing the firmware for the new CRI2 [17] readout board for CBM [18] experiment. Currently planned hardware solutions need to concentrate data either from 9 or 12 GBT Links delivering data at 160 MHz to a 256-bit wide word at a frequency up to 250 MHz. Of course, using the Beneš network with a size limited to 8x8 requires time multiplexing the input data. However, it does not require as high frequency as the high-speed polling method described in Section 2.1. For those designs, a dedicated data converter has been developed, which receives two input data sets at frequency $f_{IN}$, combines them, and then outputs them as three smaller sets at the frequency $f_{OUT}=\frac{3}{2}{f}_{IN}$. In the described system the Beneš network works at a frequency of 240 MHz, which is below 250 MHz.

In the case of 12 input links, that converter works with an 8x8 Beneš network, as shown in Figure 10. In the case of 9 input links, the converted data consist of three 6-word sets. They are delivered to the simplified 6x8 Beneš network, as shown in Figure 11.

5. Tests and Results

All five described variants of concentrators have been tested in simulations. The input data were generated as 32-bit words containing consecutive numbers starting from 0x100. The user could set probability that the data word is delivered to the individual input. The tests were performed for various values of probability: very low (0.01), low (0.1), medium (0.5), high (0.9), and very high (1.0). For all tested values of the probability, all five configurations of the concentrator worked correctly. All data delivered to the inputs were transmitted exactly once, and no invalid data were inserted into the output records. The waveforms from a simulation of a 7-inputs concentrator at probability 0.5 is shown in Figure 12.

The most complex configuration with 12 inputs has also been verified in hardware. The implementation was performed in two boards:

• KCU105 [19] AMD/Xilinx board, equipped with Kintex Ultrascale
  XCKU040-2FFVA1156E FPGA,
• TEC0330 [13] board from Trenz Electronic equipped with Xilinx Virtex-7 XC7VX330T-2FFG1157C FPGA.

For testing in hardware, a special testbench has been prepared with the structure shown in Figure 13. The input data for the concentrator are written via PCIe to the FIFO with asymmetric port width2. Similarly, the output from the concentrator is written to the second FIFO with 256-bit wide input and 32-bit wide output connected to another PCIe-accessible register. Additional control and status registers support resetting the FIFOs and the concentrator, starting the data transfer, and reading the status of both FIFOs.

The design was successfully compiled for both selected platforms. Timing closure was obtained for 160 MHz and 240 MHz frequencies. The resulting resource consumption is shown in Table 2.

The FPGA configured with the testbench FIFO is controlled with the uio_pci_generic driver and a simple Python script. The script resets the FIFOs and the concentrator. Then prepares the input data sets and writes them to the first FIFO. Afterward, it starts the data transfer. Finally, it reads the concentrated data from the second FIFO. For automated tests, the input data sets are prepared similarly to the simulations. The data words containing consecutive values were written to the consecutive inputs. With the probability defined by the user, each input could be skipped.

The tests were repeated multiple times with different probabilities of skipping the input and a different number of input data sets (of course, always smaller than the capacity of the input FIFO). In all tests, the concentrated data were correctly delivered to the output FIFO without losses or duplications.

6. Discussion and Conclusions

The novelty presented concentration method eliminates the disadvantages previous solutions. It does not require scanning inputs at high clock frequency as the high-frequency polling method (see Section 2.1). It does not require multiple width-converting FIFOs, does not inject empty data into the output stream, and does not disturb data time-ordering as the width conversion in the input channels (see Section 2.2).

The proposed method is well-suited for FPGA implementation. The Beneš network used to route the data consists of simple blocks (2x2 switches). No complex multiplexers are needed. Pipeline registers may be added inside the switches, resulting in short critical paths and high maximum clock frequency.

The Beneš network is quite old technology, but the paper describes its usage in a new application area. Additionally, it presents a method that significantly reduced the network size and simplified its configuration due to using the precalculated switches configurations.

The solution is based on an 8x8 Beneš network, but extending it for more inputs is described and tested.

The presented solution may improve the concentration of data, especially in concentrators for triggerless DAQs where a high-speed concentration of non-continuous data streams is needed.

The design is open-source and may be freely reused. The sources are available in repository [20].