Projects

B4: Hardware Monitoring System and Design Optimisation for Invasive Architectures

Principal Investigators:

Prof. Schmitt-Landsiedel, Prof. Schlichtmann

Scientific Researchers:

Qingqing Chen, Elisabeth Glocker, Shushanik Karapetyan, Daniel Müller-Gritschneder, Christoph Werner,
Martin Wirnshofer

Abstract

Subproject B4 is dedicated to the assessment of operating conditions of the invasive computing hardware, the communication of this information and optimisation of the required monitoring resources. To measure these parameters, monitor circuits are designed and their number and placement in the invasive architecture is optimised. A parameterised model for this status information, including power consumption, temperature and maximum possible performance of computing blocks as well as their ageing-related degradation status, is provided for system simulation, optimisation and emulation on the FPGA-demonstrator.
In the first funding phase, Project B4 has developed concepts for monitoring invasive computing systems (both RISC and TCPA tiles). Specifically, concepts for monitoring power, temperature and ageing have been investigated. Communication interfaces between the monitors and higher levels of invasive computing systems have been explored. A control loop concept has been developed. For the essential monitoring concepts, a method has been developed to emulate them on an FPGA. The major challenge for FPGA emulation was that most monitors contain analogue circuits. With the achieved FPGA emulation, our concepts can be evaluated in the context of an entire invasive computing system even without an ASIC hardware implementation.

Synopsis

Integrated circuits today and even more in the future are subject to significant variations—between different manufactured components (resulting from fluctuations in the manufacturing process) as well as over space (e.g., "hot spots" due to heavy local switching activity) and time (short-term resulting from fluctuations in the operating conditions such as supply voltage and temperature; long-term resulting from degradation effects due to ageing). Therefore, different processing elements even on the same invasive IC can exhibit significantly different processing capabilities and susceptibility to degradation resulting from processing loads. This also results in differing risk of IC failures.
Resource-aware programming as one of the most essential points of innovation of invasive computing shall enable an application to make its decisions for execution based on actual physical hardware properties. In order to allow invasive algorithms to exploit the state of the invasive hardware for optimal distribution of the load, this subproject will provide means to measure and communicate the specific status of a processing element. This requires new ways of hardware design optimisation specifically in view of the new capabilities of invasion, including the design of dedicated monitor circuits. This project considers optimisation strategies and design of corresponding circuits and interfaces including the demonstration by simulation, emulation on the FPGA hardware prototype platform and later on by implementation on ASIC hardware prototypes.
This comprises classification of potential monitor types and interfacing systems, circuit design and analysis. It also includes algorithmic analysis and optimisation with respect to the complete invasive system, to calibrate monitors and to optimise their number, their performance regarding accuracy and speed, and their placement. Interfacing and information propagation has to be optimised to ensure best possible utilisation of each processor block based on its individual capabilities. This will also potentially reduce manufacturing costs of invasive architecture ICs, as processor blocks can be utilised according to their individual capabilities, rather than having to discard processors that do not meet predefined performance requirements.
The activities in the first funding phase will also lay the groundwork to enable optimised invasive architecture implementations in ASICs in the second and third funding phase, by optimising power consumption and reducing susceptibility to manufacturing variations and age-dependent degradation.

Research goal in first funding phase

With the introduced resource awareness of an invasive computing system, applications have the ability to explore the system and make decisions for execution (e.g. number and selection of invaded cores) based on the current state of the hardware platform, including physical hardware properties. For realising an invasive multi-tile architecture, a closed-loop control system between applications, run-time support system (OctoPOS), agent system and the underlying hardware including the monitoring system is necessary. The monitoring system provides the system with the needed monitoring data to control the physical hardware conditions and to use knowledge about hardware-health during the resource allocation in the invade phases. This becomes even more important (especially with thousands or more processors integrated on a single chip) when considering the significantly different processing capabilities and susceptibility to degradation of modern integrated circuits as compared to older and more robust processes. So, the research goal of Project B4 that will be fulfilled at the end of Phase I has been to measure the specific status of hardware elements, preprocess these data and implement the overall monitoring system. To effectively monitor the invasive hardware, different parameters, such as temperature evolution, power consumption and maximum and age-dependent performance capability have to be monitored. A foundation for this was developed in Phase I. The monitoring of the latter two parameters will be implemented in Phase II. This information is communicated with different levels of detail to other system components: higher hardware layers, run-time support system (OctoPOS), agent system and applications. The system is then able to act considering the monitor information, e.g. during the invade phase to choose appropriate processing elements or to react, if a critical status is detected. In turn, these actions may influence the status of hardware elements and with that the measured monitoring data. Project B4 has considered optimisation strategies and the design of corresponding interfaces, including the demonstration by simulation and emulation on the FPGA hardware prototyping platform.

Methods

Hardware-monitoring concept and models:

In-situ delay monitoring: In [Wirnshofer, ISIC 2011] and [Wirnshofer, DDECS 2011], we have demonstrated the monitoring of the maximum possible performance in terms of speed and frequency by in-situ delay monitoring. Before Phase I, published performance/speed monitors were mostly critical path replicas [A. Drake, et al., "A distributed critical-path timing monitor for a 65nm high-performance microprocessor, ISSCC 2007]. In [Wirnshofer, DDECS 2012] we have demonstrated the use of in-situ delay monitors for use in adaptive voltage scaling (AVS) and have evaluated the performance improvement and power saving potential. In-situ delay monitors are enhanced flip-flops that observe the timing of the circuit. Critical, but not yet erroneous signal transitions are detected as pre-errors. The pre-error rate is used as indicator whether the remaining timing slack of the circuit is sufficient. By use of these in-situ delay monitors, all kinds of variation and ageing effects are detected inside the real circuit and thus reliable performance information is provided. When using this monitor type in an online AVS technique, the supply voltage can be regulated during normal circuit operation—without a need for test intervals. In [Aryan, ARS 2012], different designs to implement in-situ delay monitors have been presented and the reliability of the timing information as well as the power overhead have been carefully analysed.
Ageing monitoring: Before Phase I, monitors that determine more advanced system features (e.g. ageing status) were just attracting initial research efforts in the research community.
In [Lorenz, Tech. Report, 2011], we have demonstrated an innovative approach to periodically monitor the ageing of ICs during operation. The basic concept is to identify all paths that potentially might become critical during the lifetime of an IC. As different paths can age at different rates, the critical path can change during the life of an IC. Ageing depends on operating and environmental conditions and therefore cannot be determined exactly before an IC is actually being used. But it is possible to identify a range within which the delay of a path will always be, regardless of where specifically it resides within the manufacturing window and what operating conditions (temperature, supply voltage, or switching activity) it will experience. It turns out that if this window is considered, for many circuits the number of paths that can potentially become critical is reduced significantly, often by one or two orders of magnitude. Therefore, it appears to be an option to test these paths periodically during the operation of an IC to detect any ageing that might endanger correct computation. This approach can be considered as a supplement or an alternative to the methods discussed above.
The research presented in [Knoth, PATMOS 2011], [Knoth, DATE 2012], [Chen, PATMOS 2011], [Li, TCAD 2013], [Li, TCAD 2012] and [Chen, IET CDS 2012] addresses related topics. This will become especially useful for a future ASIC design of the invasive multi-tile architectures.
In [Knoth, PATMOS 2011], SWAT, a highly optimised statistical timing analyser for digital circuits has been presented that combines the accuracy of a transistor-level analysis with the performance of a gate-level analysis. SWAT is based upon a CSM (current source model) for logic cells which considers transistor ageing and process variation and employs waveform truncation and dedicated solvers to significantly improve analysis performance without noticeable loss of accuracy. Parameter variations and ageing can be handled by Monte Carlo simulations and by a special sensitivity propagation mode, which expresses arrival times as a function of local and global parameter variations. This will allow very fast, yet accurate analysis of an ASIC design, considering variations and ageing to ensure very robust InvasIC ASIC design. In [Knoth, DATE 2012], the emphasis is put on power analysis instead of timing analysis.
In [Chen, PATMOS 2011], a flip-flop timing model has been presented that allows interdependency of different computation stages to be analysed via a static timing analysis at gate level. This is done by breaking the timing boundaries by explicitly building the functional relationship between clock-to-q delay and timing parameters at the flip-flop data input. Ageing effects HCI (hot carrier injection) and NBTI (negative bias temperature instability) are also considered in the modelling to pave the way for precise and realistic ageing analysis. Application of this approach in system emulation and later on also ASIC design will improve design performance even further.
[Li, TCAD 2013] has investigated the challenges in hierarchical timing analysis considering process variations. With abstract statistical timing models containing interfacing constraints, this flow can reduce the complexity of design and verification of large SoC systems effectively. For each of the three basic circuit types (combinational, flip-flop-based and latch-controlled) methods to extract statistical timing models are proposed to prune the unnecessary timing information from the underlying modules. With additional methods for the reconstruction of correlation between modules and for system-level verification, the complete framework is several times faster than analysing the flattened circuit directly, therefore providing an efficient flow for statistical timing verification of invasive multi-tile architectures.
[Li, TCAD 2012] has evaluated the statistical timing performance of circuits with level-sensitive latches, which are widely used in high-performance designs, such as CPUs. Circuits of this type, however, impose more complexity in timing analysis due to latch transparency. With reduced iterations and graph transformations, the proposed method extracts setup-time constraints at latches and across sequential loops very efficiently, more than ten times faster than other state-of-the-art methods, while still maintaining a good accuracy in the computed minimum clock period in a parametric form. The proposed method contributes a fast tool for statistical timing evaluation in the optimisation iterations of invasive computing systems, in which the aforementioned latch circuits always serve as the source of flexibility and robustness.
[Chen, IET CDS 2012] has introduced a modelling framework for the timing behaviour of a flipflop by building a nonlinear functional relationship between the clock-to-q delay and the data/clock alignment. The proposed framework makes it possible to carry out static timing analyses at gate level taking into consideration the interdependency of different computation stages. An iterative timing analysis method is developed to find out whether a circuit can work at a given clock frequency and to determine the minimal acceptable clock period of the circuit. The new method will be able to further improve the performance and the yield of the ASIC design for invasive multi-tile architectures, especially when process variations and ageing are considered.

Implementation and emulation for FPGA demonstrator platform:

Since it has been decided that there will be no full InvasIC ASIC implementation, our focus for the first funding phase has changed somewhat: Instead of preparing the hardware demonstrator (ASIC implementation) as originally proposed, we have worked on the modelling, implementation and emulation of the monitors on the FPGA demonstrator platform, to enable an FPGA emulation an invasive multi-tile architecture in close cooperation with the whole CRC, especially together with Project B2 and Project B3.
Before the start of Phase I, monitoring of parameters such as power consumption, temperature, performance (in terms of speed or maximum operating frequency) was already state-of-the-art in modern high performance microprocessors, [Duarte et al., "Temperature sensor design in a high volume manufacturing 65nm CMOS digital process", CICC 2007] and [Tschanz et al. "Adaptive frequency and biasing techniques for tolerance to dynamic temperature-voltage variations and aging", ISSCC 2007]. These monitor data were used, e.g. for power limitation by frequency or supply voltage control or for complete shutdown of processing elements to prevent damage [Rotem et al. "Temperature measurement in the Intel CoreTM duo processor", 2006]. But they were not used to control and optimise the complete system. So, no sophisticated interaction between the physical parameter level and run-time support system or application layer were present in existing processors.
In the invasive computing architecture, hardware monitors, which are a necessary part of the resource management feedback control loop, have been included. Consequently those hardware monitors must also be included in the prototyping system. Hardware monitors such as processor core load, communication link load (e.g. AHB bus load and iNoC load) and memory access (e.g. cache miss rate) monitors that are fully digital circuits are easy to implement using the digital logic resources of an FPGA. However, other hardware monitors that are usually realised as analogue circuits are difficult to implement in the prototyping system, since our FPGA demonstrator platform, the Synopsys CHIPit system, is based on digital FPGA technology without any reconfigurable analogue circuit resources. Therefore, for FPGA prototyping, we have taken a real-time emulation approach for such analogue monitors including power monitors, temperature monitors and subsequently will take this approach also for ageing monitors in Phase II.
The figure below shows the structure of the implemented circuit of our real-time emulation approach for power and temperature monitoring.

Power monitoring and emulation on FPGA: Power monitors for processor cores of the RISC compute tiles, i.e. the LEON3 cores, have been emulated using a run-time instruction-energy look-up approach: An instruction-energy look-up table (LUT)—containing pre-characterised average energy consumption values for each kind of processor instruction—is looked up when a new incoming instruction is executed by a processor core. For a predefined time period (in accordance with monitoring frequency), the energy values per instruction are accumulated, and the accumulated value is divided by time at the end of the period to produce the power value for that period. Power monitor emulations for tightly-coupled processor arrays (TCPAs) have taken a different approach than those for LEON3 cores, since TCPA processing elements (PEs) are based on a VLIW architecture supporting instruction level parallelism (ILP). Therefore, a simple energy LUT construction and fast instruction type determination at run time are not feasible, and thus an event- counter-based energy model has been applied: Pre-characterised energy consumption values for subprocessor modules (e.g. ALU, register file and instruction decoder) are summed up and accumulated based on event counter status. Same as for LEON3 cores, the accumulated values are divided by time to produce power values for a predefined time period. The emulated real-time power consumption information as well as the accumulated energy data can be communicated to higher system levels not only for observation purposes and the evaluation of power and energy management strategies, but the power values are also used for the emulation of temperature monitors.
Temperature monitoring and emulation on FPGA: For the real-time emulation of the temperature monitor for RISC tiles, an approach that is based on the use of a power-temperature look-up table has been used. The LUT contains the resulting steady-state temperature for all possible power consumption values (for a predefined time period) received from the power monitor. Those temperature values are pre-characterised based on a thermal RC model: In this approach, the input power leads to a temperature difference because of thermal resistances (modelling steady-state behaviour) and thermal capacitances (modelling transient behaviour) that both describe the processor architecture environment. The temperature values for the LUT are obtained for every core (as the maximum steady-state temperature of the cores) taking all possible average power values and all possible placements for "active" cores into account and considering not only the core's own activity leading to a specific temperature value, but also the influence of neighbour core activities on this temperature. These results are mapped to LUT entries and are used to obtain the resulting steady-state temperature for every core for the predefined time period (in accordance with the monitoring frequency). For TCPA tiles, the same approach has been applied. But the different architecture has made it necessary to use another thermal RC model and different power consumption values. To the best of our knowledge, our approach has been the first one that deals with real-time FPGA emulation of such a power and temperature monitoring system. We presented our approach for temperature and power monitoring and emulation of FPGA for RISC tiles in [Glocker, RACING 2014] and [Glocker, Workshop Analogschaltungen 2014].
In-situ delay monitors on FPGA: In-situ delay monitors are novel hardware monitors which can be used to monitor and predict the reliability of the monitored circuit. It is basically possible to implement them in FPGA. However, since in an FPGA ageing phenomena do not take effect in reasonable test time and under normal operating conditions (i.e. temperature and supply voltage), accelerated ageing would have to be applied to the prototyping system, which is by no means an easy task for a CHIPit system. A simple solution will again be "emulation". For this, we have developed in cooperation with industry proprietary models for ageing in dependence on time and operating conditions in another project. We intend to employ these in the second funding phase and thus have realistic data for an integrated circuit solution available.

Integration and optimisation of individual monitor types on FPGA demonstrator platform:

Integration and optimisation of power and temperature emulation in RISC tile: Digital hardware monitors such as processor core load monitors and emulated analogue monitors including power and temperature monitors have been integrated with CiC for RISC tiles in cooperation with Project B3. The number and placement of the power and temperature monitors within the overall system has been optimised such that every core of a tile has one power monitor and each tile has one temperature monitor covering all cores of a tile (giving a maximum temperature for the complete core). The time period at which the monitors operate is predefined according to the monitoring frequency.
Integration and optimisation of power and temperature emulation in TCPA tile: For power and temperature monitoring, the monitoring system will not cover every PE present on a tile, but rather cover PE regions to keep the size of the monitoring system as small as possible and to still retrieve useful and sufficiently precise results. In [Glocker, ARS 2014] we presented the approach for temperature monitor modeling and emulation for TCPAs.

Optimisation of overall monitoring system on FPGA demonstrator platform: Feedback control loop of the monitoring system:

Before Phase I, the systematic optimisation of a monitoring system in terms of circuit types, required resolution, speed and monitoring frequency as well as their number and placement has not been subject of systematic research efforts. Also, we have not been aware of techniques that allow the calibration of a generic monitor to a specific design and use case.
Use of monitoring data for resource allocation: We have studied possible improvements that can be made if monitoring data are used during resource allocation to achieve different control targets. Taking temperature monitoring data for example, different task allocation techniques and application characteristics as well as different physical conditions such as package types, material parameters and cooling all result in different temperature scenarios. Also, reasonably priced processor packaging do not cover the worst case temperature hot spot scenario anymore, which would occur without an intelligent power and temperature monitoring and control as proposed here. So, hot spot temperatures must be avoided, e.g. by intelligent task allocation. In [Glocker, ARS 2013], we have modelled different scenarios in a multicore system and evaluated the temperature distribution of cores. In a multicore system, a reciprocal influence between the core temperatures of neighbouring cores occurs, so an intelligent active core placement in a non-full-usage scenario can further decrease the current temperature. We also evaluated different temperature limiting measures: The best choice is either an intelligent core choice—resulting from intelligent resource allocation—combined with lower usage-rates or lowering of the power consumption, e.g. by implementing supply voltage or frequency scaling. Since temperature should be regulated during run time, a combined implementation of different concepts and choosing a temperature limiting measure for the individual situation during run time appears to be the best solution.
Communication of monitor data/feedback control loop of the monitoring system: Instead of communicating monitor data of every monitor type through the whole invasive computing system, the monitoring data is "bundled": For using monitoring data for resource allocation and monitoring of the current hardware health, the monitoring data has to be given to the agent system—included in the run-time support system—that handles inter-tile resource allocation. The feedback control loop is shown in the figure below for a sample RISC compute tile.

In InvadeX10, several application classes (such as, e.g. high performance, communication intense, high reliability) have been defined in cooperation with Project A1, Project B2, Project B3, Project C1, Project D1 and Project D3, so that an application can express to which class it belongs This is important for realising the inter-tile resource allocation that fulfils the application needs. In every application class monitor data of different monitor types are bundled, abstracted and weighted in a way to fit the needs of the individual application class. The tile-local resource allocation is done for RISC compute tiles in the CiC. For TCPA compute tiles, the tile-local resource allocation is done by a Configuration & Communication Processor. Monitor data is also abstracted and weighted for tile-local resource allocation.

Publications