Projects


D1: Invasive Software-Hardware Architectures for Robotics

Principal Investigators:

Prof. Dillmann, Prof. Asfour, Prof. Stechele

Scientific Researchers:

Manfred Kröhnert, Johny Paul

Abstract

Subproject D1 will explore the specific benefits and restrictions of invasive architectures in challenging real-time embedded systems and in particular in humanoid robotics. We will investigate the implementation of a cognitive robot control architecture with its different processing hierarchies, both on invasive TCPA and RISC-based MPSoC. The goal is to explore techniques of self-organisation to efficiently allocate available resources for the timely varying requirements of robotic applications. We expect that less computing resources are needed to fulfill the application requirements compared to traditional resource assignment at compile-time.

Synopsis

In subproject D1, we will explore the specific benefits and restrictions of invasive architectures in challenging real-time embedded systems and in particular in humanoid robotics. To be autonomous, humanoid robots should be able to learn to operate in the real world and to interact and communicate with humans. They have to model and reflectively reason about their perceptions and actions in order to learn, act, predict and react appropriately. Performing these tasks in the real world, especially in real-time, demands, not only substantial high computing power but also concurrent hardware and software architectures that support situation-dependent context switching between different applications such as natural dialogue management, visual perception of the environment, situation interpretation, task and motion planning, as well as action execution. Such architectures should allow to process the large amount and variety of sensory data in parallel and allocate resources appropriately.

In InvasIC, we will investigate the implementation of a cognitive robot control architecture with its different processing hierarchies using the aimed-at invasive hardware techonologies, language and architectural methodologies. The goal is to explore techniques of self-organisation (by means of invade/retreat) to efficiently use available resources fo the timely varying requirements of robotic applications. In particular, we will investigate a) the partitioning of robotics algorithms, e.g., computer vision, task and motion planning on both tightly-coupled processor arrays and heterogeneous parts of invasive architectures, and b) the adaption of such algorithms in order to efficiently use the full power of invasion.

Subproject D1 will contribute twofold to the InvasIC project: 1) We will contribute requirements and application code from a challenging real-time application, i.e., humanoid robotics. 2) We will evaluate the invasive computing paradigm in challenging real-time application scenarios, thus demonstrating the benefits and limitations of invasive computing. We expect that less computing resources are needed to fulfill our application requirements than with traditional resource assignment at compile-time. This will be validated by comparing performance and resource utilisation of invasive implementations with previous traditional implementations from KIT.

Approach

During the first phase the subproject D1 focuses on analyzing several algorithms from the robotics domain which are then adapted to take benefit of the proposed invasive architecture. Each algorithm forms a separate unit using invasive methods to optimize its behaviour. For e.g., several approaches are investigated for calculating the disparity map images. Ranging from single-core variants for processing multiple image pairs at the same time to multi-core variants which use different data mappings for processing a single stereo-image pair in parallel. However, the benefits of the invasive platform show up more prominentely when combining the single units to form a complete and integrated system.

Publications

[1] Nidhi Anantharajaiah, Tamim Asfour, Michael Bader, Lars Bauer, Jürgen Becker, Simon Bischof, Marcel Brand, Hans-Joachim Bungartz, Christian Eichler, Khalil Esper, Joachim Falk, Nael Fasfous, Felix Freiling, Andreas Fried, Michael Gerndt, Michael Glaß, Jeferson Gonzalez, Frank Hannig, Christian Heidorn, Jörg Henkel, Andreas Herkersdorf, Benedict Herzog, Jophin John, Timo Hönig, Felix Hundhausen, Heba Khdr, Tobias Langer, Oliver Lenke, Fabian Lesniak, Alexander Lindermayr, Alexandra Listl, Sebastian Maier, Nicole Megow, Marcel Mettler, Daniel Müller-Gritschneder, Hassan Nassar, Fabian Paus, Alexander Pöppl, Behnaz Pourmohseni, Jonas Rabenstein, Phillip Raffeck, Martin Rapp, Santiago Narváez Rivas, Mark Sagi, Franziska Schirrmacher, Ulf Schlichtmann, Florian Schmaus, Wolfgang Schröder-Preikschat, Tobias Schwarzer, Mohammed Bakr Sikal, Bertrand Simon, Gregor Snelting, Jan Spieck, Akshay Srivatsa, Walter Stechele, Jürgen Teich, Furkan Turan, Isaías A. Comprés Ureña, Ingrid Verbauwhede, Dominik Walter, Thomas Wild, Stefan Wildermann, Mario Wille, Michael Witterauf, and Li Zhang. Invasive Computing. FAU University Press, August 16, 2022. [ DOI ]
[2] Zehang Weng, Fabian Paus, Anastasiia Varava, Hang Yin, Tamim Asfour, and Danica Kragic. Graph-based task-specific prediction models for interactions between deformable and rigid objects. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5453–5460, 2021.
[3] Nael Fasfous, Manoj-Rohit Vemparala, Alexander Frickenstein, Mohamed Badawy, Felix Hundhausen, Julian Höfer, Naveen-Shankar Nagaraja, Christian Unger, Hans-Jörg Vögel, Jürgen Becker, Tamim Asfour, and Walter Stechele. Binary-lorax: Low-power and runtime adaptable xnor classifier for semi-autonomous grasping with prosthetic hands. In International Conference on Robotics and Automation (ICRA), 2021. [ http ]
[4] Fabian Paus and Tamim Asfour. Probabilistic representation of objects and their support relations. In International Symposium on Experimental Robotics (ISER), 2020.
[5] Fabian Paus, Teng Huang, and Tamim Asfour. Predicting pushing action effects on spatial object relations by learning internal prediction models. In IEEE International Conference on Robotics and Automation (ICRA), pages 10584–10590, 2020.
[6] Felix Hundhausen, Denis Megerle, and Tamim Asfour. Resource-aware object classification and segmentation for semi-autonomous grasping with prosthetic hands. In IEEE/RAS International Conference on Humanoid Robots (Humanoids), pages 215–221, 2019.
[7] Akshay Srivatsa, Sven Rheindt, Dirk Gabriel, Thomas Wild, and Andreas Herkersdorf. Cod: Coherence-on-demand – runtime adaptable working set coherence for dsm-based manycore architectures. In Dionisios N. Pnevmatikatos, Maxime Pelcat, and Matthias Jung, editors, Embedded Computer Systems: Architectures, Modeling, and Simulation, pages 18–33, Cham, 2019. Springer International Publishing.
[8] Dirk Gabriel, Walter Stechele, and Stefan Wildermann. Resource-aware parameter tuning for real-time applications. In Martin Schoeberl, Christian Hochberger, Sascha Uhrig, Jürgen Brehm, and Thilo Pionteck, editors, Architecture of Computing Systems – ARCS 2019, pages 45–55. Springer International Publishing, 2019. [ DOI ]
[9] Daniel Krauß, Philipp Andelfinger, Fabian Paus, Nikolaus Vahrenkamp, and Tamim Asfour. Evaluating and optimizing component-based robot architectures using network simulation. In Winter Simulation Conference, Gothenburg, Sweden, December 2018.
[10] Rainer Kartmann, Fabian Paus, Markus Grotz, and Tamim Asfour. Extraction of physically plausible support relations to predict and validate manipulation action effects. IEEE Robotics and Automation Letters (RA-L), 3(4):3991–3998, October 2018. [ DOI ]
[11] Johny Paul. Image Processing on Heterogeneous Multiprocessor System-on-Chip using Resource-aware Programming. Dissertation, Technische Universität München, July 25, 2017.
[12] Fabian Paus, Peter Kaiser, Nikolaus Vahrenkamp, and Tamim Asfour. A combined approach for robot placement and coverage path planning for mobile manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6285–6292, 2017.
[13] Jürgen Teich. Invasive computing – editorial. it – Information Technology, 58(6):263–265, November 24, 2016. [ DOI ]
[14] Stefan Wildermann, Michael Bader, Lars Bauer, Marvin Damschen, Dirk Gabriel, Michael Gerndt, Michael Glaß, Jörg Henkel, Johny Paul, Alexander Pöppl, Sascha Roloff, Tobias Schwarzer, Gregor Snelting, Walter Stechele, Jürgen Teich, Andreas Weichslgartner, and Andreas Zwinkau. Invasive computing for timing-predictable stream processing on MPSoCs. it – Information Technology, 58(6):267–280, September 30, 2016. [ DOI ]
[15] Manfred Kröhnert. A Contribution to Resource-Aware Architectures for Humanoid Robots. Dissertation, High Performance Humanoid Technologies (H2T), KIT-Faculty of Informatics, Karlsruhe Institute of Technology (KIT), Germany, July 22, 2016.
[16] Manfred Kröhnert, Raphael Grimm, Nikolaus Vahrenkamp, and Tamim Asfour. Resource-Aware Motion Planning. In IEEE International Conference on Robotics and Automation (ICRA), pages 32–39, May 2016. [ DOI ]
[17] Mirko Wächter, Simon Ottenhaus, Manfred Kröhnert, Nikolaus Vahrenkamp, and Tamim Asfour. The ArmarX Statechart Concept: Graphical Programming of Robot Behaviour. Frontiers in Robotics and AI, 3(33), 2016. [ DOI ]
[18] Johny Paul, Walter Stechele, Benjamin Oechslein, Christoph Erhardt, Jens Schedel, Daniel Lohmann, Wolfgang Schröder-Preikschat, Manfred Kröhnert, Tamim Asfour, Éricles R. Sousa, Vahid Lari, Frank Hannig, Jürgen Teich, Artjom Grudnitsky, Lars Bauer, and Jörg Henkel. Resource-awareness on heterogeneous MPSoCs for image processing. Journal of Systems Architecture, 61(10):668–680, November 6, 2015. [ DOI ]
[19] N. Vahrenkamp, M. Wächter, M. Kröhnert, K. Welke, and T. Asfour. The robot software framework armarx. Information Technology, 57(2):99–111, 2015.
[20] Johny Paul, Benjamin Oechslein, Christoph Erhardt, Jens Schedel, Manfred Kröhnert, Daniel Lohmann, Walter Stechele, Tamim Asfour, and Wolfgang Schröder-Preikschat. Self-adaptive corner detection on mpsoc through resource-aware programming. Journal of Systems Architecture, 2015. [ DOI ]
[21] Johny Paul, Walter Stechele, Éricles R. Sousa, Vahid Lari, Frank Hannig, Jürgen Teich, Manfred Kröhnert, and Tamim Asfour. Self-adaptive harris corner detector on heterogeneous many-core processor. In Proceedings of the Conference on Design and Architectures for Signal and Image Processing (DASIP). IEEE, October 2014. [ DOI ]
[22] Manfred Kröhnert, Nikolaus Vahrenkamp, Johny Paul, Walter Stechele, and Tamim Asfour. Resource prediction for humanoid robots. In Proceedings of the First Workshop on Resource Awareness and Adaptivity in Multi-Core Computing (Racing 2014), pages 22–28, May 2014. [ arXiv ]
[23] Éricles Sousa, Vahid Lari, Johny Paul, Frank Hannig, Jürgen Teich, and Walter Stechele. Resource-aware computer vision application on heterogeneous multi-tile architecture. Hardware and Software Demo at the University Booth at Design, Automation and Test in Europe (DATE), Dresden, Germany, March 2014.
[24] Johny Paul, Walter Stechele, Manfred Kröhnert, Tamim Asfour, Benjamin Oechslein, Christoph Erhardt, Jens Schedel, Daniel Lohmann, and Wolfgang Schröder-Preikschat. Resource-aware harris corner detection based on adaptive pruning. In Proceedings of the Conference on Architecture of Computing Systems (ARCS), number 8350 in LNCS, pages 1–12. Springer, February 2014. [ DOI ]
[25] Johny Paul, Walter Stechele, Manfred Kröhnert, Tamim Asfour, Benjamin Oechslein, Christoph Erhardt, Jens Schedel, Daniel Lohmann, and Wolfgang Schröder-Preikschat. A resource-aware nearest neighbor search algorithm for K-dimensional trees. In Proceedings of the Conference on Design and Architectures for Signal and Image Processing (DASIP), pages 80–87. IEEE Computer Society Press, October 2013.
[26] Éricles Sousa, Alexandru Tanase, Vahid Lari, Frank Hannig, Jürgen Teich, Johny Paul, Walter Stechele, Manfred Kröhnert, and Tamim Asfour. Acceleration of optical flow computations on tightly-coupled processor arrays. In Proceedings of the 25th Workshop on Parallel Systems and Algorithms (PARS), volume 30 of Mitteilungen – Gesellschaft für Informatik e. V., Parallel-Algorithmen und Rechnerstrukturen, pages 80–89. Gesellschaft für Informatik e.V., April 2013.
[27] Kai Welke, Nikolaus Vahrenkamp, Mirko Wächter, Manfred Kröhnert, and Tamim Asfour. The armarx framework - supporting high level robot programming through state disclosure. In Informatik 2013 Workshop on robot control architectures, 2013.
[28] David Schiebener, Julian Schill, and Tamim Asfour. Discovery, segmentation and reactive grasping of unknown objects. In 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 71–77, November 2012.
[29] Johny Paul, Andreas Laika, Christopher Claus, Walter Stechele, Adam El Sayed Auf, and Erik Maehle. Real-time motion detection based on sw/hw-codesign for walking rescue robots. Journal of Real-Time Image Processing, pages 1–16, 2012. [ DOI ]
[30] Johny Paul, Walter Stechele, Manfred Kröhnert, Tamim Asfour, and Rüdiger Dillmann. Invasive computing for robotic vision. In Proceedings of the 17th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 207–212, January 2012. [ DOI ]
[31] Jürgen Teich, Jörg Henkel, Andreas Herkersdorf, Doris Schmitt-Landsiedel, Wolfgang Schröder-Preikschat, and Gregor Snelting. Invasive computing: An overview. In Michael Hübner and Jürgen Becker, editors, Multiprocessor System-on-Chip – Hardware Design and Tool Integration, pages 241–268. Springer, Berlin, Heidelberg, 2011. [ DOI ]
[32] Jürgen Teich. Invasive algorithms and architectures. it - Information Technology, 50(5):300–310, 2008.