Position within the page tree

Integrative Computational Design and Construction for Architecture (T3.0)
Research
Research Projects
RP 26-1 – AI-supported Collaborative Control and Trajectory Generation of Mobile Manipulators

AI-Supported Collaborative Control and Trajectory Generation of Mobile Manipulators for Indoor Construction Tasks

Research Project 26-1 (RP 26-1)

IntCDC Reserach Project 26-1 – AI-supported Collaborative Control of Mobile Manipulators

AI-SUPPORTED COLLABORATIVE CONTROL AND TRAJECTORY GENERATION OF MOBILE MANIPULATORS FOR INDOOR CONSTRUCTION TASKS

For the automation of indoor construction tasks, especially in existing buildings, mobile robots offer great flexibility and provide digital support for time-consuming tasks such as the positioning of building elements. In this research project, we focus on collaborative path and trajectory generation for a heterogeneous group of collaborating mobile robots equipped with sensors and different manipulator arms. The mobile manipulators can perform positioning tasks autonomously and with high precision. In order to optimise the construction in terms of execution time and accuracy, we investigate optimisation-based and AI-supported methods for decentralised path and trajectory generation to enable real-time capability and to allow an adaptive replanning of trajectories based on sensor feedback and environmental maps. This will require the development of indoor positioning and environmental sensing algorithms using e.g. total stations, laser scanners, cameras, and IMUs. We will exploit the capabilities of Simultaneous Localisation and Mapping (SLAM) algorithms both for positioning of the mobile manipulators and for acquiring 3D information of the respective indoor environment. The main objective will be to modify and develop high-precision SLAM approaches for real-time positioning and data acquisition for a digital twin of the construction site. This digital twin consists of an up-to-date semantically enriched consistent 3D model as represented by a labelled 3D point cloud or 3D voxel grid.

PRINCIPAL INVESTIGATORS

Prof. Dr.-Ing. Dr. h.c. Oliver Sawodny
Institute for System Dynamics (ISYS), University of Stuttgart
Prof. Dr.-Ing. habil. Volker Schwieger
Institute of Engineering Geodesy (IIGS), University of Stuttgart
Prof. Dr.-Ing. Uwe Sörgel
Institute for Photogrammetry (IFP), University of Stuttgart

TEAM

Dr.-Ing. Li Zhang (IIGS)
apl. Prof. Dr.-Ing. Norbert Haala (IFP)
Sahar Abolhasani (IIGS)
Alice Hierholz (ISYS)
Lena Joachim (IFP)
Vincent Reß (IFP)

PEER-REVIEWED PUBLICATIONS

2024
1. Ress, V., Zhang, W., Skuddis, D., Haala, N., & Soergel, U. (2024). SLAM for Indoor Mapping of Wide Area Construction Environments. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, X-2–2024, 209--216. https://doi.org/10.5194/isprs-annals-X-2-2024-209-2024
  - BibTeX
  BibTeX
  @article{isprs-annals-X-2-2024-209-2024, abstract = {Simultaneous localization and mapping (SLAM), i.e., the reconstruction of the environment represented by a (3D) map and the concurrent pose estimation, has made astonishing progress. Meanwhile, large scale applications aiming at the data collection in complex environments like factory halls or construction sites are becoming feasible. However, in contrast to small scale scenarios with building interiors separated to single rooms, shop floors or construction areas require measures at larger distances in potentially texture less areas under difficult illumination. Pose estimation is further aggravated since no GNSS measures are available as it is usual for such indoor applications. In our work, we realize data collection in a large factory hall by a robot system equipped with four stereo cameras as well as a 3D laser scanner. We apply our state-of-the-art LiDAR and visual SLAM approaches and discuss the respective pros and cons of the different sensor types for trajectory estimation and dense map generation in such an environment. Additionally, dense and accurate depth maps are generated by 3D Gaussian splatting, which we plan to use in the context of our project aiming on the automatic construction and site monitoring.}, author = {Ress, V. and Zhang, W. and Skuddis, D. and Haala, N. and Soergel, U.}, doi = {10.5194/isprs-annals-X-2-2024-209-2024}, journal = {ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences}, pages = {209--216}, title = {SLAM for Indoor Mapping of Wide Area Construction Environments}, url = {https://isprs-annals.copernicus.org/articles/X-2-2024/209/2024/}, volume = {X-2-2024}, year = 2024 }
2. Ress, V., Brändle, M., & Haala, N. (2024). Analysis and Implementation of Rotation-invariant Neural Network Architectures for Feature Extraction. 44. Wissenschaftlich-Technische Jahrestagung Der DGPF in Remagen – Publikationen Der DGPF, 32, 12–19. https://www.dgpf.de/src/tagung/jt2024/proceedings/paper/02_KKNP_dgpf2024_Ress_et_al.pdf
  - BibTeX
  BibTeX
  @article{ress2024analysis, author = {Ress, V. and Brändle, M. and Haala, N.}, journal = {44. Wissenschaftlich-Technische Jahrestagung der DGPF in Remagen – Publikationen der DGPF}, pages = {12-19}, title = {Analysis and Implementation of Rotation-invariant Neural Network Architectures for Feature Extraction }, url = {https://www.dgpf.de/src/tagung/jt2024/proceedings/paper/02_KKNP_dgpf2024_Ress_et_al.pdf }, volume = 32, year = 2024 }
3. Gienger, A., Stein, C., Lauer, A. P. R., Sawodny, O., & Tarín, C. (2024). Data-Based Reachability Analysis and Optimized Robot Positioning for Co-Design of Construction Processes. 2024 IEEE/SICE International Symposium on System Integration (SII).
  - BibTeX
  BibTeX
  @conference{noauthororeditor, author = {Gienger, Andreas and Stein, Charlotte and Lauer, Anja Patricia Regina and Sawodny, Oliver and Tarín, Cristina}, booktitle = {2024 IEEE/SICE International Symposium on System Integration (SII)}, title = {Data-Based Reachability Analysis and Optimized Robot Positioning for Co-Design of Construction Processes}, year = 2024 }
2023
1. Parlapanis, C., Frontull, M., & Sawodny, O. (2023). Feed-Forward Control of a Construction Vehicle’s Hydro-Mechanical Powertrain to Prevent Engine Stalling. Proceedings of the 2023 IEEE Conference on Systems, Man, and Cybernetics (SMC).
  - BibTeX
  BibTeX
  @inproceedings{christosparlapanis2023feedforward, author = {Parlapanis, Christos and Frontull, Matthias and Sawodny, Oliver}, booktitle = {Proceedings of the 2023 IEEE Conference on Systems, Man, and Cybernetics (SMC)}, title = {Feed-Forward Control of a Construction Vehicle's Hydro-Mechanical Powertrain to Prevent Engine Stalling}, year = 2023 }
2. Koelle, M., Walter, V., Schmohl, S., & Soergel, U. (2023). LEARNING ON THE EDGE: BENCHMARKING ACTIVE LEARNING FOR THE SEMANTIC SEGMENTATION OF ALS POINT CLOUDS. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, X-1/W1-2023, 945--952. https://doi.org/10.5194/isprs-annals-X-1-W1-2023-945-2023
  - BibTeX
  BibTeX
  @article{isprs-annals-X-1-W1-2023-945-2023, author = {Koelle, M. and Walter, V. and Schmohl, S. and Soergel, U.}, doi = {10.5194/isprs-annals-X-1-W1-2023-945-2023}, journal = {ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences}, pages = {945--952}, title = {LEARNING ON THE EDGE: BENCHMARKING ACTIVE LEARNING FOR THE SEMANTIC SEGMENTATION OF ALS POINT CLOUDS}, url = {https://isprs-annals.copernicus.org/articles/X-1-W1-2023/945/2023/}, volume = {X-1/W1-2023}, year = 2023 }
3. Hierholz, A., Gienger, A., & Sawodny, O. (2023). Cooperative Time-Optimal Trajectory Generation for a Heterogeneous Group of Redundant Mobile Manipulators. Proceedings of the 2023 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 1296–1301. https://doi.org/10.1109/AIM46323.2023.10196138
  - BibTeX
  BibTeX
  @inproceedings{hierholz2023cooperative, abstract = {In order to increase productivity and resource efficiency in the construction sector, new approaches are being pursued to automate tasks in the interior fitting of existing buildings. For this purpose, a heterogeneous group of redundant mobile manipulators is used for the time-consuming exact positioning of workpieces. The focus lies hereby on cooperative time-optimal trajectory generation for mobile manipulators. To minimize the effort for modeling and formulating the optimal control problem for the different robots, a unified approach based on the Unified Robotics Description Format is developed. The optimal control problem considers the equations of motion of the robots and existing kinematic and dynamic constraints as well as collisions. To enable cooperative trajectories, additional kinematic constraints are imposed on the poses of the end effectors of each mobile manipulator. Different numerical methods, degrees of model abstraction and cost-functionals are investigated with respect to the generated trajectories and the required computation time towards real-time capability for future adaptive trajectory generation using sensor feedback.}, author = {Hierholz, Alice and Gienger, Andreas and Sawodny, Oliver}, booktitle = {Proceedings of the 2023 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)}, doi = {10.1109/AIM46323.2023.10196138}, issn = {2159-6255}, month = {06}, pages = {1296-1301}, title = {Cooperative Time-Optimal Trajectory Generation for a Heterogeneous Group of Redundant Mobile Manipulators}, year = 2023 }
4. Haala, N., Zhang, W., Joachim, L., Skuddis, D., Abolhasani, S., Schwieger, V., & Soergel, U. (2023). Zum Potenzial von SLAM-Verfahren für geodätische Echtzeit-Messaufgaben. Allgemeine Vermessungsnachrichten (Avn), 163–172. https://gispoint.de/artikelarchiv/avn/2023/avn-ausgabe-052023/7879-zum-potenzial-von-slam-verfahren-fuer-geodaetische-echtzeit-messaufgaben.html
  - BibTeX
  BibTeX
  @article{haala2023potenzial, abstract = {Simultaneous Localization and Mapping (SLAM) zielt auf die Echtzeiterfassung einer Umgebungskarte mittels einer mobilen Roboterplattform, wobei gleichzeitig die Lokalisierung des Roboters bezüglich seiner Umgebung erfolgt. Zu diesem Zweck kombiniert der Roboter Navigationssensoren, wie IMU oder Odometer, mit Sensoren, wie Laserscanner und/oder monokulare, Stereo- oder RGB-D-Kameras, zur Erfassung der Umgebung. Analog zum bekannten Structure-from-Motion-Verfahren basiert die simultane Bestimmung der Aufnahmepose und der 3D-Karte der Umgebung auf einer photogrammetrischen Bündelblockausgleichung. Hierzu werden zugeordnete Bildpunkte und/oder Laserscans zwischen den jeweiligen Aufnahmen genutzt. SLAM-Verfahren haben mittlerweile einen beachtlichen Entwicklungsstand erreicht. Sie kommen in immer mehr praktischen Anwendungen zum Einsatz und können zunehmend auch für ingenieurgeodätische Messaufgaben genutzt werden. Von besonderem Interesse sind dabei Echtzeitanwendungen. In diesem Beitrag wird das Potenzial existierender SLAM-Verfahren für geodätische Aufgaben am Beispiel aktueller Projekte, wie der Erfassung von Baustellen mittels auf einem Kran montierten Kameras und der mobilen 3D-Kartierung von Innenräumen, demonstriert und diskutiert.}, author = {Haala, Norbert and Zhang, Wei and Joachim, Lena and Skuddis, David and Abolhasani, Sahar and Schwieger, Volker and Soergel, Uwe}, journal = {Allgemeine Vermessungsnachrichten (avn)}, month = {05}, pages = {163-172}, title = {Zum Potenzial von SLAM-Verfahren für geodätische Echtzeit-Messaufgaben}, url = {https://gispoint.de/artikelarchiv/avn/2023/avn-ausgabe-052023/7879-zum-potenzial-von-slam-verfahren-fuer-geodaetische-echtzeit-messaufgaben.html}, year = 2023 }
5. Haala, N., Zhang, W., Joachim, L., & Skuddis, D. (2023). SLAM für die Echtzeit-Erfassung von Innenräumen. 22. Internationale Geodätische Woche Obergurgl 2022, 188–179.
  - BibTeX
  BibTeX
  @article{haala2023echtzeiterfassung, author = {Haala, N. and Zhang, W. and Joachim, L. and Skuddis, D.}, editor = {Verlag, Wichmann}, isbn = {ISBN 978-3-87907-738-0}, journal = {22. Internationale Geodätische Woche Obergurgl 2022}, pages = {188-179}, title = {SLAM für die Echtzeit-Erfassung von Innenräumen }, year = 2023 }
6. Collmar, D., Walter, V., Koelle, M., & Soergel, U. (2023). A two-step approach for the acquisition of individual tree outlines using paid crowdsourcing. 43. Wissenschaftlich-Technische Jahrestagung Der DGPF, 22.-23. März 2023 in München, 163–173. https://doi.org/10.24407/KXP:1841079707
  - BibTeX
  BibTeX
  @inproceedings{TIBKAT:1841079707, address = {Stuttgart}, author = {Collmar, David and Walter, Volker and Koelle, Michael and Soergel, Uwe}, booktitle = {43. Wissenschaftlich-Technische Jahrestagung der DGPF, 22.-23. März 2023 in München}, doi = {10.24407/KXP:1841079707}, pages = {163-173}, publisher = {Geschäftsstelle der DGPF }, title = {A two-step approach for the acquisition of individual tree outlines using paid crowdsourcing}, url = {https://www.tib.eu/de/suchen/id/TIBKAT%3A1841079707}, year = 2023 }
7. Christos Parlapanis, Matthias Frontull, Oliver Sawodny. (2023). Feed-Forward Control of a Construction Vehicle’s Hydro-Mechanical Powertrain to Prevent Engine Stalling. Proceedings of the 2023 IEEE Conference on Systems, Man, and Cybernetics (SMC).
  - BibTeX
  BibTeX
  @inproceedings{christosparlapanis2023feedforward, author = {{Christos Parlapanis, Matthias Frontull, Oliver Sawodny}}, booktitle = {Proceedings of the 2023 IEEE Conference on Systems, Man, and Cybernetics (SMC)}, title = {Feed-Forward Control of a Construction Vehicle's Hydro-Mechanical Powertrain to Prevent Engine Stalling}, year = 2023 }

OTHER PUBLICATIONS

DATA SETS

Prof. Dr.-Ing. habil Volker Schwieger

Principal Investigator

Phone: +49 711 685 84040

E-Mail

Prof. Dr.-Ing. Uwe Sörgel

Principal Investigator

Phone: +49 711 685 83336

E-Mail

Prof. Dr.-Ing. habil. Dr. h.c. Oliver Sawodny

Board of Directors

Phone: +49 711 685 66302

E-Mail