publications

2024

1. Software X

   Behavior metrics: An open-source assessment tool for autonomous driving tasks

   Sergio Paniego, Roberto Calvo-Palomino, and JoséMaría Cañas

   SoftwareX 2024

   Abs Bib HTML Code Website

   The development and validation of autonomous driving solutions require testing broadly in simulation. Addressing this requirement, we present Behavior Metrics (BM) for the quantitative and qualitative assessment and comparison of solutions for the main autonomous driving tasks. This software provides two evaluation pipelines, one with a graphical user interface used for qualitative assessment and the other headless for massive and unattended tests and benchmarks. It generates a series of quantitative metrics complementary to the simulator’s, including fine-grained metrics for each particular driving task (lane following, driving in traffic, route navigation, etc.). It provides a deeper and broader understanding of the solutions’ performance and allows their comparison and improvement. It uses and supports state-of-the-art open software such as the reference CARLA simulator, the ROS robotics middleware, PyTorch, and TensorFlow deep learning frameworks. BehaviorMetrics is available open-source for the community.

   @article{PANIEGO2024101702,
     title = {Behavior metrics: An open-source assessment tool for autonomous driving tasks},
     journal = {SoftwareX},
     volume = {26},
     pages = {101702},
     year = {2024},
     issn = {2352-7110},
     doi = {https://doi.org/10.1016/j.softx.2024.101702},
     url = {https://www.sciencedirect.com/science/article/pii/S2352711024000736},
     author = {Paniego, Sergio and Calvo-Palomino, Roberto and Cañas, JoséMaría},
     keywords = {Evaluation tool, Autonomous driving, Imitation learning},
   }

2. IEE RAL - ICRA

   Model Optimization in Deep Learning Based Robot Control for Autonomous Driving

   Sergio Paniego, Nikhil Paliwal, and JoséMaría Cañas

   IEEE Robotics and Automation Letters 2024

   Abs Bib HTML Code Website

   Deep Learning (DL) has been successfully used in robotics for perception tasks and end-to-end robot control. In the context of autonomous driving, this work explores and compares a variety of alternatives for model optimization to solve the visual lane-follow application in urban scenarios with an imitation learning approach. The optimization techniques include quantization, pruning, fine-tuning (retraining), and clustering, covering all the options available at the most common DL frameworks. TensorRT optimization for specific cutting-edge hardware devices has been also explored. For the comparison, offline metrics such as mean squared error and inference time are used. In addition, the optimized models have been evaluated in an online fashion using the autonomous driving state-of-the-art simulator CARLA and an assessment tool called Behavior Metrics, which provides holistic quantitative fine-grain data about robot performance. Typically the performance of robot applications depends both on the quality of the control decisions and also on their frequency. The studied optimized models significantly increase inference frequency without losing decision quality. The impact of each optimization alone has also been measured. This speed-up allows us to successfully run DL robot-control applications even in limited computing hardware. All the work presented here is open-source, including models, weights, assessment tool, and dataset, for easy replication and extension.

   @article{Paniego2024,
     author = {Paniego, Sergio and Paliwal, Nikhil and Cañas, JoséMaría},
     journal = {IEEE Robotics and Automation Letters},
     title = {Model Optimization in Deep Learning Based Robot Control for Autonomous Driving},
     year = {2024},
     volume = {9},
     number = {1},
     pages = {715-722},
     doi = {10.1109/LRA.2023.3336244},
   }

2023

1. ROBOT2022

   RL-Studio: A Tool for Reinforcement Learning Methods in Robotics

   Pedro Cabo, Rubén Lucas, Ignacio Arranz, and 2 more authors

   In ROBOT2022: Fifth Iberian Robotics Conference 2023

   Abs Bib HTML Code

   This paper introduces RL-Studio, an open-source software that eases the implementation of reinforcement learning algorithms to solve a problem. This software enables the integration of any simulator to recreate the problem where a reinforcement learning algorithm can be applied. Some relevant advantages of this tool are the generalization of common software components and the unified architecture. RL-Studio permits to just focus on fine tuning the hyperparameters of the algorithm and redefine the goal and reward function of the previously integrated projects. In case a non-integrated scenario, algorithm or simulator is needed, RL-Studio also provides some already integrated libraries that can be reused to save time. It has been experimentally validated in research projects and some of the canonical problems such as Robot Mesh or Mountain Car.

   @inproceedings{FernandezCabo2022,
     author = {Fern{\'a}ndez de Cabo, Pedro and Lucas, Rub{\'e}n and Arranz, Ignacio and Paniego, Sergio and Ca{\~{n}}as, Jos{\'e} M.},
     editor = {Tardioli, Danilo and Matell{\'a}n, Vicente and Heredia, Guillermo and Silva, Manuel F. and Marques, Lino},
     title = {RL-Studio: A Tool for Reinforcement Learning Methods in Robotics},
     booktitle = {ROBOT2022: Fifth Iberian Robotics Conference},
     year = {2023},
     publisher = {Springer International Publishing},
     address = {Cham},
     pages = {502--513},
     isbn = {978-3-031-21062-4},
   }

2022

1. Arxiv

   Memory based neural networks for end-to-end autonomous driving

   Sergio Paniego, Sakshay Mahna, Utkarsh A. Mishra, and 1 more author

   2022

   Abs arXiv Bib Code

   Recent works in end-to-end control for autonomous driving have investigated the use of vision-based exteroceptive perception. Inspired by such results, we propose a new end-to-end memory-based neural architecture for robot steering and throttle control. We describe and compare this architecture with previous approaches using fundamental error metrics (MAE, MSE) and several external metrics based on their performance on simulated test circuits. The presented work demonstrates the advantages of using internal memory for better generalization capabilities of the model and allowing it to drive in a broader amount of circuits/situations. We analyze the algorithm in a wide range of environments and conclude that the proposed pipeline is robust to varying camera configurations. All the present work, including datasets, network models architectures, weights, simulator, and comparison software, is open source and easy to replicate and extend.

   @misc{PaniegoMemory2022,
     doi = {10.48550/ARXIV.2205.12124},
     author = {Paniego, Sergio and Mahna, Sakshay and Mishra, Utkarsh A. and Canas, JoseMaria},
     keywords = {Robotics (cs.RO), FOS: Computer and information sciences, FOS: Computer and information sciences},
     title = {Memory based neural networks for end-to-end autonomous driving},
     publisher = {arXiv},
     year = {2022},
     copyright = {Creative Commons Attribution Non Commercial No Derivatives 4.0 International},
     image = {/assets/img/pub_img/transformer_transform.png},
   }

2. Sensors

   Open Source Assessment of Deep Learning Visual Object Detection

   Sergio Paniego, Vinay Sharma, and José María Cañas

   Sensors 2022

   Abs Bib HTML Code

   This paper introduces Detection Metrics, an open-source scientific software for the assessment of deep learning neural network models for visual object detection. This software provides objective performance metrics such as mean average precision and mean inference time. The most relevant international object detection datasets are supported along with the most widely used deep learning frameworks. Different network models, even those built from different frameworks, can be fairly compared in this way. This is very useful when developing deep learning applications or research. A set of tools is provided to manage and work with different datasets and models, including visualization and conversion into several common formats. Detection Metrics may also be used in automatic batch processing for large experimental tests, saving researchers time, and new domain-specific datasets can be easily created from videos or webcams. It is open-source, can be audited, extended, and adapted to particular requirements. It has been experimentally validated. The performance of the most relevant state-of-the-art neural models for object detection has been experimentally compared. In addition, it has been used in several research projects, guiding in selecting the most suitable network model architectures and training procedures. The performance of the different models and training alternatives can be easily measured, even on large datasets.

   @article{PaniegoOSAssessment2022,
     author = {Paniego, Sergio and Sharma, Vinay and Cañas, José María},
     title = {Open Source Assessment of Deep Learning Visual Object Detection},
     journal = {Sensors},
     volume = {22},
     year = {2022},
     number = {12},
     article-number = {4575},
     url = {https://www.mdpi.com/1424-8220/22/12/4575},
     pubmedid = {35746357},
     issn = {1424-8220},
     doi = {10.3390/s22124575},
     image = {/assets/img/pub_img/transformer_transform.png},
   }

2021

1. CIN

   Robust Real-Time Traffic Surveillance with Deep Learning

   Jessica Fernández, José María Cañas, Vanessa Fernández, and 1 more author

   Computational Intelligence and Neuroscience 2021

   Abs Bib HTML Code

   Real-time vehicle monitoring in highways, roads, and streets may provide useful data both for infrastructure planning and for traffic management in general. Even though it is a classic research area in computer vision, advances in neural networks for object detection and classification, especially in the last years, made this area even more appealing due to the effectiveness of these methods. This study presents TrafficSensor, a system that employs deep learning techniques for automatic vehicle tracking and classification on highways using a calibrated and fixed camera. A new traffic image dataset was created to train the models, which includes real traffic images in poor lightning or weather conditions and low-resolution images. The proposed system consists mainly of two modules, first one responsible of vehicle detection and classification and a second one for vehicle tracking. For the first module, several neural models were tested and objectively compared, and finally, the YOLOv3 and YOLOv4-based network trained on the new traffic dataset were selected. The second module combines a simple spatial association algorithm with a more sophisticated KLT (Kanade–Lucas–Tomasi) tracker to follow the vehicles on the road. Several experiments have been conducted on challenging traffic videos in order to validate the system with real data. Experimental results show that the proposed system is able to successfully detect, track, and classify vehicles traveling on a highway on real time.

   @article{Fernndez2021RobustRT,
     title = {Robust Real-Time Traffic Surveillance with Deep Learning},
     author = {Fern{\'a}ndez, Jessica and Ca{\~n}as, Jos{\'e} Mar{\'i}a and Fern{\'a}ndez, Vanessa and Paniego, Sergio},
     journal = {Computational Intelligence and Neuroscience},
     year = {2021},
     volume = {2021},
   }