publications
2024
- NeurocomputingEnhancing end-to-end control in autonomous driving through kinematic-infused and visual memory imitation learningSergio Paniego, Roberto Calvo-Palomino, and JosĂ©MarĂa CañasNeurocomputing 2024
This paper presents an exploration, study, and comparison of various alternatives to enhance the capabilities of an end-to-end control system for autonomous driving based on imitation learning by adding visual memory and kinematic input data to the deep learning architectures that govern the vehicle. The experimental comparison relies on fundamental error metrics (MAE, MSE) during the offline assessment, supplemented by several external complementary fine-grain metrics based on the behavior of the ego vehicle at several urban test scenarios in the CARLA reference simulator in the online evaluation. Our study focuses on a lane-following application using different urban scenario layouts and visual bird-eye-view input. The memory addition involves architectural modifications and different sensory input types. The kinematic data integration is managed with a modified input. The experiments encompass both typical driving scenarios and extreme never-seen conditions. Additionally, we conduct an ablation study examining various memory lengths and densities. We prove experimentally that incorporating visual memory capabilities and kinematic input data makes the driving system more robust and able to handle a wider range of challenging situations, including those not encountered during training, in terms of reduction of collisions and speed self-regulation, resulting in a 75% enhancement. All the work we present here, including model architectures, trained model weights, comparison tool, and the dataset, is open-source, facilitating replication and extension of our findings.
@article{Paniego2024, title = {Enhancing end-to-end control in autonomous driving through kinematic-infused and visual memory imitation learning}, journal = {Neurocomputing}, volume = {600}, pages = {128161}, year = {2024}, issn = {0925-2312}, doi = {https://doi.org/10.1016/j.neucom.2024.128161}, url = {https://www.sciencedirect.com/science/article/pii/S0925231224009329}, author = {Paniego, Sergio and Calvo-Palomino, Roberto and Cañas, JosĂ©MarĂa}, keywords = {End-to-end autonomous driving, Imitation learning, Deep learning, Lane-following}, } - Phd ThesisEnd-to-end Vision-based Autonomous Driving using Deep LearningSergio Paniego2024
@article{PaniegoPhD2024, title = {End-to-end Vision-based Autonomous Driving using Deep Learning}, author = {Paniego, Sergio}, school = {Universidad Rey Juan Carlos}, year = {2024}, address = {Madrid, Spain}, url = {https://sergiopaniego.github.io/phd_thesis/}, } - NeurocomputingAutonomous Driving in Traffic with End-to-End Vision-based Deep LearningSergio Paniego, Enrique Shinohara, and JosĂ©MarĂa CañasNeurocomputing 2024
This paper presents a shallow end-to-end vision-based deep learning approach for autonomous vehicle driving in traffic scenarios. The primary objectives include lane keeping and maintaining a safe distance from preceding vehicles. This study leverages an imitation learning approach, creating a supervised dataset for robot control from expert agent demonstrations using the state-of-the-art Carla simulator in different traffic conditions. This dataset encompasses three different versions complementary to each other and we have made it publicly available along with the rest of the materials. The PilotNet neural model is utilized in two variants: the first one with complementary outputs for brake and throttle control commands along with dropout; the second one incorporates these improvements and adds the vehicle speed. Both models have been trained with the aforementioned dataset. The experimental results demonstrate that the models, despite their simplicity and shallow architecture, including only small-scale changes, successfully drive in traffic conditions without sacrificing performance in free-road environments, broadening their area of application widely. Additionally, the second model adeptly maintains a safe distance from leading cars and exhibits satisfactory generalization capabilities to diverse vehicle types. A new evaluation metric to measure the distance to the front vehicle has been created and added to Behavior Metrics; an open-source autonomous driving assessment tool built on CARLA that performs experimental validations of autonomous driving solutions.
@article{Paniego2025, title = {Autonomous Driving in Traffic with End-to-End Vision-based Deep Learning}, journal = {Neurocomputing}, pages = {127874}, year = {2024}, issn = {0925-2312}, doi = {10.1016/j.neucom.2024.127874}, url = {https://www.sciencedirect.com/science/article/pii/S0925231224006453}, author = {Paniego, Sergio and Shinohara, Enrique and Cañas, JosĂ©MarĂa}, keywords = {End-to-end autonomous driving, Imitation learning, Deep learning, Lane-following}, } - Software XBehavior metrics: An open-source assessment tool for autonomous driving tasksSergio Paniego, Roberto Calvo-Palomino, and JosĂ©MarĂa CañasSoftwareX 2024
The development and validation of autonomous driving solutions require testing broadly in simulation. Addressing this requirement, we present Behavior Metrics (BM) for the quantitative and qualitative assessment and comparison of solutions for the main autonomous driving tasks. This software provides two evaluation pipelines, one with a graphical user interface used for qualitative assessment and the other headless for massive and unattended tests and benchmarks. It generates a series of quantitative metrics complementary to the simulator’s, including fine-grained metrics for each particular driving task (lane following, driving in traffic, route navigation, etc.). It provides a deeper and broader understanding of the solutions’ performance and allows their comparison and improvement. It uses and supports state-of-the-art open software such as the reference CARLA simulator, the ROS robotics middleware, PyTorch, and TensorFlow deep learning frameworks. BehaviorMetrics is available open-source for the community.
@article{PANIEGO2024101702, title = {Behavior metrics: An open-source assessment tool for autonomous driving tasks}, journal = {SoftwareX}, volume = {26}, pages = {101702}, year = {2024}, issn = {2352-7110}, doi = {10.1016/j.softx.2024.101702}, url = {https://www.sciencedirect.com/science/article/pii/S2352711024000736}, author = {Paniego, Sergio and Calvo-Palomino, Roberto and Cañas, JosĂ©MarĂa}, keywords = {Evaluation tool, Autonomous driving, Imitation learning}, } - IEE RAL - ICRAModel Optimization in Deep Learning Based Robot Control for Autonomous DrivingSergio Paniego, Nikhil Paliwal, and JosĂ©MarĂa CañasIEEE Robotics and Automation Letters 2024
Deep Learning (DL) has been successfully used in robotics for perception tasks and end-to-end robot control. In the context of autonomous driving, this work explores and compares a variety of alternatives for model optimization to solve the visual lane-follow application in urban scenarios with an imitation learning approach. The optimization techniques include quantization, pruning, fine-tuning (retraining), and clustering, covering all the options available at the most common DL frameworks. TensorRT optimization for specific cutting-edge hardware devices has been also explored. For the comparison, offline metrics such as mean squared error and inference time are used. In addition, the optimized models have been evaluated in an online fashion using the autonomous driving state-of-the-art simulator CARLA and an assessment tool called Behavior Metrics, which provides holistic quantitative fine-grain data about robot performance. Typically the performance of robot applications depends both on the quality of the control decisions and also on their frequency. The studied optimized models significantly increase inference frequency without losing decision quality. The impact of each optimization alone has also been measured. This speed-up allows us to successfully run DL robot-control applications even in limited computing hardware. All the work presented here is open-source, including models, weights, assessment tool, and dataset, for easy replication and extension.
@article{Paniego2026, author = {Paniego, Sergio and Paliwal, Nikhil and Cañas, JosĂ©MarĂa}, journal = {IEEE Robotics and Automation Letters}, title = {Model Optimization in Deep Learning Based Robot Control for Autonomous Driving}, year = {2024}, volume = {9}, number = {1}, pages = {715-722}, doi = {10.1109/LRA.2023.3336244}, }
2023
- ROBOT2022RL-Studio: A Tool for Reinforcement Learning Methods in RoboticsPedro Cabo, Rubén Lucas, Ignacio Arranz, and 2 more authorsIn ROBOT2022: Fifth Iberian Robotics Conference 2023
This paper introduces RL-Studio, an open-source software that eases the implementation of reinforcement learning algorithms to solve a problem. This software enables the integration of any simulator to recreate the problem where a reinforcement learning algorithm can be applied. Some relevant advantages of this tool are the generalization of common software components and the unified architecture. RL-Studio permits to just focus on fine tuning the hyperparameters of the algorithm and redefine the goal and reward function of the previously integrated projects. In case a non-integrated scenario, algorithm or simulator is needed, RL-Studio also provides some already integrated libraries that can be reused to save time. It has been experimentally validated in research projects and some of the canonical problems such as Robot Mesh or Mountain Car.
@inproceedings{FernandezCabo2022, author = {Fern{\'a}ndez de Cabo, Pedro and Lucas, Rub{\'e}n and Arranz, Ignacio and Paniego, Sergio and Ca{\~{n}}as, Jos{\'e} M.}, editor = {Tardioli, Danilo and Matell{\'a}n, Vicente and Heredia, Guillermo and Silva, Manuel F. and Marques, Lino}, title = {RL-Studio: A Tool for Reinforcement Learning Methods in Robotics}, booktitle = {ROBOT2022: Fifth Iberian Robotics Conference}, year = {2023}, publisher = {Springer International Publishing}, address = {Cham}, pages = {502--513}, isbn = {978-3-031-21062-4}, }
2022
- ArxivMemory based neural networks for end-to-end autonomous drivingSergio Paniego, Sakshay Mahna, Utkarsh A. Mishra, and 1 more author2022
Recent works in end-to-end control for autonomous driving have investigated the use of vision-based exteroceptive perception. Inspired by such results, we propose a new end-to-end memory-based neural architecture for robot steering and throttle control. We describe and compare this architecture with previous approaches using fundamental error metrics (MAE, MSE) and several external metrics based on their performance on simulated test circuits. The presented work demonstrates the advantages of using internal memory for better generalization capabilities of the model and allowing it to drive in a broader amount of circuits/situations. We analyze the algorithm in a wide range of environments and conclude that the proposed pipeline is robust to varying camera configurations. All the present work, including datasets, network models architectures, weights, simulator, and comparison software, is open source and easy to replicate and extend.
@misc{PaniegoMemory2022, doi = {10.48550/ARXIV.2205.12124}, author = {Paniego, Sergio and Mahna, Sakshay and Mishra, Utkarsh A. and Canas, JoseMaria}, keywords = {Robotics (cs.RO), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {Memory based neural networks for end-to-end autonomous driving}, publisher = {arXiv}, year = {2022}, copyright = {Creative Commons Attribution Non Commercial No Derivatives 4.0 International}, image = {/assets/img/pub_img/transformer_transform.png}, } - SensorsOpen Source Assessment of Deep Learning Visual Object DetectionSergio Paniego, Vinay Sharma, and JosĂ© MarĂa CañasSensors 2022
This paper introduces Detection Metrics, an open-source scientific software for the assessment of deep learning neural network models for visual object detection. This software provides objective performance metrics such as mean average precision and mean inference time. The most relevant international object detection datasets are supported along with the most widely used deep learning frameworks. Different network models, even those built from different frameworks, can be fairly compared in this way. This is very useful when developing deep learning applications or research. A set of tools is provided to manage and work with different datasets and models, including visualization and conversion into several common formats. Detection Metrics may also be used in automatic batch processing for large experimental tests, saving researchers time, and new domain-specific datasets can be easily created from videos or webcams. It is open-source, can be audited, extended, and adapted to particular requirements. It has been experimentally validated. The performance of the most relevant state-of-the-art neural models for object detection has been experimentally compared. In addition, it has been used in several research projects, guiding in selecting the most suitable network model architectures and training procedures. The performance of the different models and training alternatives can be easily measured, even on large datasets.
@article{PaniegoOSAssessment2022, author = {Paniego, Sergio and Sharma, Vinay and Cañas, JosĂ© MarĂa}, title = {Open Source Assessment of Deep Learning Visual Object Detection}, journal = {Sensors}, volume = {22}, year = {2022}, number = {12}, article-number = {4575}, url = {https://www.mdpi.com/1424-8220/22/12/4575}, pubmedid = {35746357}, issn = {1424-8220}, doi = {10.3390/s22124575}, image = {/assets/img/pub_img/transformer_transform.png}, }
2021
- CINRobust Real-Time Traffic Surveillance with Deep LearningJessica Fernández, JosĂ© MarĂa Cañas, Vanessa Fernández, and 1 more authorComputational Intelligence and Neuroscience 2021
Real-time vehicle monitoring in highways, roads, and streets may provide useful data both for infrastructure planning and for traffic management in general. Even though it is a classic research area in computer vision, advances in neural networks for object detection and classification, especially in the last years, made this area even more appealing due to the effectiveness of these methods. This study presents TrafficSensor, a system that employs deep learning techniques for automatic vehicle tracking and classification on highways using a calibrated and fixed camera. A new traffic image dataset was created to train the models, which includes real traffic images in poor lightning or weather conditions and low-resolution images. The proposed system consists mainly of two modules, first one responsible of vehicle detection and classification and a second one for vehicle tracking. For the first module, several neural models were tested and objectively compared, and finally, the YOLOv3 and YOLOv4-based network trained on the new traffic dataset were selected. The second module combines a simple spatial association algorithm with a more sophisticated KLT (Kanade–Lucas–Tomasi) tracker to follow the vehicles on the road. Several experiments have been conducted on challenging traffic videos in order to validate the system with real data. Experimental results show that the proposed system is able to successfully detect, track, and classify vehicles traveling on a highway on real time.
@article{Fernndez2021RobustRT, title = {Robust Real-Time Traffic Surveillance with Deep Learning}, author = {Fern{\'a}ndez, Jessica and Ca{\~n}as, Jos{\'e} Mar{\'i}a and Fern{\'a}ndez, Vanessa and Paniego, Sergio}, journal = {Computational Intelligence and Neuroscience}, year = {2021}, volume = {2021}, }