GSoC 2024 Project Recap
End-to-End Autonomous Vehicle Driving Based on Text-Based Instructions
GitHub | Website | Video | Meeting Logs | Model Card | Data Card
Summary
The 2024 GSoC project focused on the development of an end-to-end system in CARLA, built upon multimodal large models and a modified PilotNet architecture. Specifically, the agent component translates user inputs provided as natural language commands into structured control signals, including target distance and high-level control (HLC). These signals are processed by the control model, which uses distance-aware metrics to guide vehicle behavior. Furthermore, a Streamlit-based web app was developed to streamline interaction and training by providing an interface for generating and training user commands with LLMs. Finally, the entire system was tested in CARLA, where the generated control signals were applied to a simulated vehicle.
I would like to thank Google Summer of Code (GSoC), JdeRobot, and everyone who supported me throughout this project for the opportunity to contribute to the open-source community. I am especially grateful to my mentors for their guidance, particularly Sergio Paniego Blanco, whose insights and guidance were instrumental to my progress.
Weekly Blog
Week 21
We concluded the model setup by incorporating all the components into one model to test and evaluate.
Week 20
In Week 20, we improved CARLA’s vehicle control system by integrating distance-aware metrics. This update required refining data collection to track new metrics. A proof of concept showed successful action-switching based on distance, with plans to analyze datasets using HDF5 and scale up model training.
Week 19
During Week 19, we parsed CARLA’s Town02 map in OpenDRIVE format, generating structured test cases with start/end points.
Week 18
In Week 18, we introduced a distance-aware control method in CARLA to refine autonomous actions based on distance traveled, developed prompts for distance-specific commands, and validated real-time distance measurement in Town01.
Week 17
In this week, we explored integrating Visual Question Answering with reinforcement learning to enhance autonomous driving.
Weeks 15-16
In Week 15 and 16, we explored ideas to enhance autonomous vehicle capabilities through natural language instructions, particularly for complex, multi-turn interactions. Key efforts included compiling a list of potential research directions, assessing technical feasibility.
Week 14
In Week 14, we developed an "Ideas List" to explore potential research paths and merged new app updates into the main branch, including model testing in CARLA and a Streamlit web app. Documentation was refreshed, and recent progress was shared on social media.
Weeks 12-13
For the Weeks of 12 and 13, we focused on improving model evaluation and setting future research goals. Key tasks included enhancing model accuracy, addressing deployment issues, and preparing a social media update with a refined demo video.
Weeks 10-11
In Week 10 and 11, we focused on preparing for the mid-term evaluation, improving dataset generation, and building a Streamlit app to streamline model training and analysis. Key challenges included managing model storage and refining evaluation techniques.
Week 9
This week’s update covers progress in using LLMs to generate training data, training a BERT model, and integrating it into CARLA to improve the simulator's response to human instructions.
Week 8
This week focused on improving model performance, and flexibility. Key updates included setting up CARLA with a graphical interface, optimizing model training and refactoring the codebase.
Week 7
We then added command-based controls and tested various configurations of the model for improving its accuracy. We were trying to fine-tune the responsiveness of this model in various conditions.
Week 6
We have worked this week on fine-tuning the training to improve its efficiency and get early accuracy. Tuning the parameters contributed to making the process of training more stable, which is a very positive phenomenon to witness.
Weeks 3-5
In these two weeks, we focused on model refinement and optimized the collection of data. We worked to smoothen the workflows for higher efficiency and higher accuracies in our data.
Week 2
We developed improved data generation techniques and set up initial model training.
Week 1
During this week, we set up the skeleton of the project along with initial data pipelines and prepared the ground so that core development tasks could be rolled out.
Community Bonding Week 2
Our attention was placed on planning milestones. The coherence in the community brought better understanding of the roles to be played within the project in the coming stages.
Community Bonding Week 1
The blog website setup, commencement of literature research, and laying the backbone for the project were some of the initial works. This included resource organization and initial documentation in preparation for the project kickoff.