Summary

The past 18 weeks have been an incredible journey filled with learning, challenges, and achievements. In the initial phase, we successfully trained an autonomous driving agent that exhibited obstacle avoidance skills and could follow a predefined route while responding to high-level turning commands. This agent was built using imitation learning and deep neural networks. During the latter part of the program, our focus shifted to expanding the capabilities of Behavior Metrics. We introduced support for generating traffic, allowing users to configure scenarios with other vehicles and pedestrians. Additionally, we added a new follow-route task, enabling agents to follow predefined routes involving various turns and junctions. To accurately evaluate the performance of agents in these new scenarios, we designed and implemented new evaluation metrics. These metrics included route completion ratios, success rates, weighted success rates, and detailed infraction tracking, providing a comprehensive assessment of agent performance.

None of this would have been possible without the guidance and support of my mentors, Sergio Paniego Blanco and Nikhil Paliwal, whose expertise and dedication were invaluable throughout this journey. I’d also like to express my gratitude to Jose Maria Canas for his insightful contributions.Thank you to Google for organizing GSoC, and to everyone who has been a part of this journey!

Below is a brief overview of the work completed.

Demo

Code

Issues

  • #603 Add a launch for Town 01 with parked vehicles
  • #613 Town01 scenario with dynamic objects
  • #625 Generate traffic using CARLA Python API

Pull Requests

  • #606 new launch file for Town01 with a stopped car
  • #620 added on npc vehicle with ad agent
  • #626 generate traffic
  • #627 Update instructions on installing carla-birdeye-view
  • #632 Added segmentation based imitation learning model as a new brain
  • #643 Add support for a new follow-route task and evaluation metrics

Weekly Progress

Week 17

In Week 17, the main focus was on expanding the evaluation metrics in Behavior Metrics to accommodate the new “follow-route” task.

Week 16

In Week 16, the primary focus was on developing Behavior Metrics. Specifically, the team worked on implementing the new “follow-route” task in Behavior Metrics.

Week 15

During Week 15, the focus was on improving model performance by experimenting with different architectures. Simultaneously, work continued on expanding Behavior Metrics by introducing user customization for different task types and integrating additional evaluation metrics.

Week 14

During Week 14, the primary focus was on integrating the current model into Behavior Metrics to complete the project pipeline. The model was successfully added as a new brain in Behavior Metrics, enabling its use within the platform.

Week 12 & 13

During Weeks 12 and 13, there was a focus on improving the model’s obstacle avoidance capabilities.

Week 11

In Week 11, efforts to enhance the model’s performance, particularly in obstacle avoidance, continued. Additionally, progress was made in integrating traffic generation functionality into the Behavior Metrics platform.

Week 10

In Week 10, the focus shifted towards refining and evaluating the model’s performance. Efforts also began to integrate the model into the Behavior Metrics platform, expanding its capabilities to include traffic.

Week 9

In Week 9, we implemented Data Aggregation (DAgger) to iteratively enhance the model’s behavior and refined the evaluation metrics to detect whether the model correctly follows turning instructions.

Week 8

In Week 8, we continued to address the “halting” problem encountered in the model, particularly when making turns at intersections. The team explored various strategies to optimize data collection, including data trimming and prioritization.

Week 7

In Week 7, the focus was on enhancing the model’s performance, particularly addressing the “halting” problem where the agent occasionally stops and gets stuck during navigation.

Week 6

In Week 6, the main focus was on enhancing the evaluation process by incorporating more sophisticated metrics inspired by the CARLA Leaderboard.

Week 5

In Week 5, the primary focus was on refining the model’s adherence to traffic lights by incorporating traffic light status as an additional input and experimenting with one-hot encoding for high-level commands.

Week 4

This week, efforts were focused on enhancing the model’s ability to make turns at intersections in any direction by incorporating high-level commands. Data collection was adjusted to record these commands, and the model architecture was updated to accommodate them.

Week 3

This week, the focus was on improving the versatility of the lane-following model by enabling it to navigate routes with turns and intersections.

Week 2

In the second week of coding, the focus was on improving the training data quality. The primary enhancement was the introduction of noise injection into the expert agent’s control commands to simulate recovery from disturbances. This approach proved effective in teaching the model how to auto-correct when it deviates from the center of the lane.

Week 1

Implemented a data collection tool and collecting sample data, primarily consisting of simple scenarios with straight routes and dynamic obstacles. Simultaneously, a modified model, DeepestLSTMTinyPilotNet, was explored and trained on the collected data.

Community Bonding Week 2

During the second week of community bonding, the focus was on researching data collection methods for the project.

Community Bonding Week 1

The week involved setting up the blog website, conducting literature research, and laying the project’s groundwork.