Preliminaries

This week our primary goal was to integrate our current model into Behavior Metrics, completing the entire pipeline for the project. Additionally, we expanded our dataset by training the model on 160 episodes, up from the previous 80. While this typically augments learning, our evaluation results indicated marginal improvements from our prior model.

Objectives

  • Update traffic generation PR
  • Add more data
  • Add model as new brain in Behavior Metrics
  • Include more information in evaluation results (e.g. collision per km)

Execution

Training on a larger dataset

More data usually improves learning. This week we increased the dataset to twice the size. Below is the evaluation results for 50 testing episodes. Notably, there wasn’t a significant improvement in terms of overall driving score and succss rate compared to the previous combined control model models.

  • Driving Score: 0.54
  • Success Rate: 0.36
  • Weighted Success Rate: 0.47
  • Infractions:
    • Wrong Turn: 18
    • Red Light: 1
    • Collision w/ Other Vehicle: 10
    • Collision w/ Other: 40
    • Junction Time Out: 10
    • Non-junction Time Out: 4 Below are two video demonstrations - one showcasing a successful run, and the other detailing a failed attempt.

Behavior Metrics integration

This week, This week the bulk of the work lies on adding our segmentation-based imitation learning model to BehaviorMetrics as a new brain. Specifically, we added the model as a new brain in Behavior Metrics and the required utilities. A PR has been created to reflect the following changes:

  • Added new yaml file, launch file, and object file as example
  • Added new brain for loading imitation learning pytorch models
  • Added ir_models directory for storing models
  • Added utils function for running imitation learning models
  • Added PilotNetOneHot model architecture implementation

The following is a demonstration where our model gets loaded into Behavior Metrics: