Preliminaries

In the previous week, we successfully introduced support for a new follow-route task in Behavior Metrics. This task requires the agent to follow a sequence of high-level commands, making turns at junctions to reach a destination. However, with this new task comes the need for additional evaluation metrics tailored to its unique requirements. We have decided to incorporate the same evaluation metrics as those previously used during the development of our follow-route agent. These metrics include a route completion ratio, success rate, weighted success rate, counts of various infractions, and an overall driving score that considers route completion and infractions. The challenge now is to integrate these evaluation metrics into Behavior Metrics’ existing workflow.

Objectives

  • Update PR
  • Implement new evaluation metrics
  • Create a pull request for the new evaluation metrics
  • Produce the final demo video
  • Upload models and dataset to Hugging Face

Execution

Workflow

The image above outlines the current workflow of Behavior Metrics, particularly in the context of the newly added follow-route task:

  1. The main program, driver_carla.py, takes a YAML file containing configurations for the robot and experiments, a .launch file that initiates the CARLA ROS Bridge, and a test suite file containing all the testing routes.
  2. Within the YAML file, multiple models and worlds can be listed for evaluation. For each combination of (world, model, route), the Test Suite Manager is launched to execute a single experiment and record the evaluation metrics.
  3. Evaluation metrics are obtained from two sources: (1) data such as odometry and collision information published in the ROS topics by the CARLA ROS Bridge and (2) additional infractions calculated and recorded by the brain, including traffic light infractions and termination causes.
  4. These two sources of evaluation metrics are processed and merged to generate the final performance evaluation report.

Issues

Several issues need attention in the future:

  • Test Suite Manager almost never exits with code 0, causing repeated attempts.
  • A delay exists between stopping the recording of metrics and stopping the simulation.
  • The newly added evaluation metrics are not plotted in the output image.