Preliminaries

Starting this week, our primary focus shifts towards the development of Behavior Metrics. Initially, Behavior Metrics only supported the follow-lane task, where the robot circulates around a map without making turns. In this week’s efforts, we continued to implement the new follow-route task in Behavior Metrics. This task allows users to evaluate their models on custom pre-defined routes, each with a specified start and end location and distance. The agent must navigate from the start to the end, following high-level commands for correct turns at junctions.

Objectives

  • Understand the generation of Behavior Metrics’ evaluation metrics
  • Implement script mode for the follow-route task
  • Implement route loading functionality
  • Implement termination mechanism

Execution

Follow-Route Task

This week’s achievements in Behavior Metrics development include:

  • Enabling the loading of a test suite containing various routes, each defined by the map name, start and end location coordinates, and route length.
  • Implementing script mode for running multiple test routes consecutively, recording and outputting evaluation results after each run.
  • Detection and recording of traffic light infractions by calculating the actual vehicle turn angles at each junction and comparing them to the provided high-level turning instructions.

Moreover, we conducted an in-depth exploration of the codebase to comprehend how existing evaluation metrics are generated after each run. In the upcoming week, we plan to complete the implementation of the follow-route task and expand Behavior Metrics to include new evaluation metrics for this task.

Demo

In the video below, we demonstrate how script mode is used to run multiple testing episodes sequentially for the follow-route task in Behavior Metrics: