Preliminaries

This week, we continue to work on model enhancement. Despite expanding our dataset, model performance stayed the same. As a result, we opted for experimenting with different architectures. Interestingly, a deeper PilotNet structure didn’t always translate to better outcomes.

Simultaneously, we worked on expanding Behavior Metrics by introducing user customization for task types and integrating additional evaluation metrics.

Objectives

  • Test various model architectures.
  • Add “Follow Lane” and “Follow Route” as different tasks in Behavior Metrics
  • Incorporate more evaluation metrics into Behavior Metrics.
  • Dataset balancing

Execution

A Deeper Network

We expanded the PilotNet network with more convolutional and fully connected layers. Yet, the model’s performance remained unchanged. Key results included:

  • Driving Score: 0.48
  • Success Rate: 0.36
  • Weighted Success Rate: 0.56
  • Infractions:
    • Wrong Turn: 12
    • Red Light: 0
    • Collision w/ Other Vehicle: 7
    • Collision w/ Other: 150
    • Junction Time Out: 13
    • Non-junction Time Out: 7

Behavior Metrics Integration

Tasks in Behavior Metrics are now categorized as:

  • FollowLane: The basic task without turn instructions.
  • RandomRoute: Directions are chosen randomly at intersections.
  • FollowRoute: Route directions are loaded from a .txt file.

Users can customize their simulations through the configuration file, as demonstrated below. Moreover, traffic can be toggled on/off based on user settings.

We’re also focusing on incorporating more detailed evaluation metrics into Behavior Metrics. Our challenge is determining the simulation’s endpoint. To precisely measure metrics like success rate and driving scores, we must:

  • Establish defined test routes with start and end points.
  • Run the model across numerous episodes.

Currently, Behavior Metrics concludes a simulation once the stop button is pressed, without necessarily completing the route. We’re brainstorming ways to seamlessly integrate these new features into Behavior Metrics’ existing framework.