Modifications to PilotNetOneHot model to incorporate distance-aware control
This week, we focused on the integration of new command functionalities. We introduced a distance-based command to help the model learn state transitions. This post outlines the modifications made to the PilotNetOneHot model to incorporate a new command Run50 an additional input, distance, which is a vector of size 1 range within (0,50) meters. Key changes were made in the model’s architecture.
This section explains how the data collection process integrates distance commands.
The distance vector is a critical part of the dataset as it determines how the vehicle behaves in response to the target distance. Specifically, when the vehicle is within a 50-meter range of a stopping point, the corresponding control commands will reflect a reduction in throttle and an increase in braking, teaching the model to stop the vehicle.
When the vehicle reaches or exceeds a distance of 50 meters from the target, the data reflects a command to stop. During training, the model will learn to replicate this behaviour, applying the brake and reducing the throttle when the vehicle is within 50 meters of the stopping point.
For example, during data collection, when the distance reaches 50 meters, the control signals recorded will show a reduction in throttle and an increase in braking:
distance = 50.0
controls = {
'throttle': 0.0, # Reduce throttle to 0 to stop the vehicle
'brake': 1.0 # Apply brake to stop the vehicle
}
In the collected data, the “Run50” command appears infrequently. Therefore, we extracted the segments containing this command for model training.
The structure of the PilotNetOneHot model remains unchanged. The key change is in the fully connected layers, where we account for the additional distance input. The name for the new model is PilotNetOneHotDistance.
Since we are adding a distance input with a size of 50, the updated initialization becomes:
self.fc_1 = nn.Linear(8 * 35 * 24 + 1 + num_hlc + num_light + 1, 50)
In the original model, the forward pass handled the concatenation of the image features, speed, HLC, and traffic light inputs. Now, we modify the forward pass to include the distance as well.
This modification allows the model to process the distance input along with the existing features. The distance input is concatenated with the flattened image features, speed, HLC, and traffic light data before being passed through the fully connected layers.
The updated forward pass in the training loop becomes:
for i, data in enumerate(train_dataloader):
img, speed, hlc, light, distance, controls = data # Include distance
img = img.to(device)
speed = speed.to(device)
hlc = hlc.to(device)
light = light.to(device)
distance = distance.to(device)
controls = controls.to(device)
# Forward pass
outputs = model(img, speed, hlc, light, distance)
loss = criterion(outputs, controls)
# Backward pass and optimization
optimizer.zero_grad()
loss.backward()
optimizer.step()
In the video, the final distance is set to 44 meters. This could be because, within the 0–50 meter training, the model encounters situations like waiting at traffic lights or stopping for obstacles. Such instances may reduce the model’s ability to learn precise control.
We also tested switching to a high-level command. In the following video, you can see an action transition, although there is a delay when restarting. This delay might occur because, during data collection, distinct states are triggered, and the model has mostly learned the stop command. As a result, after executing the Run50 command, the model defaults to a stop signal, needing extra time to switch to the next high-level command.