Coding Period Week 3

The third week of the coding period focused on improving the PerceptionMetrics documentation and preparing a nuImages image segmentation tutorial. After working on SemanticKITTI support and the LiDAR segmentation tutorial last week, I shifted focus to making the existing PerceptionMetrics workflows easier to understand and documenting the steps needed to evaluate an image segmentation model on nuImages.

Pull Requests

The main pull requests related to this week’s work are:

• nuImages tutorial: JdeRobot/PerceptionMetrics#599
• Documentation tabs: JdeRobot/PerceptionMetrics#602

  Documentation Improvements

This week, I reviewed the existing PerceptionMetrics documentation to understand what was already available and what still needed to be explained more clearly. The project already has structured documentation and autogenerated Python API pages.

After discussing the documentation structure with the mentors, we agreed that separate documentation tabs for datasets, models, metrics, and other major workflows would make the documentation easier to navigate. Based on that, I worked on adding more human-readable documentation pages for the main parts of the library. This included documentation for datasets, models, and metrics. The goal was to explain what is currently supported in the codebase, how the components are connected.

For the dataset documentation, I focused on describing the available dataset adapters and how they fit into the PerceptionMetrics evaluation workflow. I also checked which datasets are already connected to the CLI and which ones still need clearer CLI support or documentation.

For the model documentation, I documented the available model wrappers and clarified how model registration works in PerceptionMetrics.

I also added a metrics documentation page. This explains the segmentation metrics reported by PerceptionMetrics, including precision, recall, accuracy, F1-score, IoU, and Dice score.

nuImages Segmentation Tutorial

The second major task this week was preparing the nuImages image segmentation tutorial. As suggested by David, I fine-tuned a SegFormer model on the nuImages dataset and uploaded the trained checkpoint to Hugging Face. I then used this model in the tutorial and connected it to the PerceptionMetrics image segmentation evaluation workflow.

The tutorial covers loading the nuImages dataset through the PerceptionMetrics adapter, preparing the model configuration, loading the segmentation model, running predictions, and evaluating the results.

I created a model ontology that matches the labels predicted by the trained model. I also created an evaluation ontology with an ignore class so that classes not predicted by the model, such as background and vehicle.ego, can be handled properly during evaluation.

The tutorial exports the dataset labels into the evaluation ontology before running metrics. I added prediction inspection steps to the notebook. These print the full set of classes predicted by the model and the classes expected by the dataset, which helps debug ontology mismatches. I added visualization of the input image, prediction, and ground-truth label side by side as expected in a typical image segmentation tutorial.

The model initialization and dataset initialization are now separated into different cells, and I added markdown explanations around the ontology export, model wrapper setup, prediction inspection, and evaluation steps.

Alongside this, David reviewed and merged my commit for Cityscapes. This helped close one more part of the image segmentation dataset work and gave me a clearer reference for how the nuImages workflow should fit with the rest of the project.

Challenges

The main challenge this week was deciding how much detail should go into documentation pages and how much should stay in tutorials. The documentation pages should explain the supported components and expected formats, while tutorials should show a complete workflow with real examples.

Another challenge was handling the ontology mismatch in the nuImages tutorial. The model and dataset do not use exactly the same label space, so I had to carefully set up the model ontology, evaluation ontology, ignored classes, and translation files.

Next Steps

Next week, as suggested by David, I plan to look into the evaluation function and explore how it can work directly with ontology translation without requiring a separate dataset export step. This would make the evaluation workflow cleaner, especially for datasets where the model and dataset ontologies do not match exactly.

Another goal for next week is to explore GUI support for LiDAR segmentation. I plan to look for bugs in the current GUI workflow and take support from Praveena to refine the implementation and develop support for LiDAR segmentation evaluation.