Hi 🤝, I’m Kartik. This was my open source project this summer. I’ve contributed to JdeRobot since January, and before that was a Research Intern at GahanAI, working on a humanoid robotics survey paper. My ML journey started about a year ago.

The starting point, measured during community bonding, was a 29.5GB image built across 30 layers, with a cold build taking 102 minutes 50 seconds.

My mentors were David Pascual-Hernández, Javier Izquierdo, Md. Shariar Kabir, and Nikhil Gupta. Javier reviewed and merged the seven PRs that shipped.

What Exists Now

RoboticsBackend is smaller than it was in May. My four merged PRs (cache cleanup, the ghost-layer fix, the PyTorch swap, the ARG/ENV fix) brought the image from 29.5GB to 18.2GB. Alongside that, not after it, Javier’s parallel PR removing legacy Gazebo 11 dependencies (#3875, merged 2026-06-23) took it to 10.7GB, a combined 18.8GB drop, roughly 64%. A working rootless-Podman path with GPU passthrough also exists via three more merged PRs: sudo apt install podman podman-compose plus the video and render groups.

Two things are tested but not shipped: a multi-stage rebuild of Dockerfile.dependencies_humble reached 9.55GB in testing, but it’s on hold since the size win didn’t justify the build complexity. A rootless file-permission fix is also open.

One known limitation ships as-is: Ctrl+C during podman-compose up doesn’t stop the stack, since podman-compose 1.0.6 doesn’t act on SIGINT. Workaround: a second terminal running podman-compose down.

Phase 1: Shrinking the Image

PR #3848 — Cache Cleanup

Both Dockerfiles were leaking build cache: apt cleanup wasn’t clearing package lists, and pip installs were missing --no-cache-dir. Both fixed, along with two dead packages, netron and seaborn. Full image: 29.5GB to 26.4GB; dependencies_humble dropped 3.4GB.

PR #3856 — The Ghost Layer

Cloning RoboticsInfrastructure, moving its contents, then deleting .git were three separate RUN instructions, so the ~4.6GB clone stayed baked in regardless. One atomic RUN fixed it. Full image: 26.4GB to 21.8GB.

PR #3863 — PyTorch Out, CUDA Wheels In (merged 2026-06-12)

David asked whether I could replace PyTorch with something lighter, since we only do inferencing, not training. The fix swapped in seven hard-pinned CUDA wheels plus a linker-cache entry to survive ROS 2 sourcing; Javier caught a protobuf crash in review, fixed by forcing Python’s pure protobuf implementation. Net reduction was smaller than the claimed 4.7GB; dependencies_humble sat at 12.8GB after merge.

PR #3890 — Two Small Correctness Fixes (merged 2026-07-20)

NVIDIA_VISIBLE_DEVICES had been a build ARG, silently breaking GPU visibility; changed to ENV. A second fix removed a dead source command with no effect. No size change; pure correctness.

The Multi-Stage Build (PR #3900 — open, on hold, not merged)

The plan was to split Dockerfile.dependencies_humble into a builder and a slim runtime stage, harder than expected since part of the source tree is still needed at runtime. Fixing an early regression brought the image from 11.9GB to 10.7GB, later refined to 9.55GB. It didn’t ship: build time roughly doubled, from 20 to 40 minutes, so it’s been open since July, still in draft.

PR #3953 — Rootless File Permissions (open, not merged)

Saving workspace files under Podman threw a blank error popup: five os.chmod(path, 0o777) calls fail under rootless containers, since Podman’s “root” can write to a file it doesn’t own but can’t change its permissions. The fix removes all five calls. Still open.

Phase 2: Podman, Rootless, and the GPU

Docker has a rootless mode too, but needs a daemon and manual setup; Podman is daemonless by design. The first Podman build failed on an unqualified base image reference, since Podman won’t assume Docker Hub by default. Fixed via Podman’s registry config, not the Dockerfile.

Getting the compose stack running took setup around Podman’s socket and image store. Autosave also failed on a file created under Docker: rootless Podman can write new content into a file it doesn’t own, but can’t chmod it, the same bug PR #3953 later fixes.

GPU passthrough needed the Container Device Interface (CDI) instead of Docker-style runtime hooks, which surfaced the hardest bug: nvidia-ctk generates CDI specs with a field, additionalGids, that Ubuntu 24.04’s shipped Podman (4.9.3) can’t parse, silently falling back to CPU. The passthrough that first worked had been tested on Podman 6.0.2; the real 4.9.3 reproduced the failure. The fix, from an upstream NVIDIA issue, was a compatibility flag generating an older-format spec 4.9.3 can read. Once pinned, it’s just sudo apt install podman podman-compose plus the video and render groups. Passthrough was then confirmed working via vglrun glxinfo.

Two PRs shipped here: #3959 (2026-08-07), new CDI compose files, and #3960 (2026-08-13), a -p flag switching launcher scripts to podman-compose.

Phase 3: Documentation

PR #3961 (merged 2026-08-13) added a standalone Podman setup guide: install, CDI setup, -p launch, GPU verification, and Docker migration, linked from existing docs.

Demo Video

https://youtu.be/XqefB9AnR3w

Pull Requests

Future Work

A CPU-only Windows/WSL2 test loaded the web interface under rootless Podman, but the robot didn’t respond to exercise code. Two more PRs continue that direction: #3966 (Windows GPU compose) and #3973 (WSL2 NVIDIA workflow). The compose setup also has 8 nearly-identical files across {user, dev} × {cpu, gpu, nvidia, nvidia-windows} needing updates.

Challenges and What I Learned

The hardest problem for me was to write the perfect documentation which needs to be very simple but also covers everything so that any user or developer does not need to sit for hours to start using RoboticsAcademy with Podman. I was able to find a way to run it in my system, but finding out the simplest to run it in every system was challenging. The other major one was the Podman GPU passthrough bug: an undocumented additionalGids field in the CDI spec that Ubuntu’s own shipped Podman couldn’t parse, silently falling back to CPU with no error at all. Finding it meant realizing that the environment I’d been developing and testing against, a newer Podman installed via Homebrew, wasn’t the environment a real user would have.

Thanks

Thanks to my mentors on this project: David Pascual-Hernández and Javier Izquierdo, and Nikhil Gupta and Md. Shariar Kabir.

Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.

Initial Attempt

The first test with user_humble_cpu.yaml showed that Podman could start the stack, but the current jderobot/robotics-academy:latest image could not initialize Django because the required react_frontend Python files were absent from the image. We verified this directly from the image contents, establishing that this was independent of Podman.

Development Workflow

We then moved to dev_humble_cpu.yaml. One important Podman-specific detail was relative bind-mount resolution. With the compose file under compose_cfg, paths such as ./database were resolved relative to that directory. Using the repository’s documented Windows approach of placing the development configuration at the repository root as docker-compose.yaml allowed the expected bind mounts to resolve correctly.

Frontend

Because the development configuration bind-mounts the repository into the container, the local frontend artifacts also needed to be generated. We installed the frontend dependencies, generated common.zip, and successfully built the frontend, producing webpack-stats.json.

The first browser run exposed a React runtime error. Rebuilding the frontend with an explicit NODE_ENV=production resolved that error and allowed the RoboticsAcademy web interface to load.

Current State

The CPU stack is now running successfully on Windows with WSL2, rootless Podman, and podman-compose. The RoboticsAcademy website is accessible on localhost:7164, and the Follow Line exercise interface and Gazebo simulation load correctly.

However, this does NOT yet mean the exercise itself is fully functional. The current remaining issue is robot execution/actuation: the Follow Line simulation is visible, but the car does not currently respond and move as expected when the exercise code is running.

We have therefore verified the containerized application and frontend stack, but not yet the complete end-to-end exercise workflow.

Next Step

Before moving to GPU testing, the CPU investigation should identify why the robot is not responding and verify one complete exercise execution under Podman.

Once that works, we can use the working CPU configuration as the baseline for investigating the GPU workflow.

The simplest documentation problem

One of the biggest challenges I faced during the whole coding period was finding the exact workflow any Linux user can follow to run Podman, while keeping it as simple as possible.

Last week, when I ran RoboticsAcademy with and without GPU passthrough, I was using Podman 6.0.2 and podman-compose 1.6.0. Neither of those comes from Ubuntu 24.04’s apt packages. I had installed Podman through Homebrew and podman-compose through pip. So I wanted to find a solution that works with Podman 4.9.3, the version Ubuntu 24.04 actually ships.

The blocker was that nvidia-ctk generates a CDI specification containing an additionalGids field, which Podman 4.9.3 cannot parse, so it rejects the whole file and finds no GPU. Thankfully NVIDIA had already proposed a solution in this issue: https://github.com/NVIDIA/nvidia-container-toolkit/issues/1860. Generating the spec with --feature-flag no-additional-gids-for-device-nodes produces an older format that 4.9.3 reads fine.

Once I was on 4.9.3, two other problems I had been documenting disappeared. Ubuntu’s podman-compose 1.0.6 and crun 1.14.1 both work correctly with it. They only break against the newer Podman I had been using. So the whole installation step is now just sudo apt install podman podman-compose.

One extra step users do need: they must add themselves to the video and render groups. The compatibility flag above works by removing the group IDs from the spec, so those permissions have to come from the host instead.

Container stopping challenge

Normally, when a container is running in the terminal and the user presses Ctrl+C, Docker stops it. Podman doesn’t. The application shuts down its launchers, but the container keeps running and serving requests.

The reason is that podman-compose 1.0.6 doesn’t exit on SIGINT, and the cleanup code in the script never gets a chance to run. To stop the stack, a separate podman-compose -p roboticsacademy down has to be run from the project root directory in a second terminal.

Work done this week

Apart from the documentation work, there are now three PRs open:

• Documentation: https://github.com/JdeRobot/RoboticsAcademy/pull/3961
• Scripts, adding the -p flag for Podman: https://github.com/JdeRobot/RoboticsAcademy/pull/3960
• Compose files for Podman: https://github.com/JdeRobot/RoboticsAcademy/pull/3959

Next week’s targets

1. Test the problems and workflow for Windows users

2. Change in PR #3961: move the Podman instructions into a separate file rather than adding them to InstructionsForDevelopers

3. Testing the podman documentation by setting everything up from scratch

Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.

I started by modifying dev-test-compose.podman.nvidia.yaml to translate Docker’s GPU passthrough mechanisms into Podman-compatible CDI mappings. While getting the stack running, I encountered a few blocking issues:

Network Namespace Corruption

When running the container attached (podman-compose up), stopping it via Ctrl+C often timed out and resulted in a forceful SIGKILL. This corrupted the networking namespace, causing aardvark-dns errors (“runs in a different netns”) on the next startup and breaking internal DNS resolution.

The workaround is to always stop the environment with podman-compose down in a separate new terminal after ctrl+C to stop the logs there. This fully wipes the container and its networking configuration, guaranteeing a clean slate.

The Infinite Buffering Bug

When running in detached mode (podman-compose up -d), refreshing the browser tab caused the Gazebo VNC viewer to get stuck on an infinite “Connecting…” spinner.

I traced this to a websocket disconnect triggered by the page refresh. An unhandled Django error in save_file interrupted the backend cleanup sequence in manager.py. Because cleanup aborted, the original Xvnc and websockify processes became zombies holding onto port 6080. Upon reconnecting, a new Gazebo simulation spawned on the next available port (6082). But since the React frontend (tools.tsx) has hardcoded ports, it attempted to connect to the leaked session on 6080, causing the buffering.

If a refresh happens, a full down and up cycle is needed to clear the leaked processes.

Camera “No image available”

The camera widget was reporting “No image available” (and HAL.getImage() returned None). I initially thought this might be a GPU passthrough issue, but it occurred on CPU as well.

The root cause was a code version mismatch. The local repo was updated to include namespace prefixes in ROS2 topics (e.g., /f1/camera/image_raw), but the Docker image was built before this rename. Since the compose files did not bind-mount the RoboticsInfrastructure submodule, the built-in Gazebo models were publishing to the old topic names while the local Python code listened to the new names.

GPU Validation Outcomes

Despite the issues, I proved GPU passthrough works under rootless Podman. Running vglrun glxinfo inside the container correctly reported the NVIDIA GTX 1650. This offloads the 3D rendering workload and reduces CPU overhead.

Here are recordings of the Follow Line exercise running with both configurations:

• GPU Accelerated Follow Line
• CPU Accelerated Follow Line

What’s Next

The next focus is solving the “Error saving file” bug from Week 8. I also need to find a solid workaround to stop Podman containers gracefully, similar to Docker, since the forceful terminations currently break the environment.

Docker also works via rootless, then why not use it?

According to official Docker docs, it also supports rootless mode, but it needs manual configuration by the end-user ,the user needs to execute the rootless setup script

# it creates a user systemd service and a user socket
dockerd-rootless-setuptool.sh install

Then enable and start the user Docker service.In short the user needs to do an additional one-time configuration step to switch from system-wide daemon to per-user daemon.The friction for this step is unnecessary and as told in last time it is not daemonless just like podman , it still works with daemon as a background service permanently.Switching to podman decreases the resource usage there is no background process unless needed. Podman natively has better security and also has Docker compatible CLI commands.Podman is better for RoboticsAcademy as it has true daemonless architecture and also has minimal setup for students/users.For Maintainers it will have fewer failures since the failures that included the daemon are no longer there.

How much difference would there be with Podman instead of Docker?

For most RoboticsAcademy users, very little changes. The existing workflows such as building images, running the exercises, using Compose, mounting volumes and exposing ports remain largely the same because both Docker and Podman implement the OCI container standards.

The differences become noticeable mainly in the following scenarios:

• No need for the docker group. With Docker, many users add themselves to the docker group to avoid using sudo, even though Docker’s documentation warns that this effectively grants root-equivalent access. Podman avoids this entirely by running containers as the current user.

• Better support for shared laboratory environments. Since Podman is rootless by default, multiple students can use containers on the same machine without requiring administrator privileges or sharing a system-wide daemon. This makes deployments on university lab machines considerably easier.

• Simpler recovery for beginners. A user’s containers and images remain isolated within their own home directory. If a student wants to reset their environment, they can remove only their own Podman storage without affecting other users or requiring privileged cleanup.

From a maintainer’s perspective, the migration mainly simplifies the container runtime itself rather than changing the development workflow. Existing commands and Compose files require minimal changes, while eliminating the daemon removes an entire component that previously had to be installed, managed and debugged.

Overall, the migration is less about introducing new functionality and more about providing a simpler and safer default environment without significantly changing how RoboticsAcademy is used. How much difference would be there with Podman instead of Docker?

Current Status of the Podman Migration testing :

Mentor’s ask was narrow: get the RoboticsAcademy image building and running under Podman, no GPU required for this pass, and call it a success if the image compiles and one exercise launches inside RA. Performance, the full exercise catalog, and GPU support were explicitly out of scope for this round.

Getting the Full Stack Up

A standalone container is one thing, a compose stack is another. Two things had to be sorted before docker-compose running against the Podman backend would even attempt to bring the stack up.

First, Podman doesn’t run a background daemon by default, so compose had nothing to talk to until I started the API socket: systemctl --user start podman.socket, enabled for persistence afterward.

Second, the compose file I was testing with requested an NVIDIA device reservation and /dev/dri access, which Podman couldn’t interpret and wasn’t needed for this pass anyway. Made a GPU-free copy, dev-test-compose.podman.yaml, for the test.

Third, Podman keeps its own image store, separate from Docker’s. The images built in last week’s Podman test weren’t visible to compose under the names it expected, jderobot/robotics-academy:test and jderobot/robotics-database:latest, until I retagged them.

With both sorted, the stack came up clean, database healthy, app Up.

Stale Checkouts Across Three Repos

Getting an exercise to actually load turned into the most confusing part of the week, and none of it was Podman’s fault .I just hadn’t pulled the latest commits across three different repos, all renamed the same day, July 7th, as part of one coordinated cleanup.

The frontend bundle was still calling get_universes_list, renamed to get_worlds_list, fixed with yarn install && yarn run build. The RoboticsInfrastructure submodule had renamed universes.sql to worlds.sql, breaking a bind mount, fixed with a one-line path change (the same stale reference is still in the tracked compose files under compose_cfg/ on humble-devel). And src/, a separate nested clone of RoboticsApplicationManager about 35 commits behind its upstream, had renamed world to scene in the frontend-backend message format, throwing KeyError: 'world', fixed with a git pull.

Once all three were current, follow_line launched end to end, noVNC came up, the simulation loaded, and running the exercise’s code got a response from the robot.

A Real Podman Difference: chmod Fails on Root-Owned Files

Autosave, the save that fires automatically when leaving an exercise’s editor, threw an “Error saving file:” popup with no message attached. This one was worth tracing properly rather than shrugging off.

Turned out the save itself worked. The file’s new content wrote to disk without a problem. What failed was the step right after it, an attempt to reset the file’s permissions back to fully open, and that failure is what surfaced as the error.

The reason traces back to how this particular file came to exist. It’s a per-student workspace file, filesystem/follow_line/academy.py, that RoboticsAcademy creates the first time an exercise gets opened, and this one had been sitting around since May, created weeks earlier while testing with plain rootful Docker, whose containers run as genuine root on the host. Rootless Podman doesn’t work that way by design. Its container’s “root” is really just the local user wearing a disguise, so it could still write new content into a file it doesn’t own, but it couldn’t change that file’s permission bits, since that specifically requires real ownership. The error code confirmed exactly this: EPERM, not the more familiar EACCES, the standard signature of an ownership mismatch rather than a plain access problem.

Nothing was lost. The edit itself saved, only the permission reset failed, and the popup came back blank because of a separate, preexisting bug where the frontend only reads error text from one field while this class of backend error sends it under another. Unrelated to Podman, and it would hide the same message under regular Docker too.

Result

The full stack ran under Podman, follow_line launched, and the simulation responded to code, which was the whole goal for this round. Everything that got there outside the codebase itself: two one-time host/Podman config changes, one scratch GPU-free compose file, three unrelated stale checkouts pulled up to date, and one genuine, reproducible Podman-vs-Docker behavior difference around rootless UID handling on legacy root-owned files.

What’s Next

Testing if the podman works with the GPU enabled which is the main pain-point of using podman Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.

Docker vs Podman GPU pass-through

As noted in the previous week’s blog, Podman is daemonless, so the GPU setup has to happen through CDI instead of a rootful runtime hook. With NVIDIA, CDI is what tells Podman which device to expose and which driver libraries to mount; without those libraries, the container would see the device but would not render correctly.

UID/GID and rootless

Since rootless Podman maps the container’s root back to my unprivileged host user, the process inside the container can still fail on a host device file if the UID or GID does not line up. The GPU render node on the host is owned by group render, so if the container does not get the right group access the GPU gets denied and quietly falls back to CPU rendering. keep-id keeps the user mapping aligned, and CDI injects the render/video group access automatically, so no extra group setup was needed. On my machine this currently works only because my desktop login grants temporary access to the GPU device; that will not exist on a server.

Problems and their solutions

The blocker: an outdated podman I forgot to update

This one was my own mistake, I hadn’t kept podman updated. When two tools that share a file drift apart in version, one writes something the other can’t read. Here nvidia-ctk (which generates the CDI spec) wrote a newer field, additionalGids, that my older podman’s parser doesn’t recognize. Instead of ignoring the unknown field, the parser rejects the entire spec (json: unknown field "additionalGids") → 0 devices found → no GPU. This happens before compose is even involved, so compose would hit the same wall.

It’s not only my mistake though. Ubuntu 24.04 ships podman 4.9.3 in its repositories and that’s the ceiling, so a plain apt upgrade won’t move past it. Any student following the default install path on Ubuntu 24.04 would hit this exact wall, which makes updating podman a real prerequisite for the project and not just something I overlooked.

The fix is to update Podman so its parser understands the newer spec. I also found that my own containers.conf was overriding Podman’s built-in CDI paths with a personal one, which made this work only because of a leftover debugging file from the version mismatch. Removing that override left GPU passthrough working the same way, but now on the paths a fresh install already has; ~/.config/cdi also held two conflicting spec files, which I cleared as part of the same fix.

Which compose engine to use

There are two “podman compose” tools and they don’t behave the same. podman compose (which delegates to docker-compose) mangles nvidia.com/gpu=all into a plain bind-mount and needs the podman socket, which is disabled in rootless. The Python podman-compose forwards --device nvidia.com/gpu=all to podman untouched, so that’s the engine I’ll use. I also hit a PATH issue inside podman-compose, where it was resolving the wrong helper binary at first, so the same stale-tooling pattern showed up twice before the stack would start. I also hit a PATH issue inside podman-compose, where it was resolving the wrong helper binary at first, so the same stale-tooling pattern showed up twice before the stack would start.

NVIDIA environment variables

NVIDIA_DRIVER_CAPABILITIES stays, it controls which graphics libraries get mounted. NVIDIA_VISIBLE_DEVICES is dropped as a GPU selector here: it belongs to the old runtime-hook mechanism, and in CDI mode the device is chosen by its CDI name instead.

Extra read — compose file duplication

Right now the compose_cfg contains 8 compose files which are mostly identical-every combination of {user, dev} × {cpu, gpu, nvidia, nvidia-windows}.One change needs to be replicated in all other files. I can put base code into one file and put the actual difference(GPU bits) into small override fragments I layer on top.base + nvidia-override, base + intel-override, base + nothing for CPU.

Conclusion

The path forward is CDI-based GPU passthrough run through Python podman-compose, with keep-id and keep-groups handling rootless identity.

Dependency Fixes from Mentor Review

Mentor review on the split: 11.9GB to 10.7GB didn’t justify the added complexity, and clean build time roughly doubled — 20 to 40 minutes — since deps-final keeps the full ROS toolchain, paying colcon’s cost twice. Also flagged: the narrow /usr/local COPY from the size-regression fix had silently broken OMPL’s Python bindings and pg_dump.

OMPL’s bindings install via install-ompl-ubuntu.sh --python into /usr/lib/python3/dist-packages/ompl/, not /usr/local, so that COPY missed them — fixed with one more explicit COPY at Dockerfile.dependencies_humble:598.

pg_dump was missing outright: deps-final only had libpq5, and the admin section’s save_exercise_db/save_universe_db views call it directly. Added postgresql-client-18 at line 463, pinned because the database container runs Postgres 18 and pg_dump won’t dump a newer server than itself.

That surfaced a second bug: deps-final had PGDG’s source copied in before ca-certificates existed, so the HTTPS fetch to apt.postgresql.org failed. Fixed with a ca-certificates-only install at line 414, ahead of the PGDG COPY, matching deps-builder’s order.

Verified both directly: imported OMPL’s bindings, ran pg_dump against a live Postgres 18 container. Net size: 9.55GB, up slightly from 9.49GB before these fixes.

First Podman Build

Set up a podman-compat-test branch off humble-devel with the original Dockerfiles, kept separate from the refactor so results don’t mix. First failure: Dockerfile.dependencies_humble’s FROM line — nvidia/opengl:1.2-glvnd-runtime-ubuntu22.04 is an unqualified image name, and Podman won’t assume Docker Hub for those the way Docker does.

There were two ways to fix this. One was fully qualifying the reference in the Dockerfile itself, docker.io/nvidia/opengl:1.2-glvnd-runtime-ubuntu22.04 — no config changes, but it means editing the file, and the point of this branch was testing the original Dockerfiles unmodified. The other was configuring a default search registry on the Podman side, which doesn’t touch the Dockerfile at all: adding unqualified-search-registries = ["docker.io"] to ~/.config/containers/registries.conf. Rootless Podman reads that file before the system-wide /etc/containers/registries.conf, so it doesn’t need sudo either. I went with the registry config for this test, since keeping the Dockerfile untouched was the whole point. Worth noting for later, though, fully qualifying the image reference is probably the better fix for the actual codebase, since it means anyone building this under Podman doesn’t need to know about registry config at all. That’s a small, separate change I’d want to raise once this testing phase moves further.

After that, both Dockerfiles built clean under Podman unmodified — 26 workspace packages compiled. Running it standalone, GPU detection found nothing and fell back to CPU-only without crashing (no GPU needed for this pass), and Django plus the RAM manager’s websocket server both started. The only error was Django failing to reach a nonexistent database host — expected in a standalone run, same as plain Docker with no DB present.

Next is the full compose stack under Podman. The compose file requests GPU reservations Podman can’t interpret, so I’m preparing a GPU-free variant — then the goal is launching one exercise inside RA to confirm end-to-end compatibility.

Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.

I worked in a .multistage copy of the file the entire time, leaving Dockerfile.dependencies_humble itself untouched until the split was fully verified — safer to break a throwaway file fifty times over than the one file everyone else’s builds actually depend on.

Skeleton First

First pass was just the skeleton. deps-final did a wholesale COPY --from=deps-builder / /, and I went through every ENV, WORKDIR, and ARG by hand to make sure each one carried across the stage boundary correctly. It did. Then I built Dockerfile.humble unedited on top of that image and ran it through the full smoke test — Django, the RAM websocket, noVNC once an exercise launches, an ldd sweep across the compiled workspace libraries. Everything came back clean, so the boundary itself wasn’t the problem, which was a good place to start before touching anything else.

The Toolchain Question

Dockerfile.humble runs its own colcon build on top of this image to compile CustomRobots, jderobot_drones, Industrial, and common_interfaces_cpp, which get moved into /home/ws/src/ a few lines earlier. That one call site means deps-final can’t be a stripped-down runtime image — it has to keep the full ROS build toolchain. I ran that by the mentor before dropping anything, since getting it wrong would mean a broken downstream build with a confusing error months from now.

So the trimming ended up narrower than I’d planned. Tracing through entrypoint.sh, manager.py, and the launcher scripts for anything that actually runs at boot or exercise-launch time turned up tmux and tmuxinator with zero invocations anywhere, just a leftover bash-completion stub. postgresql-18 became libpq5 once I confirmed the Postgres server binary itself never starts inside this image — only psycopg2 needs the client library to link against.

The harder calls needed more than a grep. I ran ldd across every compiled binary in the image, matched each shared library back to its apt package with dpkg -S, and checked every pip package against real import statements in the codebase. Two of those checks went against what I expected going in. onnxruntime-gpu and its CUDA wheels showed no ldd or import evidence anywhere, but three exercise templates compile against onnxruntime’s C++ API at load time — nothing static analysis would ever catch, so it stays. And pylint, black, and jedi, which I’d have called dev tooling on sight, turned out to be declared runtime dependencies of the RAM’s web IDE in its pyproject.toml.

The Regression

The first full deps-final build came out bigger than the original single-stage image, which is the opposite of the entire point. docker history and dive’s wasted-space report pointed at a blanket COPY --from=deps-builder /usr/local/ /usr/local/ — it was dragging across roughly 3.5GB that five separate pip RUN blocks in that same stage were already reinstalling from scratch. I swapped it for a handful of explicit COPY lines that only pull the non-pip content actually living under /usr/local: OMPL and onnxruntime’s compiled C++ libraries, about 40MB total.

A second, unrelated break showed up downstream. Folding lxde-common into a consolidated --no-install-recommends apt block silently dropped the two packages it normally pulls in with it, lxde-core and openbox-lxde-session — and Dockerfile.humble needs those to place rc.xml into a directory only they create. Putting lxde-common back on its own unflagged line, the way the original file already had it, fixed the build.

Where This Landed

I rebuilt both versions the same day for a fair comparison: the original single-stage image is 11.9GB, deps-final comes in at 10.7GB. Checked that with docker history, another ldd sweep across both workspaces, and a full pass through docker compose — Django routes, DB-backed queries, the RAM websocket handshake, clean shutdown. The split holds together and it’s smaller, which was the actual goal for the week.

Still Open

Loading an exercise through docker compose throws relation "exercises" does not exist, even though Postgres itself reports healthy. I don’t yet know if that’s a stale data volume left over from earlier test runs or something in how the init SQL files get loaded — that’s what I’m chasing next.

Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.

The NVIDIA Fix (PR #3890)

Dockerfile.dependencies_humble had this near the top:

ARG NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=all

ARG variables exist only during the build — not written into any image layer, so the running container never sees them. NVIDIA_VISIBLE_DEVICES controls which GPUs the NVIDIA container runtime exposes. Setting it as ARG meant GPU visibility was silently broken at runtime. No error, no warning, just no GPUs. Changed to ENV.

The second fix: Dockerfile.humble had a standalone source call in its own RUN instruction before the colcon build. Docker starts a new shell for each RUN, so the sourced environment was gone before the next line ran. Dead line. Removed it.

Both fixes are one PR — small, isolated, and reviewable without touching the refactor.

The Refactor Plan

Before touching any Dockerfile I wrote out the full multi-stage split as a design document. At this scale — two images, each with its own builder/runtime boundary, one FROM the other — getting the COPY boundaries wrong in code means broken container runs that are slow to diagnose. The structure: Dockerfile.dependencies_humble gets a builder that runs the full Aerostack2 build and a slim runtime that receives only the compiled artifacts. Dockerfile.humble chains off that slim runtime, adds its own builder to compile /home/ws/, and produces the shipped image.

The key finding that changed what gets COPY --from‘d came from the --symlink-install verification done in Week 4. The assumption was that it writes .pth files for Python packages, meaning only a few small src/ directories would need to travel with install/. Running find inside both images returned zero .pth files. What --symlink-install actually produces is filesystem symlinks — 3,206 from /home/ws/install/ back into /home/ws/src/, and 347 from /home/drones_ws/install/ back into /home/drones_ws/src/. Copying install/ alone into the runtime stage leaves all of those dangling. The minimal subset of src/ the symlink graph requires has to be traced from the links — that question goes to the mentor before implementation starts.

The 83 NEEDS_CLARIFICATION packages from the Week 4 audit are kept as-is in the plan. Their classification needs confirmation before they can be moved or dropped.

The implementation is split into sequential PRs — one per Dockerfile — so each can be built and tested before the next opens.

Next week is implementation, starting with the builder/runtime stage skeleton for Dockerfile.dependencies_humble.

Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.

The NVIDIA ARG That Does Nothing at Runtime

Near the top of Dockerfile.dependencies_humble, lines 4 and 5:

# Make all NVIDIA GPUS visible
ARG NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=all

NVIDIA_DRIVER_CAPABILITIES is ENV so it persists. NVIDIA_VISIBLE_DEVICES is ARG — gone after the build, not present in the running container. Passing --gpus all at runtime compensates today via automatic toolkit injection, but the correct fix is ENV. It goes in the runtime stage.

–symlink-install: The Constraint and What It Costs

The mentor confirmed --symlink-install has to stay, so the question became what it actually produces and what that means for copying artifacts into a runtime stage. The assumption going into verification was that it creates .pth editable installs for Python packages, meaning only a handful of small src/ directories would need to travel with install/.

That assumption was wrong. Running find inside both images found zero .pth files. What --symlink-install actually creates is filesystem symlinks — 3,206 of them from /home/ws/install/ back into /home/ws/src/ across 21 packages, and 347 from /home/drones_ws/install/ back into /home/drones_ws/src/. Copying install/ alone into a runtime stage leaves all of those dangling. The “copy 268K of Python source” framing from the earlier audit was incorrect and is withdrawn.

What this means for the split is that src/ cannot be excluded wholesale. The question is which subset of src/ the symlink graph actually requires at runtime — not all of /home/ws/src/ (2.5G) needs to ship, but the exact minimal set has to be traced from the symlinks rather than assumed. That tracing is the open question going to Slack before implementation starts.

Two things the verification did close: AEROSTACK2_PATH appears only in the Dockerfile’s export line and has no runtime reader — so /home/drones_ws/src/aerostack2 is not needed for that reason. The MoveIt and OMPL packages used by Industrial exercises are present in the inherited ROS install — no special COPY and no patched headers needed.

One correction from the package audit stands: colcon must be in the runtime stage because manager.py line 403 runs colcon build --symlink-install at runtime when loading a custom universe. Running objdump -p on libdrone_lib_cpp.so confirmed no RPATH is baked in — resolution is through LD_LIBRARY_PATH from sourcing /home/drones_ws/install/setup.bash, so that directory must be present in the runtime stage.

The Dead Source Line and Why It Doesn’t Break Anything

Dockerfile.humble lines 39-41 have this pattern:

WORKDIR /home/ws
RUN /bin/bash -c "source /home/drones_ws/install/setup.bash"
RUN /bin/bash -c "source /opt/ros/humble/setup.bash; colcon build --symlink-install ..."

Docker starts a new shell for each RUN instruction. The shell on line 40 exits when it finishes, so the sourced environment is gone before line 41 starts. The source call on line 40 does nothing useful. I flagged this on Slack before touching the file.

Someone had already fixed this in git by collapsing the source and the build into a single RUN. It was reverted for process reasons — the PR was scoped to something unrelated, mentor asked for a clean revert — not because the fix was wrong.

What I didn’t understand until the mentor explained it is why line 40 doesn’t cause a build failure. Dockerfile.dependencies_humble at lines 382 and 385 appends both source commands into ~/.bashrc, and Dockerfile.humble at line 65 sets ENV BASH_ENV=/.env. BASH_ENV tells bash to source that file at startup for every non-interactive shell — including every RUN /bin/bash -c instruction — so drones_ws is already in the environment before line 40 even runs. Line 40 is sourcing something that’s already there. The source instructions in Dockerfile.humble are not needed because Dockerfile.dependencies_humble already handles them through BASH_ENV.

What the Runtime Stage Actually Needs

A package is build-time only if nothing running inside the container after the entrypoint fires ever calls, imports, or loads it — the criterion that drove every classification below.

To figure out which packages actually need to ship in the runtime image, I traced every apt package in Dockerfile.dependencies_humble through the files that run at container start: entrypoint.sh, manager.py, set_dri_name.sh, check_ram_version.sh, start_vnc.sh, and all the launcher files. The goal was to classify each package as build-time or runtime, with no guessing. The build-time list is long and expected: build-essential, cmake, the full colcon and rosdep stack, all the ament-cmake lint and test packages, flake8 and its plugins, and the APT infrastructure tools — curl, wget, gnupg, lsb-release, git. None of those ship in the runtime stage.

Three packages looked like dev tools but turned out to be runtime. manager.py imports black at line 14 and calls black.format_str() to format code that users submit through the web editor. At line 620 it calls jedi.Script() to power live autocomplete in that same editor. At line 518 it runs pylint via subprocess to do live code analysis on exercise submissions. All three run on every exercise session, so pylint, black, and jedi go in the runtime stage.

Two packages that look like runtime dependencies turned out to be build-time only. python3-pip is the bootstrap — the pip binary available at runtime is /usr/local/bin/pip, pip 23.3.1, installed from the wheel upgrade done during the build. I confirmed this by running which pip inside the container. The apt package was only needed to get pip installed in the first place. python-is-python3 is also build-time only. The only file in the repo with a bare #!/usr/bin/env python shebang is build_world.py in tello_phy, which has no external callers anywhere in the codebase and no registered universe in universes.sql.

The shared library situation needed a different tool. I ran dpkg -S inside the dependencies-humble container against libGeographic.so, libPocoFoundation.so, libpcl_common.so, libncurses.so, and libyaml-cpp.so and got the same pattern every time. The unversioned .so symlink belongs to the -dev package — that’s what the linker follows at compile time. That job is done once the binary is compiled and never repeated — at runtime the dynamic linker loads the versioned .so from the separate runtime package, not the -dev package. The versioned .so belongs to a separate runtime package and is what the dynamic linker actually loads when the container runs. The runtime stage therefore needs libgeographic19, libpocofoundation80, libpcl-common1.12, libncurses6, and libyaml-cpp0.7, not the -dev packages.

What’s Next

One question goes to Slack before the Dockerfile is written: what is the minimal subset of src/ the symlink graph requires at runtime, and where does gz_ros2_control fit in the two-Dockerfile split. Those two answers determine the COPY instructions for the runtime stage. Everything else the audit needed to know is verified.

Part of my GSoC 2026 work with JdeRobot. Project tracked at github.com/TheRoboticsClub/gsoc2026-Kartik_Jangid.