My production Postgres container kept running on an orphaned Portainer volume after an upgrade, and nothing warned me

Upgrading a container like Portainer is normally one of the least stressful parts of running a homelab. I have done it many times without a second thought. The last upgrade I did also felt normal because everything was working exactly the way it was supposed to. Weeks later, I was pulling compose files from my homelab for an article I was working on. But the compose files weren’t where I expected them to be. Those missing files made me investigate, and what I found underneath was a mess.

I went looking for compose files, and half my stacks were gone

The missing file didn't make sense

One fine morning, I was setting up an experiment to gather all the compose files of my homelab and feed them to a local LLM for analysis. It was all set. Since I'd been using Portainer from day one, I assumed all my stack compose files would be neatly organized in separate directories inside a single Portainer volume. I wasn’t auditing Portainer or troubleshooting Docker. I was just trying to pull the Docker Compose files for the test.

I started with docker inspect to get the exact location of the compose files, followed by sudo find to locate all the docker-compose.yml files. The second command’s result confused me because it only listed nine out of twenty stacks. But I wasn't alarmed at that point since all my services were up and running properly.

After an hour of SSH sessions, I ran docker volume ls, almost as an afterthought. I was expecting one Portainer volume, but I got two. I still wasn’t surprised, though. I thought it was an old leftover. I was almost ready to move on with those nine compose files and return later to look for other files. But instinctively, I decided to open the old Portainer volume and see what was there. That was my first genuine surprise.

Both volumes contained Portainer data, not duplicates or leftovers. The stacks weren’t identical. Some compose files only existed in one volume and others in the second. When I counted them briefly, the combined total was close to my number of deployed stacks. At first glance, it wasn't possible to tell which directory belonged to which stack, since Portainer only numbers them instead of using stack names. Finding two volumes with two different directories made me realize I first needed to figure out what had been happening behind the scenes.

The hidden split

One upgrade, two volumes, and a number that meant two different things

My first assumption was that one was an abandoned leftover from an old deployment, since you don't normally end up with two Portainer volumes. My leftover theory was disproved when I checked the Compose directories of both the volumes. I compared their timestamps, labels, and mount points with docker volume inspect, then examined the files and Portainer databases stored inside each volume.

You know what the best thing about Portainer was when I first picked it? I knew that Portainer stored almost everything — stacks, settings, notifications — inside a Docker volume, not the container itself. So, even if something happens with the Portainer container, my Portainer data would survive replacing the container. That was what happened, but in a very strange way.

I still don’t know the exact trigger, but the evidence suggests that one of my Portainer redeployments resolved the data-volume declaration differently from the original installation. A changed Compose project name, volume name, or deployment method may have caused Compose to create a second volume instead of mounting the existing one. The old volume’s missing Compose project label supported that theory, but it wasn’t enough to prove exactly when or why the split happened.

Nothing forced me to investigate because all the services were working perfectly. The Portainer UI worked as usual; I could deploy new stacks without any issues. And the containers didn’t care which unused volume was sitting on disk.

But on that day, while comparing the stack directories, I noticed something strange. The same Stack 3 was on both volumes, and when I looked carefully, both turned out to be different services. The reused number wasn’t the problem itself; Portainer didn’t know the same number already existed on the old volume. The real problem wasn’t that Docker created another volume — it was that the two volumes weren't copies of the same history. They were two independent timelines that happened to collide.

The scarier part was still waiting to reveal itself. I still hadn't realized that one of my production services was quietly connected to the old one.

The stack Portainer forgot it was managing

Healthy on the dashboard. Invisible everywhere else.

While I was chasing down the orphaned volume and compose directories, one stack immediately stood out. It wasn’t a test deployment that I had disposed of at some point; it was Immich storing my family’s photos and videos. Investigating it further was my only option because the stack mattered to me.

If you are running Immich on your homelab, you already know Immich depends on multiple containers to work properly, including Redis and PostgreSQL. When I cross-checked the compose files against the running containers, I found out that most of the Immich containers pointed to the current setup, except one — the Postgres container. I checked each container's com.docker.compose.project.config_files label using docker ps. Three of the four — the server, machine learning, and Redis containers — pointed to /data/compose/3/docker-compose.yml, the current volume. The Postgres container's label pointed to /var/lib/docker/volumes/portainer_data/_data/compose/3/docker-compose.yml instead, the orphaned volume I'd found earlier.

I have a hunch about when it happened: a few days ago, WatchTower auto-updated Immich to v3.0.1, but the Postgres image still needed a manual migration step. That was when I manually resurrected it from the old file, and it's been running that way since, healthy, undetected. Here’s what I’d check first if I suspect the same thing again: I would start with docker volume ls to check the volumes, followed by docker volume inspect to cross-check the timestamps, and then finally verify it against the labels in docker ps.

Once this evidence surfaced, I opened Portainer, and this time I noticed something I had been unintentionally ignoring for months. The Immich stack, along with many others, carried a limited control badge. It was Portainer’s own admission that it had detected the containers but no longer owned the original definitions. This made one thing clear: from that event onwards, even though the dashboard showed everything healthy, edits, redeployments, restores, and rollbacks would only affect what Portainer thought it managed.

Even though the Immich stack was working fine because the actual photo library lived on my NAS and my server only handled the compute, it was exactly the kind of silent drift that was difficult to spot until something actually broke. Nothing about this looked wrong from the dashboard because it didn’t give me any reason to investigate. That's exactly what made it dangerous.

Nothing was broken, which was more concerning

The scariest part of all these shenanigans was that everything looked fine: no failed containers, all healthy services, and the dashboard looked exactly the same. This wasn’t even a Portainer problem — it’s what happens when disposable containers and persistent data are treated the same way. The lesson I learned from this is that persistent data deserves far more attention than the containers themselves.

So, after every major upgrade — Portainer or otherwise — every service gets a quick volume audit because I no longer trust the “it looks fine” signal.