“

On the evening of September 16, 2026, after six months of running all sorts of self-hosted apps, my Tencent Cloud Lighthouse server finally seized up completely on an otherwise ordinary night. This post is a complete record of the whole process: tracking down the cause of the crash, discovering signs of an intrusion, removing the backdoors, and rebuilding with better hardening. If you're also running a pile of containers on a small-memory VPS, I hope this post-mortem helps you out.

Background

This server's specs were nothing to write home about:

• 2-core CPU / 3.6GB RAM, 4GB swap, 50GB disk
• Ubuntu 24.04, Tencent Cloud Lighthouse
• Running on top: the whole Coolify stack (about 20 Docker containers) — Vaultwarden, Memos, OpenGist, Uptime Kuma, Chronoframe, Artalk, ClickHouse, OpenList, and so on
• I'd also previously installed the MCSManager panel to manage a Minecraft server

The symptoms that night were textbook: SSH logins were agonizingly slow, everything lagged, and eventually the load average spiked to 353 (a catastrophic number for a 2-core box), iowait hit 31%, swap was completely full, and the machine was basically catatonic. The only option was a forced reboot from the Tencent Cloud console.

Step 1: Pinpointing the cause of the crash

Once I finally managed to squeeze in over SSH, I started with sar (historical data recorded by sysstat) to trace the load curve during the crash window, then used journalctl and dmesg to check the OOM logs, and the truth surfaced fast:

1. The chronoframe container was the main culprit. This photo-wall app (Node.js) peaked at ~2.4GB RSS, with no memory cap at all on a 3.6GB machine
2. It had already been killed by the OOM killer many times before the crash (exit 137 — on 9/14 and again on 9/16, the days of both crashes)
3. At 20:59 on the day of the crash, chronoframe triggered OOM again, swap filled up completely, and the load avalanched to 353
4. On top of that, Coolify's bundled Horizon was perpetually eating about 93% of a single core, making already-tight resources even worse

Conclusion: a 3.6GB machine running 20 containers with no memory limits was guaranteed to hit OOM — it was only a matter of time.

But the real “surprise” was still waiting further down the line.

Step 2: An unexpected discovery — a crypto-mining trojan

While working through the system state, I found a few things in the systemd unit list that had no business being there:

xmrig.service had already logged 108,000 restarts — meaning it had been failing and retrying every 10 seconds for 12 days straight. Since the binary had long since been deleted (later confirmed: Tencent Cloud's YunJing quarantined it shortly after installation), systemd could do nothing but spin in place forever.

Reconstructing the attack chain

Drawing on /var/log/auth.log and journalctl records, the full attack chain looks like this:

1. 2026-08-16 22:26 — the attacker (IP 23.156.153.36) registered an account with the username 123456 via MCSManager
2. 23:13 — exploiting an authentication middleware bypass in the MCSManager v10.4.0 – v10.16.2 API (which lets you call every panel endpoint without authorization) to escalate 123456 to an administrator with permission level 10
3. 23:15 — used the panel's built-in terminal to get a shell on the instance, then escalated to root
4. 23:16 — downloaded XMRig 6.21.3 to /123/xmrig-6.21.3, installed xmrig.service, configured the mining pool auto.c3pool.org:23333, and injected a Monero mining address
5. That night CPU load spiked to 68% and XMRig ran successfully for a while, until YunJing quarantined it

Afterward I went to confirm the vulnerability — MCSManager had already published an advisory: every version up to and including v10.16.2 was affected, and my panel was running v10.16.2, the very last version before the patch.

Earlier traces

SSH was in bad shape too: root password login was exposed to the public internet, drawing 12,000+ brute-force attempts a week. During the investigation I personally watched two unfamiliar IPs probing repeatedly at the preauth stage.

Step 3: Cleanup

Once I'd confirmed the intrusion, I did the cleanup in three rounds:

1. Close the entry point

systemctl stop mcsm-daemon.service mcsm-web.service
systemctl disable mcsm-daemon.service mcsm-web.service
rm -rf /opt/mcsmanager
# 删除 unit 文件，daemon-reload

Ports 23333/23334 were cleared, with no leftover processes.

2. Stop all containers

Ran docker stop on all 20 containers. This also relieved the memory pressure as a side effect — swap usage dropped to zero, freeing up safe headroom for the work ahead.

3. Remove malicious persistence

One small hiccup: tianji-reporter.service had a second copy in /usr/lib/systemd/system/ that the first cleanup pass missed; systemctl list-unit-files turned it up again, and only after deleting that copy was everything truly clean. The xmrig binary in YunJing's quarantine was preserved at /usr/local/qcloud/YunJing/quara/ as forensic evidence.

Step 4: A thorough backdoor sweep

With the known backdoors gone, I still had to make sure the attacker hadn't left any other “gifts” behind. I went through it item by item:

For the batch of binaries changed in system directories between July 28 and early September (systemd, curl, openssl, dockerd, etc.), I cross-checked the apt history — all were normal unattended-upgrades updates. Conclusion: apart from the three known spots, I found no other backdoors.

Step 5: Hardening and rebuilding

Once the sweep was done, I did three things:

fail2ban

# /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true
maxretry = 5
findtime = 10m
bantime = 1h
bantime.increment = true
bantime.maxtime = 48h

It took effect the moment it was installed — 16 brute-force failures logged within 10 minutes. This machine gets attacked every single day.

Swap 4G → 8G

swapoff /swap.img
dd if=/dev/zero of=/swap.img bs=1M count=8192
chmod 600 /swap.img && mkswap /swap.img && swapon /swap.img

I did this with all containers stopped and the memory pressure relieved, so it was perfectly safe.

Restore containers + memory limits

Ran docker start on all containers. Along the way I hit one snag: the clickhouse container failed to start, reporting no such volume — the crash had wiped out Docker's volume metadata, but the data directory under /var/lib/docker/volumes/ was still there.docker volume create After recreating a volume with the same name, the data was intact and the container started up normally.

Finally, I gave the OOM culprit chronoframe a memory cap:

docker update --memory 1536m --memory-swap 2g chronoframe-xxx

Note that docker update may get overwritten when Coolify redeploys, so you need to set it in the app config as well.

Post-mortem summary

Afterword

I never imagined MCSManager would turn out to have a vulnerability this big.

What surprised me even more is how little information about this vulnerability is out there! I only found it by searching keywords after GLM-5.3-Flash mentioned MCSManager's compromise path.

The official handling was even more absurd — they just posted a Release: https://github.com/MCSManager/MCSManager/releases/tag/v10.17.0, didn't file it under Security advisories, assigned no CVE, and didn't wait long enough before disclosing the flaw (though I doubt many people are in the habit of updating anyway). Either way, that approach is a bit off. Like I said: just set up your server by hand from the command line (

Root causes of this incident

1. A blind spot in security awareness: MCSManager is a panel exposed to the public internet, and I neither followed its security advisories nor kept it updated — the advisory was right there, I just never looked
2. Overselling resources: running 20 unlimited containers on 3.6GB of RAM made OOM a matter of time
3. SSH left wide open: root + password login + public exposure = brute-forced daily

Lessons I learned

1. Treat every web service exposed to the internet as if it could be breached at any moment. Subscribe to security advisories for the software you use, or at least check versions regularly
2. On a low-memory machine, always set a memory limit on every container. A single runaway container can drag the whole machine down, and when OOM strikes the kernel kills processes at random — it might kill your most important one
3. fail2ban is the bare minimum for SSH, and it costs almost nothing
4. systemd units are a backdoor hotspot. When investigating an intrusion, systemctl list-unit-files combined with file creation times is the highest-value first step
5. Logs get rotated away and lost. Back up logs from the critical window immediately while investigating; journalctl -b -1 can save you in a pinch, but it isn't a cure-all

Still to do (next steps)

• Switch SSH to key-only login and change the root password
• Add firewall allowlists for 1Panel (port 5227) and the Docker swarm ports (2377/7946)
• Add memory limit settings for all long-running apps in Coolify

A 2-core, 3.6GB machine crammed with self-hosted services, plus an exposed panel and wide-open SSH — that combination was bound to blow up sooner or later. I hope this post-mortem helps readers avoid a few of these pitfalls.

(The IPs, mining pool addresses, and other details in this article are included only to reconstruct the incident. Please do not use them for any illegal purpose.)