A Full Post-Mortem of a Server Crash: From OOM to a Crypto-Mining Trojan Attack Chain
·(edited)· / , , , 0AI-written
This article was generated entirely by AI without human authorship. Please read with discretion, or have an AI verify its accuracy.
AI TranslationSimplified ChineseEnglish
Key Insights
On September 16, 2026, a 2-core, 3.6GB server was brought down by 20 containers running without memory limits — chronoframe being the main offender — which triggered an OOM and sent load soaring to 353. While digging into it, the author stumbled on an XMRig mining trojan: the attacker had abused an API authentication bypass in MCSManager v10.16.2 to escalate to root, and the miner had been running for 12 days. After clearing out the backdoor and persistence, they checked accounts, cron, systemd, SUID files, and Docker images, confirmed nothing else was amiss, then rebuilt with hardening: fail2ban deployed, swap expanded to 8G, and memory caps set on containers. The lessons: stay on top of security advisories for anything public-facing, always cap container memory on small machines, and disable password login over SSH.
“
On the evening of September 16, 2026, after six months of running all sorts of self-hosted apps, my Tencent Cloud Lighthouse server finally seized up completely on an otherwise ordinary night. This post is a complete record of the whole process: tracking down the cause of the crash, discovering signs of an intrusion, removing the backdoors, and rebuilding with better hardening. If you're also running a pile of containers on a small-memory VPS, I hope this post-mortem helps you out.
Background
This server's specs were nothing to write home about:
2-core CPU / 3.6GB RAM, 4GB swap, 50GB disk
Ubuntu 24.04, Tencent Cloud Lighthouse
Running on top: the whole Coolify stack (about 20 Docker containers) — Vaultwarden, Memos, OpenGist, Uptime Kuma, Chronoframe, Artalk, ClickHouse, OpenList, and so on
I'd also previously installed the MCSManager panel to manage a Minecraft server
The symptoms that night were textbook: SSH logins were agonizingly slow, everything lagged, and eventually the load average spiked to 353 (a catastrophic number for a 2-core box), iowait hit 31%, swap was completely full, and the machine was basically catatonic. The only option was a forced reboot from the Tencent Cloud console.
Step 1: Pinpointing the cause of the crash
Once I finally managed to squeeze in over SSH, I started with sar (historical data recorded by sysstat) to trace the load curve during the crash window, then used journalctl and dmesg to check the OOM logs, and the truth surfaced fast:
The chronoframe container was the main culprit. This photo-wall app (Node.js) peaked at ~2.4GB RSS, with no memory cap at all on a 3.6GB machine
It had already been killed by the OOM killer many times before the crash (exit 137 — on 9/14 and again on 9/16, the days of both crashes)
At 20:59 on the day of the crash, chronoframe triggered OOM again, swap filled up completely, and the load avalanched to 353
On top of that, Coolify's bundled Horizon was perpetually eating about 93% of a single core, making already-tight resources even worse
Conclusion: a 3.6GB machine running 20 containers with no memory limits was guaranteed to hit OOM — it was only a matter of time.
But the real “surprise” was still waiting further down the line.
Step 2: An unexpected discovery — a crypto-mining trojan
While working through the system state, I found a few things in the systemd unit list that had no business being there:
xmrig.service had already logged 108,000 restarts — meaning it had been failing and retrying every 10 seconds for 12 days straight. Since the binary had long since been deleted (later confirmed: Tencent Cloud's YunJing quarantined it shortly after installation), systemd could do nothing but spin in place forever.
Reconstructing the attack chain
Drawing on /var/log/auth.log and journalctl records, the full attack chain looks like this:
2026-08-16 22:26 — the attacker (IP 23.156.153.36) registered an account with the username 123456 via MCSManager
23:13 — exploiting an authentication middleware bypass in the MCSManager v10.4.0 – v10.16.2 API (which lets you call every panel endpoint without authorization) to escalate 123456 to an administrator with permission level 10
23:15 — used the panel's built-in terminal to get a shell on the instance, then escalated to root
23:16 — downloaded XMRig 6.21.3 to /123/xmrig-6.21.3, installed xmrig.service, configured the mining pool auto.c3pool.org:23333, and injected a Monero mining address
That night CPU load spiked to 68% and XMRig ran successfully for a while, until YunJing quarantined it
Afterward I went to confirm the vulnerability — MCSManager had already published an advisory: every version up to and including v10.16.2 was affected, and my panel was running v10.16.2, the very last version before the patch.
Earlier traces
SSH was in bad shape too: root password login was exposed to the public internet, drawing 12,000+ brute-force attempts a week. During the investigation I personally watched two unfamiliar IPs probing repeatedly at the preauth stage.
Step 3: Cleanup
Once I'd confirmed the intrusion, I did the cleanup in three rounds:
Ports 23333/23334 were cleared, with no leftover processes.
2. Stop all containers
Ran docker stop on all 20 containers. This also relieved the memory pressure as a side effect — swap usage dropped to zero, freeing up safe headroom for the work ahead.
One small hiccup: tianji-reporter.service had a second copy in /usr/lib/systemd/system/ that the first cleanup pass missed; systemctl list-unit-files turned it up again, and only after deleting that copy was everything truly clean. The xmrig binary in YunJing's quarantine was preserved at /usr/local/qcloud/YunJing/quara/ as forensic evidence.
Step 4: A thorough backdoor sweep
With the known backdoors gone, I still had to make sure the attacker hadn't left any other “gifts” behind. I went through it item by item:
What I checked
Result
Accounts / sudo
✅ Only root/ubuntu/lighthouse, no stray uid 0 accounts; passwd/shadow/sudoers untouched since June
cron
✅ Only the Tencent Cloud agent and my own scripts
systemd units
✅ No custom units beyond the 3 already removed
authorized_keys
✅ Only coolify's key and my own; no attacker public key
SUID files
✅ Nothing unusual (the ones inside container snapshots are normal)
deleted-but-running binaries
✅ None
/etc/ld.so.preload
✅ Empty
/etc/hosts, LD_PRELOAD, profile.d, rc.local
✅ Clean
Executables in /tmp /var/tmp /dev/shm
✅ None
dpkg integrity
✅ Only a sudoers md5 change (my own NOPASSWD setup)
Docker images
✅ No mining images
Recent files in /root
✅ All from my own 1Panel install and ops scripts
For the batch of binaries changed in system directories between July 28 and early September (systemd, curl, openssl, dockerd, etc.), I cross-checked the apt history — all were normal unattended-upgrades updates. Conclusion: apart from the three known spots, I found no other backdoors.
I did this with all containers stopped and the memory pressure relieved, so it was perfectly safe.
Restore containers + memory limits
Ran docker start on all containers. Along the way I hit one snag: the clickhouse container failed to start, reporting no such volume — the crash had wiped out Docker's volume metadata, but the data directory under /var/lib/docker/volumes/ was still there.docker volume create After recreating a volume with the same name, the data was intact and the container started up normally.
Finally, I gave the OOM culprit chronoframe a memory cap:
Note that docker update may get overwritten when Coolify redeploys, so you need to set it in the app config as well.
Post-mortem summary
Afterword
I never imagined MCSManager would turn out to have a vulnerability this big.
What surprised me even more is how little information about this vulnerability is out there! I only found it by searching keywords after GLM-5.3-Flash mentioned MCSManager's compromise path.
The official handling was even more absurd — they just posted a Release: https://github.com/MCSManager/MCSManager/releases/tag/v10.17.0, didn't file it under Security advisories, assigned no CVE, and didn't wait long enough before disclosing the flaw (though I doubt many people are in the habit of updating anyway). Either way, that approach is a bit off. Like I said: just set up your server by hand from the command line (
Root causes of this incident
A blind spot in security awareness: MCSManager is a panel exposed to the public internet, and I neither followed its security advisories nor kept it updated — the advisory was right there, I just never looked
Overselling resources: running 20 unlimited containers on 3.6GB of RAM made OOM a matter of time
SSH left wide open: root + password login + public exposure = brute-forced daily
Lessons I learned
Treat every web service exposed to the internet as if it could be breached at any moment. Subscribe to security advisories for the software you use, or at least check versions regularly
On a low-memory machine, always set a memory limit on every container. A single runaway container can drag the whole machine down, and when OOM strikes the kernel kills processes at random — it might kill your most important one
fail2ban is the bare minimum for SSH, and it costs almost nothing
systemd units are a backdoor hotspot. When investigating an intrusion, systemctl list-unit-files combined with file creation times is the highest-value first step
Logs get rotated away and lost. Back up logs from the critical window immediately while investigating; journalctl -b -1 can save you in a pinch, but it isn't a cure-all
Still to do (next steps)
Switch SSH to key-only login and change the root password
Add firewall allowlists for 1Panel (port 5227) and the Docker swarm ports (2377/7946)
Add memory limit settings for all long-running apps in Coolify
A 2-core, 3.6GB machine crammed with self-hosted services, plus an exposed panel and wide-open SSH — that combination was bound to blow up sooner or later. I hope this post-mortem helps readers avoid a few of these pitfalls.
(The IPs, mining pool addresses, and other details in this article are included only to reconstruct the incident. Please do not use them for any illegal purpose.)
Background
This server's specs were nothing to write home about:
The symptoms that night were textbook: SSH logins were agonizingly slow, everything lagged, and eventually the load average spiked to 353 (a catastrophic number for a 2-core box), iowait hit 31%, swap was completely full, and the machine was basically catatonic. The only option was a forced reboot from the Tencent Cloud console.
Step 1: Pinpointing the cause of the crash
Once I finally managed to squeeze in over SSH, I started with sar (historical data recorded by sysstat) to trace the load curve during the crash window, then used journalctl and dmesg to check the OOM logs, and the truth surfaced fast:
Conclusion: a 3.6GB machine running 20 containers with no memory limits was guaranteed to hit OOM — it was only a matter of time.
But the real “surprise” was still waiting further down the line.
Step 2: An unexpected discovery — a crypto-mining trojan
While working through the system state, I found a few things in the systemd unit list that had no business being there:
xmrig.service had already logged 108,000 restarts — meaning it had been failing and retrying every 10 seconds for 12 days straight. Since the binary had long since been deleted (later confirmed: Tencent Cloud's YunJing quarantined it shortly after installation), systemd could do nothing but spin in place forever.
Reconstructing the attack chain
Drawing on /var/log/auth.log and journalctl records, the full attack chain looks like this:
Afterward I went to confirm the vulnerability — MCSManager had already published an advisory: every version up to and including v10.16.2 was affected, and my panel was running v10.16.2, the very last version before the patch.
Earlier traces
SSH was in bad shape too: root password login was exposed to the public internet, drawing 12,000+ brute-force attempts a week. During the investigation I personally watched two unfamiliar IPs probing repeatedly at the preauth stage.
Step 3: Cleanup
Once I'd confirmed the intrusion, I did the cleanup in three rounds:
1. Close the entry point
Ports 23333/23334 were cleared, with no leftover processes.
2. Stop all containers
Ran docker stop on all 20 containers. This also relieved the memory pressure as a side effect — swap usage dropped to zero, freeing up safe headroom for the work ahead.
3. Remove malicious persistence
One small hiccup: tianji-reporter.service had a second copy in /usr/lib/systemd/system/ that the first cleanup pass missed; systemctl list-unit-files turned it up again, and only after deleting that copy was everything truly clean. The xmrig binary in YunJing's quarantine was preserved at /usr/local/qcloud/YunJing/quara/ as forensic evidence.
Step 4: A thorough backdoor sweep
With the known backdoors gone, I still had to make sure the attacker hadn't left any other “gifts” behind. I went through it item by item:
What I checked
Result
Accounts / sudo
✅ Only root/ubuntu/lighthouse, no stray uid 0 accounts; passwd/shadow/sudoers untouched since June
cron
✅ Only the Tencent Cloud agent and my own scripts
systemd units
✅ No custom units beyond the 3 already removed
authorized_keys
✅ Only coolify's key and my own; no attacker public key
SUID files
✅ Nothing unusual (the ones inside container snapshots are normal)
deleted-but-running binaries
✅ None
/etc/ld.so.preload
✅ Empty
/etc/hosts, LD_PRELOAD, profile.d, rc.local
✅ Clean
Executables in /tmp /var/tmp /dev/shm
✅ None
dpkg integrity
✅ Only a sudoers md5 change (my own NOPASSWD setup)
Docker images
✅ No mining images
Recent files in /root
✅ All from my own 1Panel install and ops scripts
For the batch of binaries changed in system directories between July 28 and early September (systemd, curl, openssl, dockerd, etc.), I cross-checked the apt history — all were normal unattended-upgrades updates. Conclusion: apart from the three known spots, I found no other backdoors.
Step 5: Hardening and rebuilding
Once the sweep was done, I did three things:
fail2ban
It took effect the moment it was installed — 16 brute-force failures logged within 10 minutes. This machine gets attacked every single day.
Swap 4G → 8G
I did this with all containers stopped and the memory pressure relieved, so it was perfectly safe.
Restore containers + memory limits
Ran docker start on all containers. Along the way I hit one snag: the clickhouse container failed to start, reporting no such volume — the crash had wiped out Docker's volume metadata, but the data directory under /var/lib/docker/volumes/ was still there.docker volume create After recreating a volume with the same name, the data was intact and the container started up normally.
Finally, I gave the OOM culprit chronoframe a memory cap:
Note that docker update may get overwritten when Coolify redeploys, so you need to set it in the app config as well.
Post-mortem summary
Afterword
I never imagined MCSManager would turn out to have a vulnerability this big.
What surprised me even more is how little information about this vulnerability is out there! I only found it by searching keywords after GLM-5.3-Flash mentioned MCSManager's compromise path.
The official handling was even more absurd — they just posted a Release: https://github.com/MCSManager/MCSManager/releases/tag/v10.17.0, didn't file it under Security advisories, assigned no CVE, and didn't wait long enough before disclosing the flaw (though I doubt many people are in the habit of updating anyway). Either way, that approach is a bit off. Like I said: just set up your server by hand from the command line (
Root causes of this incident
Lessons I learned
Still to do (next steps)
A 2-core, 3.6GB machine crammed with self-hosted services, plus an exposed panel and wide-open SSH — that combination was bound to blow up sooner or later. I hope this post-mortem helps readers avoid a few of these pitfalls.
(The IPs, mining pool addresses, and other details in this article are included only to reconstruct the incident. Please do not use them for any illegal purpose.)