What to do if your server is compromised

Work in this order: cut the machine off the network, capture what it looks like right now, find out what got in, rotate every credential it held, then rebuild it from clean media. Do not start by deleting files, and do not reboot yet.

The order exists for a reason. Isolation stops the damage continuing. Capturing the state first is what lets you find the entry point, and without the entry point a rebuild gets compromised the same way within days. Rebuilding last is the honest ending: a machine somebody else has had root on cannot be reliably cleaned in place.

Signs that this is what you are looking at

Any one of these is worth investigating. Several together is not a coincidence.

• A process you do not recognise using most of the CPU, often with a plausible-looking name, frequently a coin miner.
• Outbound traffic that has no explanation, or an abuse complaint saying your address is attacking or scanning someone.
• Sign-ins you did not make in the auth log, or from addresses and countries you have never used.
• A user account, an SSH key, a cron job, or a systemd service you did not create.
• Files in /tmp, /dev/shm, or /var/tmp that are executable, or web directories containing scripts nobody deployed.
• Log gaps. An auth log that starts abruptly, or history files emptied. Missing evidence is evidence.
• A password that stopped working, or a service that restarts itself after you stop it.
• Your game server, website, or players seeing behaviour you cannot reproduce in your own code, such as commands granting themselves administrator.

First, stop the bleeding

The goal is to remove the machine from the network while keeping yourself able to look at it.

Isolate it at the host firewall, from the console, keeping one way in for yourself. On Debian and Ubuntu with ufw:

sudo ufw default deny incoming
sudo ufw default deny outgoing
sudo ufw allow from 203.0.113.10 to any port 22 proto tcp
sudo ufw enable

Replace 203.0.113.10 with your own address. ufw keeps established connections alive, so an SSH session you already have open survives this, and you can still reach the machine from the address you allowed.

On AlmaLinux, Rocky and RHEL with firewalld, the equivalent for inbound is to make drop the default zone and allow only your own address back in:

sudo firewall-cmd --set-default-zone=drop
sudo firewall-cmd --add-rich-rule='rule family="ipv4" source address="203.0.113.10" service name="ssh" accept'

firewalld also has a panic mode, which drops every packet in both directions immediately:

sudo firewall-cmd --panic-on

Denying outbound traffic matters as much as denying inbound. It cuts the connection to whoever is controlling the machine, stops data leaving, and stops your server participating in an attack on somebody else. It also breaks package updates and DNS, which is a fair trade for now.

The ufw commands above cover both directions. The drop zone covers inbound only, so on the RHEL family the outbound half comes from stopping the compromised service, or from panic mode.

Then stop the exposed service itself if you know which one it is, for example sudo systemctl stop nginx or your game server's unit.

What not to do yet

• Do not reboot. Anything running only in memory is lost, and persistence usually survives the reboot anyway, so you lose the evidence and keep the intruder.
• Do not delete the files you find. Deleting the visible parts leaves the parts you have not found and destroys the timeline you need.
• Do not reinstall yet. That is where this ends, but a rebuild before you know the entry point rebuilds the same hole.
• Do not only block the attacker's IP. Blocking a source address is worth doing, and it is not remediation: access is already established and the address is disposable. See Blocking and allowing IPs for when it does help.

If you cannot isolate the machine at all and it is actively attacking others, powering it off is better than leaving it running. You lose memory evidence, and that is the right trade.

Capture the state before you destroy it

You are not doing courtroom forensics. You need two answers: how they got in, so the rebuild is not a repeat, and what they could read, so you know which credentials and whose data are exposed.

Run these from your own machine over SSH, assuming you kept a path in for your own address, and save the output locally rather than on the server. Redirecting from your own terminal keeps the server unchanged. If you cut the network completely, run the same commands in the panel console and read them on screen instead.

mkdir -p incident && cd incident
ssh root@203.0.113.10 'ps auxf'                             > processes.txt
ssh root@203.0.113.10 'ss -tunap'                           > connections.txt
ssh root@203.0.113.10 'systemctl list-units --type=service' > services.txt
ssh root@203.0.113.10 'systemctl list-timers --all'         > timers.txt
ssh root@203.0.113.10 'getent passwd'                       > users.txt
ssh root@203.0.113.10 'last -a; lastb | head -50'           > logins.txt
ssh root@203.0.113.10 'journalctl --since "7 days ago" --no-pager' > journal.txt

Then pull the logs themselves, which are the part most often erased next:

rsync -avz root@203.0.113.10:/var/log/ ./var-log/

Work out what got in

Go through these deliberately. Each one answers a different question.

Processes and network

Look for processes whose binary has been deleted, which is normal for malware and abnormal for anything legitimate:

sudo ls -l /proc/*/exe 2>/dev/null | grep deleted

List everything listening and everything connected out:

sudo ss -tulpn
sudo ss -tunap

Anything listening that you did not put there is a backdoor until proven otherwise. Any established outbound connection to an address you do not recognise is worth writing down, because that address often appears in the logs around the time of entry.

Users and keys

getent passwd
awk -F: '($3 == 0) {print $1}' /etc/passwd
sudo find /root /home -name authorized_keys -exec ls -l {} + -exec cat {} +

Every account with UID 0 is root by another name. Every key in every authorized_keys file is a way back in that does not need a password.

Scheduled work

Re-entry is usually scheduled, so that killing the process changes nothing.

sudo crontab -l -u root
ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly
sudo systemctl list-timers --all

User crontabs live in /var/spool/cron/crontabs on Debian and Ubuntu, and in /var/spool/cron on the RHEL family. Also read /etc/rc.local, any systemd unit with a recent modification time, and the shell startup files (.bashrc, .profile) of every account, including root.

Changed and recent files

Files modified around the time things started going wrong are the trail:

sudo find / -xdev -newermt "2026-08-01" -type f 2>/dev/null | head -100
sudo find /tmp /dev/shm /var/tmp -type f -executable -ls

Compare packaged files against what the distribution shipped. On the RHEL family:

sudo rpm -Va | head -50

On Debian and Ubuntu, after sudo apt install debsums:

sudo debsums -c

Both list files that differ from the package. Neither is conclusive, since the package database is also on the machine, but an altered sshd or ls shows up here often enough to be worth the minute it takes.

The logs, for the way in

Successful sign-ins, on Debian and Ubuntu:

sudo grep -E 'Accepted (password|publickey)' /var/log/auth.log

On the RHEL family the same lines are in /var/log/secure. The service unit is ssh on Debian and Ubuntu and sshd on the RHEL family, so journalctl -u ssh or journalctl -u sshd gives you the same history where the files have been rotated away.

Then read the application logs for the exposed service around the same timestamps: your web server's access log, your game server's console log, the admin panel's log. The entry point is nearly always one of a short list:

• A password that was guessed, reused from a leak elsewhere, or left at a default.
• A service exposed to the internet that should not have been, particularly a database, a cache, or an admin panel.
• Third-party code you installed: a plugin, a resource, or a script from a forum or a re-upload, carrying a backdoor deliberately.
• An unpatched application with a known vulnerability, usually one exposed on a web port.
• A credential leaked somewhere else: your own machine, a shared config, a screenshot, a chat.

If you cannot find the entry point, assume the worst of that list and close all of them during the rebuild.

Rotate every credential the machine held

Anything that was stored on that server, typed into it, or reachable from it is now public as far as you are concerned. Change it, do not merely stop using it.

• Root and every user password on the machine, on the rebuilt machine.
• SSH keys. Any private key that sat on the server is burnt: generate new ones and remove the old public keys from every machine that trusted them.
• Database passwords, and application secrets in .env files or config.
• API tokens and webhooks, including bot tokens, payment keys, and any webhook URL, which is itself a credential.
• Game platform licence keys issued to that server, where the platform supports reissuing them.
• TLS private keys. Reissue the certificates and revoke the old ones.
• Your Clouduxe account password, if you ever typed it on that machine or a browser session on it was signed in. See Securing your Clouduxe account.
• Anything you typed into an SSH session on it. A modified SSH server records passwords, so the credential does not need to have been stored to be taken.

Do this before the rebuild is exposed to the internet, not after.

Rebuild, do not clean

This is the part people resist, so here it is plainly: once an intruder has had root, there is no reliable way to establish that you have removed all of them. Backdoors are cheap and small, they are placed in several forms on purpose, and the tools you would use to search for them run on the same machine and can be altered. Finding and deleting three things tells you nothing about the fourth. The only state you can trust is one you built.

1. Copy off the data you need, and nothing else

   Databases, uploads, world and player data, and configuration files kept for reference rather than for restoring directly. Do not copy binaries, and treat everything you copy as untrusted input until you have looked at it.

2. Reinstall the operating system

   Rebuild from the panel, from a clean image. Do not restore a full system image of the old machine: that restores the intrusion with it.

3. Harden before you expose anything

   Do this while nothing is listening: key-only SSH, a default-deny firewall, a non-root user for the service, and automatic updates. Work through Securing a new server, SSH hardening, and The Linux firewall.

4. Reinstall the software from upstream sources

   Get the operating system packages, the game server build, and every plugin, resource, or extension from the vendor, not from your backup. If a piece of third-party code is the suspected entry point, do not install it again in the same form.

5. Restore data, deliberately

   Restore the database dumps and user content, and re-apply configuration by hand from your reference copy rather than by dropping old files back in place. Web directories and upload directories are the classic hiding place for a web shell, so check what you are restoring rather than restoring it wholesale.

6. Set new credentials everywhere

   New passwords, new keys, new tokens, from the rotation list above. Restoring a database restores its user accounts and their password hashes too, so change application passwords after the restore, not before it.

7. Put your protection rules back, then reopen

   Add the port rules for the services you are actually running again, with limits sized to your traffic. See Port rules. If players cannot connect once you reopen, work through When players cannot connect before loosening anything.

8. Watch it for a while

   Check listening ports, cron, users, and CPU over the following days, and add Fail2ban so repeated attempts are blocked rather than merely logged. A repeat within a week means the entry point is still open.

Choosing a backup that does not restore the backdoor

The intrusion is in every backup taken after it started, and often the backups themselves were a target.

• Restore from before the earliest evidence you found, with margin.
• Restore data, not system state. Dumps, uploads, and world files. Not /etc in bulk, not cron directories, not systemd units, not binaries.
• Check what you restore. Look for scripts in directories that should only hold uploads, and for files with modification dates that do not fit.
• Rebuild the application from source or from the vendor, and restore only its data alongside.
• If the only backups you have are newer than the intrusion, restore the data anyway and treat every file in it as suspect: re-import database contents rather than whole systems, and inspect user content before it is served again.

Planning backups that actually work covers the retention that makes this possible, which is keeping copies old enough to predate an intrusion you notice late.

If you need a panel snapshot restored, that goes through support and the date you name matters: see Backups and put the intrusion timeline in the ticket so the right snapshot is chosen.

Windows servers

The same order applies, with different tools. Isolate with Windows Firewall or by stopping the exposed service, then look at Task Manager or Process Explorer for unexpected processes, Task Scheduler and the Run keys for persistence, Local Users and Groups for accounts you did not create, and netstat -ano for what is listening.

Sign-ins are in the Security event log: event ID 4624 is a successful sign-in and 4625 a failed one, and logon type 10 means it came over Remote Desktop.

Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4625} -MaxEvents 50

RDP exposed to the whole internet is the usual entry point, so close that during the rebuild. See Hardening Remote Desktop.

Getting help

Open a ticket if you need something only Clouduxe can do, including a snapshot restore or help getting back in to a server you have locked yourself out of. Include what you found, when you think it started, and what you have already done, so the reply is not a round trip.

Support can act on your service. Investigating what happened inside your operating system, and rebuilding it, is yours: you have root on the machine, and nobody else can tell you which of your files belong there.

Related