Installing and updating software on Linux

Software on a Linux server comes from a package manager, which downloads packages from repositories, checks their signatures, installs their dependencies, and keeps track of what is on the machine. Which one you use depends on the image your server runs: apt on Debian and Ubuntu, dnf on RHEL, AlmaLinux and Rocky Linux.

If you are not sure which you have, cat /etc/os-release names the distribution. On the RHEL family, yum still works and is a link to dnf, so either name is safe to type.

The same jobs in both

Every command below is run as root or with sudo.

Two differences worth knowing rather than memorising:

apt update is not an upgrade. It only refreshes the list of what is available. apt upgrade is the one that installs. Running apt upgrade without a preceding apt update upgrades to whatever the stale list knew about, which on a server that has been running for a month is nothing. dnf refreshes its metadata automatically, so it has no equivalent first step.

apt upgrade will not remove packages, apt full-upgrade will. If an upgrade needs to remove something to proceed, plain apt upgrade holds it back and tells you it was kept back. full-upgrade allows the removal. Read what it proposes to remove before agreeing, particularly on a server running something you care about.

dnf also keeps a transaction history, which apt does not, and it can reverse one:

dnf history
dnf history undo 42

That is the closest thing to an undo button either package manager has, and it is worth knowing before you need it.

Restarting what you upgraded

Upgrading a package replaces the files on disk. A process that is already running keeps using the code it started with, so a patched service stays vulnerable until it is restarted, and a patched library stays in use by every process that loaded it. This is the step people miss, and it is why "I ran the updates" is not the same as "the fix is applied".

On Debian and Ubuntu, needrestart reports which services are running outdated code, and on recent releases it prompts during the upgrade itself:

apt install -y needrestart
needrestart

On RHEL, AlmaLinux and Rocky, the same job is done by a utility in dnf-utils:

dnf install -y dnf-utils
needs-restarting -s
needs-restarting -r

-s lists the services, -r answers the separate question of whether a reboot is needed, which is the case for a kernel or core library update. Restarting services is covered in Managing services with systemd.

Holding a package back

Sometimes you need a specific version to stay put: a runtime your game server is built against, a kernel with a working driver, or a package you have patched by hand. Both managers can pin one in place.

On Debian and Ubuntu:

apt-mark hold nginx
apt-mark showhold
apt-mark unhold nginx

On RHEL, AlmaLinux and Rocky, holding needs the versionlock plugin, which is not installed by default. Install it by the capability name so you do not have to guess the package name for your release:

dnf install -y 'dnf-command(versionlock)'
dnf versionlock add nginx
dnf versionlock list
dnf versionlock delete nginx

To skip a package for one command only, without a permanent hold, dnf takes --exclude:

dnf upgrade --exclude=kernel\*

Unattended upgrades

Automatic updates close the gap between a fix being published and you getting around to it, which for a server on a public address is the window that matters. The tradeoff is that software changes underneath a running service without you watching.

On Debian and Ubuntu

apt install -y unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

That writes /etc/apt/apt.conf.d/20auto-upgrades, which controls how often the list is refreshed and whether upgrades are applied. The policy itself lives in /etc/apt/apt.conf.d/50unattended-upgrades, where the default is to take security updates only and leave everything else to you. That default is the one to keep on a server running a game.

Test the configuration without installing anything:

unattended-upgrade --dry-run --debug

On RHEL, AlmaLinux and Rocky

dnf install -y dnf-automatic

Edit /etc/dnf/automatic.conf. The two settings that decide its behaviour are upgrade_type, which takes default or security, and apply_updates, which must be yes for anything to be installed rather than only downloaded and reported. Then enable the timer:

systemctl enable --now dnf-automatic.timer
systemctl list-timers dnf-automatic.timer

The restart question

Neither tool restarts your game server, and both leave the same gap described above: the new code is on disk and the old code is still running.

Debian and Ubuntu can reboot on their own, with these settings in /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";

Whether to turn that on is a real decision, not a default to accept:

• Automatic reboot on. Kernel and library fixes actually take effect. The cost is that the server goes down at that time, without warning, whether or not anyone is playing. If your player base is spread across timezones there may be no genuinely quiet hour.
• Automatic reboot off. Nothing surprises you, and the machine can sit for weeks running a patched kernel image and an unpatched kernel.

The workable middle is to let security packages install automatically, leave the reboot off, and check needs-restarting -r or /var/run/reboot-required as part of a maintenance window you announce. Whichever you pick, make sure your service comes back on its own after an unplanned reboot: that is what enable and Restart= are for, and both are on Managing services with systemd.

Adding a third-party repository

Sooner or later something you need is not in the distribution's repositories: a newer database, a language runtime, a vendor's own agent. Adding their repository is normal and often correct. It is also the largest amount of trust you will hand out on the server, so it is worth being deliberate about.

Why an unverified repository is a supply-chain risk

A repository does not only supply the one package you wanted. It supplies packages, and packages run installation scripts as root, on your machine, at install time and again at every upgrade. Whoever controls that repository can therefore run commands as root on your server at any point in the future, not only today, and can ship a package that replaces a system one.

That is fine when the repository belongs to the vendor whose software you are deliberately installing, over HTTPS, on their own domain, with a signing key you got from them. It is not fine for a repository you found in a forum post, a mirror you cannot identify, or a signing key downloaded from somewhere other than the vendor. The failure mode is not that the package is broken, it is that one compromised build server pushes an update to you automatically and nothing looks wrong.

The same reasoning applies to the install command that pipes a script from the internet into a root shell. If you are going to run it, download it first, read it, and then run it.

On Debian and Ubuntu

The modern method stores the signing key in /etc/apt/keyrings/ and ties it to one repository with signed-by:

install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://repo.example.com/gpg.key | gpg --dearmor -o /etc/apt/keyrings/example.gpg
chmod 0644 /etc/apt/keyrings/example.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/example.gpg] https://repo.example.com/apt stable main" > /etc/apt/sources.list.d/example.list
apt update

Substitute the vendor's real key URL and repository line, which their own documentation gives you. Take both from the vendor's site over HTTPS, not from a third party.

apt-key is the old way and has been removed. It is worth knowing why, because it explains what signed-by buys you: a key added with apt-key was trusted for every repository on the system, so a key you added for one vendor could validate a replacement for any package, including a core system one. signed-by scopes a key to the single repository it belongs to.

On RHEL, AlmaLinux and Rocky

Repositories are files in /etc/yum.repos.d/. Write one, keeping gpgcheck=1:

[example]
name=Example vendor packages
baseurl=https://repo.example.com/el9/x86_64/
enabled=1
gpgcheck=1
gpgkey=https://repo.example.com/RPM-GPG-KEY-example

Then import the key explicitly, so you see the fingerprint rather than accepting it in passing during an install:

rpm --import https://repo.example.com/RPM-GPG-KEY-example
dnf makecache

Many vendors ship a .repo file inside a release package instead, which is equivalent and less error-prone. EPEL, the community repository the RHEL family usually needs first, is one:

dnf install -y epel-release

Stop a third-party repository taking over base packages

A third-party repository that also carries a newer build of a system package can quietly replace it at your next upgrade. Both managers can limit a repository to the packages you added it for.

On the RHEL family, the direct way is to restrict the repository to the packages you actually want from it, by adding to its section in /etc/yum.repos.d/example.repo:

includepkgs=example,example-agent

Nothing else in that repository is then considered at all. excludepkgs is the inverse if a blocklist suits you better. Repository priority is the other lever: priority defaults to 99 and a lower number wins, so giving the base repositories priority=1 keeps them ahead of anything you add later.

On Debian and Ubuntu, the equivalent is a pin in /etc/apt/preferences.d/, which allows only the packages you name from that origin. apt-cache policy is worth running afterwards, with and without a package name, to confirm which repository each package will come from.

Related