Detection Signals: Userland Rootkit Mining on Hypervisors

1. Unexpected systemd units

A unit whose name and description look like platform software but which is not owned by any distribution package. In the case study the unit was PVE-1, described as "Proxmox VE node health monitor" — plausible until you look at where it lives and what it runs.

# List every enabled unit, then diff against package-owned units.
systemctl list-unit-files --state=enabled

# A package-owned unit can be traced back:
dpkg -S /lib/systemd/system/<name>.service     # Debian/Ubuntu
rpm -qf /usr/lib/systemd/system/<name>.service # RHEL family

# A unit with NO owning package is suspect.

Also check for units outside the standard directories:

find /etc/systemd /usr/lib/systemd /lib/systemd -name '*.service' -newer /etc/hostname

2. LD_PRELOAD outside distribution units

The rootkit scoped its preload per-unit rather than globally (the attacker's own comment explained that a global /etc/ld.so.preload breaks sshd). That makes unit-level LD_PRELOAD the tell.

grep -rn "LD_PRELOAD" /etc/systemd /usr/lib/systemd /lib/systemd 2>/dev/null

# Also check the login-path files the rootkit used:
grep -rn "LD_PRELOAD" /etc/environment /etc/profile /etc/profile.d/ 2>/dev/null

Any hit that is not systemd, snapd, or a known vendor is worth investigating.

3. Processes running from improbable paths

Mining binaries do not belong under /var/lib/systemd/ or /tmp or a hidden dot-directory. Read the real executable path from /proc, not from ps (which the rootkit filtered).

# exe path and cmdline straight from the kernel, bypassing ps:
for p in /proc/[0-9]*; do
  exe=$(readlink "$p/exe" 2>/dev/null)
  case "$exe" in
    */var/lib/*|*/tmp/*|*/.cache/*|*/.hide/*) echo "$p -> $exe";;
  esac
done

4. Mining-pool outbound connections

A hypervisor has no reason to talk to a mining pool. Watch for the common ports regardless of destination.

ss -tunap | grep -E ':(20004|3333|4444|5555|7777|9001|8080)\b'

In the case study the destination was :20004 (MoneroOcean), but the port was also hidden from ss/netstat by the preload library — so also check from a source that is not filtered:

# Raw conntrack / firewall view is outside the libc the rootkit hooks:
conntrack -L 2>/dev/null | grep -E ':20004|:3333'
nft list ruleset 2>/dev/null | grep -E '20004|3333'

5. Immutable or unexpected ld.so.preload

The file should not exist on a normal host. Its presence, and especially an lsattr that fails on it, is a strong signal.

test -e /etc/ld.so.preload && echo "PRESENT — investigate" && cat /etc/ld.so.preload
lsattr /etc/ld.so.preload    # "Permission denied" => immutable flag or hardened

6. Sustained CPU with no matching workload

Correlate host CPU against the sum of guest CPU. On a hypervisor the two should track. A gap means something is running on the host itself.

# Host-side busy (approx) vs. how much the VMs claim.
top -bn1 | head -5
qm list

7. Self-updating persistence

A unit with a timer that re-downloads its own binary is not normal platform behaviour. Look for daily timers referencing an updater script.

systemctl list-timers --all
# Inspect any timer's ExecStart for a downloader.
systemctl cat <name>-update.service

Randomized delays (e.g. RandomizedDelaySec=1800) on such a timer can be a deliberate anti-lockstep measure rather than an ops choice.

Mapping to Wazuh rules

These signals translate to Wazuh detections in the companion observability-security-stack repository:

Signal	Wazuh angle
New systemd unit not owned by a package	File integrity monitoring on /etc/systemd, /usr/lib/systemd
LD_PRELOAD in unit or login files	File integrity monitoring + rule on content match
Process from /var/lib/*, /tmp/*	Auditd execve rules + custom decoder
Mining-pool ports	Custom rule on firewall/outbound logs
ld.so.preload creation	File integrity monitoring (alert on any write)

Health warning