# vmon — Operator Guide

**For:** the support team
**Panel:** https://vmon.vpshostingservice.co
**Covers:** 257 nodes, ~10,000 customer VMs
**Last updated:** 2026-10-08

---

## 1. What this panel is

vmon watches every hypervisor in the fleet and every customer VM on it. An agent on each node
reports what it sees every few seconds; the dashboard turns that into graphs, alerts, and — where a
customer needs telling — a support ticket.

It exists to answer three questions:

| Question | Where you answer it |
|---|---|
| Is a **node** unhealthy? | **Nodes** tab, **Alerts** filtered to node alerts |
| Is a **customer VM** abusing the network or the host? | **Alerts** tab, **Fleet** tab |
| Whose VM is this, and can I contact them? | **Identity** tab, the VM's own page |

**It is a monitoring and abuse tool, not a provisioning tool.** Creating, deleting, reinstalling or
resizing a VPS is still Virtualizor's job. vmon can stop, start, suspend and rate-limit a VM, and it
can open a ticket — nothing else.

---

## 2. The short answer: how long to wait before acting

> *"How long should we monitor an alert after it exceeds the defined threshold before taking action?"*

**You do not need to wait at all. The waiting already happened.**

This is the single most important thing to understand about the panel. Every rule has a **hold-down**:
the condition must stay true continuously for that long before an alert is created. A VM that spikes
above a threshold for thirty seconds and settles produces **no alert**.

So when an alert appears, it is already proof of sustained behaviour:

| Alert you see | Means the condition held for |
|---|---|
| VM packet flood or port scan | **5 minutes** without a break |
| VM outbound bandwidth | **15 minutes** without a break |
| VM outbound mail volume | **15 minutes** without a break |
| VM disk IOPS | **30 minutes** without a break |
| VM CPU saturated | **2 hours** without a break |
| Disk utilisation (node) | **15 minutes** without a break |

An alert is not "the number touched the line". It is "the number has been over the line this whole
time and has not come back".

### What to do with that

| Situation | Action |
|---|---|
| **Critical** VM abuse alert | Act now — review the graph and decide within ~15 minutes |
| **Warning** VM abuse alert | Review within the hour; most need no customer contact |
| Alert marked **· clearing** | Do nothing. It has already fallen back and will close itself |
| Alert marked **· suspended** | Already dealt with; the host is dropping the traffic |
| **Node health** alert | Not a customer matter — pass to infrastructure |
| Same alert returning repeatedly | Treat as one ongoing problem, not several |

### The one case where waiting is right

If you open the graph and the line is **bouncing across the threshold** rather than sitting above it,
give it another 15–30 minutes. A VM that hovers at its limit is usually a customer using what they
bought, not abuse. The panel marks these for you — see **· clearing** in §5.

---

## 3. Reading the Alerts tab

The Alerts tab is your work list. Each row is one problem on one target.

| Column | What it means |
|---|---|
| **Severity** | `critical` or `warning`. Greyed out = recovered or suspended |
| **Rule** | What tripped. `+1 on the same metric` means a second severity also fired |
| **Target** | The VM or node. Click through for graphs |
| **Value** | The **current** reading, refreshed continuously |
| **Threshold** | The line it must stay under |
| **For** | How long the alert has existed |
| **State** | `open`, `acknowledged`, `ticketed` |

### Markers you will see beside the value or state

| Marker | Meaning | What you do |
|---|---|---|
| **· clearing** | Below the threshold now, inside the grace period | Nothing. It will close itself |
| **· recovered** | Ticketed, and the behaviour has stopped | Follow up the ticket, not the behaviour |
| **· suspended** | Virtualizor has the VPS suspended | Already contained |
| **+1 on the same metric** | Warning and critical both firing | One problem, not two |
| **⃰** next to a value | Last value seen before the signal stopped | Not a current reading |

### Sort order

Open alerts first, criticals above warnings, suspended and recovered ones last. **The top of the list
is the real work.** If the top is empty, the fleet is quiet regardless of how many rows follow.

---

## 4. Severity — what it actually means

**Severity is about the size of the number, not the urgency of the customer conversation.**

- **warning** — over the line, worth a look. Most need no action.
- **critical** — far enough over that an explanation is required.

A critical does **not** automatically mean suspend. A customer running a legitimate mail server can
trip a critical mail rule. A customer running a backup job can trip a critical disk rule. Always open
the graph before acting.

---

## 5. The life of an alert

```
   condition true for the hold-down
                 │
                 ▼
              [ open ]  ─── you acknowledge ──▶ [ acknowledged ]
                 │                                     │
                 │  you open a ticket                  │
                 ▼                                     ▼
            [ ticketed ] ◀────────────────────────────┘
                 │
     condition stops for the hold-down
                 │
                 ▼
            [ resolved ]   (a ticketed alert keeps its state and shows "· recovered")
```

**Alerts close themselves.** When the condition has been false for the same hold-down, the alert
resolves with no action from you. You do not need to tidy the list.

**Acknowledging** says "I have seen this and I am dealing with it". Use it so two people do not work
the same alert. It does not stop the alert closing itself.

**Resolve manually** only when you have fixed the cause and do not want to wait for the hold-down.

**False positive** suppresses that rule for that target for 24 hours. Use sparingly — it hides a real
signal if you are wrong.

---

## 6. VM abuse rules — the reference table

These are the rules that concern customers. **All of these are about a single VM's behaviour.**

| # | Rule | Fires above | Held for | Severity |
|---|---|---|---|---|
| 42 | VM packet flood or port scan | 5,000 packets/s | 5 min | warning |
| 43 | VM packet flood or port scan (critical) | 50,000 packets/s | 5 min | **critical** |
| 31 | VM outbound packet rate | 50,000 packets/s | 5 min | warning |
| 32 | VM outbound packet rate (critical) | 150,000 packets/s | 5 min | **critical** |
| 33 | VM outbound bandwidth | 500 Mbit/s | 15 min | warning |
| 34 | VM outbound bandwidth (critical) | 900 Mbit/s | 5 min | **critical** |
| 37 | VM direct-to-MX mail (port 25) | 100 connections/min | 15 min | warning |
| 38 | VM outbound mail volume | 500 connections/min | 15 min | **critical** |
| 35 | VM disk IOPS | 5,000 IOPS | 30 min | warning |
| 44 | VM disk read IOPS | 3,000 IOPS | 15 min | warning |
| 45 | VM disk write IOPS | 2,000 IOPS | 15 min | warning |
| 30 | VM CPU saturated | 125% of its allocation | 2 h | warning |
| 40 | VM denied CPU by its host | 10% denied | 15 min | warning |
| 41 | VM denied CPU by its host (critical) | 40% denied | 15 min | **critical** |

### How to read the important ones

**Packet flood (42/43)** — the clearest abuse signal there is. This counts only *small* packets
(under 150 bytes average), which is what a flood or a scan looks like. Normal traffic has large
packets and does not trip it. **A sustained critical here is almost always a compromised VM.**

**Outbound packet rate (31/32)** — all outbound packets regardless of size. A busy legitimate server
can trip the warning. The critical at 150,000/s rarely has an innocent explanation.

**Outbound bandwidth (33/34)** — raw throughput. Backups, large file serving and CDN origins all trip
this legitimately. **Check what the customer bought before acting.**

**Mail (37/38)** — rule 37 counts connections to **port 25**, which is mail delivered directly to
other mail servers. That is what a spam run does; ordinary customers send through a relay on port 587.
**Rule 37 is the stronger abuse signal even though it is only a warning.** Rule 38 counts all mail
ports and catches sheer volume.

**Disk (35/44/45)** — almost never abuse. Usually a busy database or a backup window. These matter
because one VM can slow a whole node, which is why they exist — but the answer is usually a rate
limit or a conversation, not a suspension.

**CPU saturated (30)** — the customer is using more than 100% of the vCPUs they bought, which is
normal on a burstable plan. At 125% sustained for two hours it is worth a look, but it is **not
abuse**.

**Denied CPU by its host (40/41)** — ⚠️ **this is our fault, not the customer's.** It means the node
is too busy to give the VM the CPU it paid for. **Never ticket a customer for this.** Escalate to
infrastructure: a node needs attention or the VM needs moving.

---

## 7. Node health rules — the reference table

These concern **hypervisors**, not customers. You will not open a customer ticket for any of them.
Pass them to whoever looks after the hardware.

| # | Rule | Warning | Critical | Held for |
|---|---|---|---|---|
| 1 / 2 | Host CPU busy | 90% | 97% | 15 min |
| 3 / 4 | Load per thread | 2.0 | 4.0 | 10 min |
| 5 / 6 | Memory available | below 10% | below 5% | 5 min |
| 7 / 8 | Swap in use | 80% | 95% | 10 min |
| 9 / 10 | I/O wait | 20% | 40% | 10 min |
| 12 / 13 | Disk utilisation | 85% | 95% | 15 min |
| 14 / 15 | Disk latency | 50 ms | 150 ms | 10 min |
| 16 / 17 | Filesystem usage | 85% | 92% | 5 min |
| 18 / 19 | Filesystem inodes | 85% | 95% | 5 min |
| 20 / 21 | Conntrack table | 70% | 90% | 5 min |
| 23 | NIC drops | 100 pps | — | 5 min |
| 24 | **libvirt down** | — | any | 1 min |
| 25 / 26 | Clock skew | 30 s | 300 s | 2 min |
| 27 / 28 | Agent dropping data / buffer growing | — | — | 5–10 min |
| 29 | VM count changed unexpectedly | 5 changes | — | immediate |

**The two that matter most:**

- **libvirt down (24)** — the node cannot manage its VMs at all. Escalate immediately.
- **Filesystem usage (17)** — a full disk on a hypervisor takes customer VMs down with it.

---

## 8. What to do when an alert fires

### Step 1 — Is it still happening?

Open the alert. Look at the **Value** column and the graph.

- Marked **· clearing**? → It has stopped. Close the tab, do nothing.
- Marked **· suspended**? → Already contained. No action.
- Value well above the threshold and holding? → Continue to step 2.

### Step 2 — Who is it?

Click the VM name. The VM page shows the customer, the plan, and the graphs.

- **No customer shown?** Check the **Identity** tab. A VM with no billing record cannot be ticketed —
  escalate instead of guessing.

### Step 3 — Is it actually abuse?

Look at the graph for the last 24 hours and ask:

| What you see | Likely meaning |
|---|---|
| Flat line at a high level, started suddenly | Compromised or deliberate — act |
| Regular spikes at the same time daily | A backup or cron job — usually fine |
| Gradual climb over days | Growing legitimate workload — consider an upsell |
| Bouncing across the line | At their plan limit — not abuse |
| Started right after a reinstall | Often a compromised image |

### Step 4 — Act

In order of least to most disruptive:

1. **Nothing** — note it, let it resolve. Correct for most warnings.
2. **Acknowledge** — you are watching it.
3. **Open a ticket** — tell the customer, ask them to fix it.
4. **Limit traffic** — cap throughput; the VM keeps running.
5. **Suspend network** — traffic stops instantly, VM and data untouched. **The right answer for
   active abuse.**
6. **Stop VM** — a clean shutdown. For a customer request, not for abuse.
7. **Suspend VM** — stops it *and* marks the account suspended.

**For active abuse, "Suspend network" is almost always the right action.** It stops the harm
immediately, costs the customer no data, and is reversible with one click.

---

## 9. The actions, in detail

Every destructive action asks you to type the VM's name and give a reason. **The reason is the only
record of why** — write something a colleague could understand next week.

| Action | What happens | Reversible? |
|---|---|---|
| **Suspend network** | Traffic stops at once. VM runs, data intact, connections drop | Yes — Restore network |
| **Restore network** | Traffic resumes | — |
| **Limit traffic** | Changes the throughput cap. Nothing stops | Yes — set it back |
| **Stop VM** | Guest asked to shut down cleanly. Disk kept | Yes — Start VM |
| **Start VM** | Boots it again | — |
| **Suspend VM** | Stops it **and** marks the account suspended in Virtualizor | Yes — Unsuspend |
| **Restart VM** | Reboots it. Unsaved work lost | No |

### Stop VM vs Suspend VM — the difference matters

Both stop the VM. The difference is the **account flag**:

- **Stop VM** — a power-down. Billing and the customer's own panel are unaffected. Use when the
  customer asked, or when a VM needs to stop for a technical reason.
- **Suspend VM** — a statement that the account is suspended. The customer sees it in their panel.
  Use when the account is genuinely being suspended.

### A warning about Restart

**Restarting a compromised VM destroys the evidence** of how it was compromised, and it reboots still
compromised. It is rarely the answer to abuse.

---

## 10. Opening a ticket

From an alert, or from a VM's page via **Open a ticket**.

1. Pick the rule you are reporting.
2. Pick the window you want to show the customer.
3. The template is chosen for you — one written for that specific rule.
4. **Read the preview.** It is exactly what the customer receives.
5. Send.

The graph for that window is attached automatically, so the customer can see what you saw.

### Before you send

- ✅ The **measured value and threshold** in the text are correct.
- ✅ The **customer name** is the right one.
- ✅ The template matches the evidence — do not send a spam template for a disk alert.
- ⚠️ If a banner says the ticket will appear **from the customer themselves**, stop and tell an admin.
- ⚠️ If no template is offered, the rule has none on purpose (rules 40/41 — our fault, not theirs).

### What the templates say

| Template | Used for |
|---|---|
| Outbound packet flood | Packet rate and flood rules (31, 32, 42, 43) |
| Outbound bandwidth | Bandwidth rules (33, 34) |
| Outbound mail (spam) | Mail rules (37, 38) |
| Port scanning or SYN flood | Mail rules (37, 38) **when the evidence is a sweep, not sending** — see below |
| Disk I/O / read / write saturation | Disk rules (35, 44, 45) |
| Resource saturation | CPU saturated (30) |

**Two templates are offered for the mail rules, and the choice matters.** A VM making thousands of
connections to port 25 may be *sending spam* or *scanning for open mail servers* — the rule cannot
tell them apart, because both look like outbound connections to a mail port. Open the graph and the
VM page:

- Mostly port 587, steady, with replies → **sending**. Use *Outbound mail (spam)*.
- Port 25, many different destinations, few replies → **scanning**. Use *Port scanning or SYN flood*,
  which tells the customer their server is likely compromised and should be rebuilt.

Sending the wrong one tells a customer to "stop the sending process" when nothing is being sent.

---

## 11. Worked examples

### Example A — a genuine flood

> `VM packet flood or port scan (critical)` on `v58124`, 57,528 packets/s against 50,000, open 12 minutes.

1. Value is holding well above the line, no **· clearing** marker → real.
2. VM page: customer identified, plan is a 2 GB VPS.
3. Graph: flat at ~57,000 pps since it started. Nothing gradual about it.
4. **Suspend network**, reason: *"sustained 57k pps outbound flood, suspended pending customer
   response"*.
5. Open a ticket with the **Outbound packet flood** template.

### Example B — a mail alert that is not spam

> `VM direct-to-MX mail (port 25)`, 114/min against 100, open 20 minutes.

1. Value is just over the line.
2. Graph: steady ~110/min all week, not a sudden change.
3. VM page: the customer's plan and hostname suggest a mail server.
4. **No action.** Note it. If it climbs, revisit.

A long-standing, steady rate just above a threshold is a customer doing their normal work.

### Example C — the one you must not ticket

> `VM denied CPU by its host` on three VMs, all on the same node.

1. All three on one node → the node is the problem.
2. **Do not contact the customers.** They are the victims here.
3. Escalate to infrastructure: the node is oversubscribed or something on it is misbehaving.

### Example D — an alert that cleans up after itself

> `VM disk IOPS`, value shows `4,265 · clearing peak 7,740`.

Below the threshold now, inside the grace period. **Do nothing.** It will close on its own.

---

## 12. The Identity tab

Answers "whose VM is this?" and flags VMs where the answer is missing.

| Sub-tab | Meaning |
|---|---|
| **Unmapped** | Everything below, together |
| **No billing record** | Virtualizor knows it, WHMCS has no service |
| **Unknown to Virtualizor** | A VM on a node that Virtualizor has no record of |
| **Claimed twice** | Two WHMCS services claim one VPS |
| **All VMs** | Everything |

**Why it matters to you:** a VM in any of the first four **cannot be ticketed** — there is no customer
at the end of the chain. If you hit an abuse alert on one, escalate rather than guess.

The **Virtualizor email** column is copy-to-click. For a VM with no billing record, that address is
the only link to WHMCS — paste it into client search to find out what happened.

---

## 13. Things not to do

| Don't | Why |
|---|---|
| Act on a value without opening the graph | The number is one moment; the graph is the story |
| Ticket a customer for rules 40 or 41 | That is our node failing, not their VM |
| Use **Restart** on a suspected compromise | Destroys evidence, reboots still compromised |
| Mark something a false positive to clear the list | Hides a real signal for 24 hours |
| Suspend an account when you meant to stop a VM | Different thing to the customer and to billing |
| Chase an alert marked **· clearing** | It has already stopped |
| Act on a VM with no billing record | You cannot be sure whose it is |

---

## 14. Glossary

| Term | Meaning |
|---|---|
| **Node** | A physical hypervisor running customer VMs |
| **VM / VPS** | One customer's virtual server |
| **Hold-down** | How long a condition must hold before an alert appears |
| **Threshold** | The line a value must stay under |
| **pps** | Packets per second |
| **IOPS** | Disk operations per second |
| **Direct-to-MX** | Mail sent straight to other mail servers (port 25) — the spam pattern |
| **Submission** | Mail sent through a relay (port 587) — the normal pattern |
| **Clearing** | Below the threshold, waiting out the grace period before closing |
| **Unmapped** | A VM vmon cannot link to a paying customer |

---

## 15. Escalate rather than act when

- The VM has **no billing record** or is **claimed twice**.
- The alert is **rules 40 or 41** (denied CPU) — infrastructure, not support.
- A **node health critical** is firing, especially **libvirt down**.
- Several VMs on **one node** alert at once — suspect the node.
- A ticket preview warns it will be sent **from the customer**.
- You are about to act on a VM and are not certain it is abuse.

**When in doubt: acknowledge the alert, note what you see, and ask. An alert left open costs nothing.
A wrongly suspended customer costs a great deal.**