Mail alerts for HVM clusters in HPE Morpheus VM Essentials

When a host in an HVM cluster goes down, VM Essentials shows it on the cluster page. The cluster status changes to Warning, the quorum panel marks the host unreachable, and the agents on the other hosts start moving its VMs. That works well while someone has the page open. I wanted a mail as well, sent by the VME manager itself.

So I wrote a plugin for it. It adds an HVM Alert tab to every HVM cluster and checks the cluster every two minutes. It mails what opened, what changed and what cleared, using the SMTP settings the appliance already has. The source and a ready-built jar are on GitHub: emrbaykal/hvm-cluster-alert. It is a community plugin under the Apache 2.0 licence, not an HPE product, and HPE does not support it.

I have run it on two setups. The first is VM Essentials 9.0.2 with a single manager VM and a three-host cluster: layout 1.3, HVM OS 24.04, Morpheus agent 3.3.0, GFS2 datastores on Fibre Channel. The second is Morpheus Enterprise 9.1.0.1 with three app nodes behind a load balancer and a two-host cluster: layout 2.0, HVM OS 26.04, agent 4.0.1, with an NFS heartbeat datastore.

What it watches

The host checks run through the Morpheus agent that is already on every host. On each host the agent runs one fixed, read-only script with a timeout. Apart from a 0-byte lock file under /run and its own keys in Cypher, the plugin changes nothing in Morpheus or on the hosts.

Most problems have to show up on two checks in a row before they are mailed, and must be gone for two checks before they count as resolved. That way one slow answer from a host doesn't turn into a mail. Some conditions have already waited at the source, such as quorum lost, GFS2 withdraw or a peer not reached for more than 60 seconds; those go out on the first check. Agent log events are mailed as soon as they are read.

Install

1. Download the jar and check it.

curl -LO https://github.com/emrbaykal/hvm-cluster-alert/releases/download/v1.19.0/hvm-cluster-alert-1.19.0-all.jar curl -LO https://github.com/emrbaykal/hvm-cluster-alert/releases/download/v1.19.0/hvm-cluster-alert-1.19.0-all.jar.sha256 shasum -a 256 -c hvm-cluster-alert-1.19.0-all.jar.sha256 

The last line should print OK. If you would rather build it yourself, clone the repository and run ./gradlew shadowJar with JDK 11 to 17; the jar lands in build/libs/.

2. Make sure the appliance can send mail. The fields under Administration > Settings > Email have to be filled in. The plugin reads the SMTP settings Morpheus loaded at startup. If you change them, restart the UI service on the manager:

morpheus-ctl restart morpheus-ui 

On a manager with several app nodes, restart one node at a time. If a restart has not finished after a few minutes, morpheus-ctl kill morpheus-ui lets runit start a fresh UI process.

3. Upload the plugin. Go to Administration > Plugins > Add and upload the jar. HVM Cluster Alert 1.19.0 should then appear in the plugin list.

4. Fill in the tab. Open Infrastructure > Clusters > your HVM cluster > HVM Alert, enter the mail recipients and click Save. Saving needs the role permission Clusters: Full. Tick Use these settings for new HVM clusters of this tenant to watch every other cluster with the same settings, including clusters you create later.

The four thresholds start at HPE's own defaults, which are also the lowest values the tab accepts: agent silent 2 and 5 minutes, node unreachable 60 seconds, DLM fencing wait 30 seconds.

5. Check that it runs. Within about four minutes the recipients should get a "monitor started" mail that lists any open problems. Under Administration > Health > Logs, filtered on hvm-cluster-alert, there should be one cycle on <manager> line every two minutes. If the log lines are there but no mail arrives, check the mail settings and the restart in step 2 before you rely on the plugin.

What a host loss looks like

To see the whole path on the VME cluster, I hard-rebooted one host with a sysrq reboot, so it went down without a clean shutdown. These are the times from that run:

During that window the cluster page showed the host as unreachable in the quorum panel:

The HVM Alert tab lists the same open problems for as long as they last:

The mails carry the problem, the host, the detail behind it and the time it started. Agent log events arrive with the original log line:

On the layout 2.0 cluster I powered one host off instead. The first mail came 2 minutes 42 seconds later, with the surviving host's failover lines from its agent log. The second came at 4:47 and carried the Host Unreachable alarm from VME, the agent silent for 5 minutes and the unknown power state. After I powered the host back on, the alarm cleared in a little over a minute and everything else in about three.

If you acknowledge an HVM alarm in VME while it is open, the tab keeps listing it, marked "(acknowledged)", until it clears.

In a separate test I cut one host's storage paths. All paths down and the GFS2 withdraw came in one mail on the first check, a minute and a half to two minutes after the cut. The agent's own lines about its heartbeat budget and about rebooting the host arrive only after the host has rebooted itself, about three minutes after the cut.

Before you run it in production

If the manager VM runs on the cluster it watches, it cannot mail the loss of its own host until it is running again. The "monitor started" mail after a restart includes the agent log events from the 15 minutes before it, so the failure still shows up, only later. Keep the manager on other hardware if you need the mail during that window.

The agent runs the host script through sudo, so every check adds about 3.5 KB per host to auth.log. That comes to roughly 2.5 MB a day. It matters mainly if auth.log is forwarded to a SIEM.

If an upload ever ends with "Failed to parse plugin file", delete that jar from /var/opt/morpheus/morpheus-ui/plugins/ on the manager and run morpheus-ctl restart morpheus-ui. Otherwise the failed jar is loaded again at the next restart next to the working one.

Anyone who can open a cluster's page can also see its mail recipients on the tab.

On a manager with several app nodes, only one node runs the checks and sends the mails. The list of open problems on the tab comes from that node, so a page served by another node shows "No check result yet". In my lab the browser session stayed on one node, so I have not seen this happen there, but I would expect it behind a load balancer without sticky sessions.

A few things I have not tested yet:

• a storage cut while VMs are running on the host;
• the quorum, DLM and GFS2 checks on layout 2.0;
• stretch clusters;
• clusters in a sub-tenant;
• anything larger than three hosts.

Results from any of those setups would be welcome as GitHub issues.

Upgrade and removal

To upgrade, upload the new jar the same way. The new version takes over within one to three minutes, and the log shows one scheduler stopped and scheduler started pair, then a cycle on line every two minutes.

To remove the plugin:

1. Delete it under Administration > Plugins.
2. If hvm-cluster-alert lines still appear in the log, run morpheus-ctl restart morpheus-ui.
3. In Tools > Cypher, delete the keys under secret/hvm-cluster-alert/ in every tenant that saved settings.

Checklist

1. shasum -c prints OK for the jar.
2. Email settings are filled in, and morpheus-ui was restarted after any change to them.
3. HVM Cluster Alert 1.19.0 shows in the plugin list.
4. Recipients are saved on the HVM Alert tab, with the new-cluster box ticked if every cluster should be watched.
5. The "monitor started" mail arrived and the log shows a cycle on line every two minutes.