Grafana OnCall outgoing email notifications

Grafana OnCall is a plugin for Grafana that allows to aggregate, group, and take action on alerts right in the Grafana user interface. It offers powerful functionality, but the open source version only released in June 2022 the documentation is not very detailed, and the community guides are thin on the ground; as such it can be challenging to configure.

Alert groups that are captured by Grafana OnCall can send out notifications via various channels, as configured in the settings, and in the escalation chains. These are email, Slack, telegram, and outgoing webhooks ^[1].

To send email you need to have an existing SMTP mail server, and an email user with a password. For the use case I was working on, there wasn't an existing dedicated email address for this, and relaying emails via the existing SMTP server as it was set up proved challenging. It also wasn't important what the alert's outgoing email address or domain was, so the cheapest, reliable, and fastest to set up option was signing up a mailbox.org account ^[2] .

Mailbox.org is an email hosting provider and you can get a mailbox for as little as €1 a month, with their standard tier clocking €3 a month, and premium €9 a month. You can see the features in each pricing plan here.

When you sign up for mailbox.org account they will provide you with the mail client configuration settings (in Connect your device section) which look like this:

We can ignore the entire part about the incoming server settings, as we are only using this mailbox to send email. Take note of the outgoing server settings.

This is where things can go terribly wrong, and did for me. Pay particular attention to the port number. We will get to this a little later.

Configuring the server settings in Grafana OnCall

Grafana OnCall fundamental configuration can be seen in the Env Variables section in the User Interface. As an admin you can reveal all the environment variables on that screen, except for the keys and passwords, and change or set new values. It's worth noting that there are three ways any of these values can be set:

• the default value set by Grafana OnCall
• the environment variable set at, or before the plugin start time
• the value set in the Grafana OnCall UI

Environment variables override defaults, and the value set in UI overrides environment variables.

The following values can set to allow for sending of email ^[3]:

• EMAIL_HOST - SMTP server host
• EMAIL_HOST_USER - SMTP server user
• EMAIL_HOST_PASSWORD - SMTP server password
• EMAIL_PORT (default is 587) - SMTP server port
• EMAIL_USE_TLS (default is True) - To enable/disable TLS
• EMAIL_FROM_ADDRESS (optional) - Email address used to send emails. If not specified, EMAIL_HOST_USER will be used.

For the mailbox.org configuration these values would be as follows:

• EMAIL_HOST - smtp.mailbox.org
• EMAIL_HOST_USER - your email address
• EMAIL_HOST_PASSWORD - your account password
• EMAIL_PORT - 465
• EMAIL_USE_TLS  - True
• EMAIL_FROM_ADDRESS - your email address

With the first four values being the minimum required, as the EMAIL_USE_TLS is already enabled by default, and EMAIL_FROM_ADDRESS being optional, and defaulting to the EMAIL_HOST_USER value.

• You should set EMAIL_FROM_ADDRESS if your email provider login is not the full email address (for example just the local part before @ and domain) or if you are using an alias.

Here is where things can go terribly wrong.

While mailbox.org does accept emails on port 465, and Grafana will send emails to that port without displaying or logging an error, they will never arrive.

Given that as a user you will not have access to the mailbox.org SMTP server logs, and neither did I, I can't confirm what precise type of failure occurred, however I can highlight several things:

• in one yet unresolved GitHub issue there's a mention of port 465 no longer being supported
• IANA (Internet Assigned Numbers Authority) has reassigned this port to other services, with port 587 being used for message delivery
• unsetting the port number to default to port 587 solved the email delivery issue, therefore mailbox.org does allow message delivery on that port as an alternative to 465 - and it's likely they have failed to update their UI and/or documentation to indicate the port change

If you are having a similar issue, I recommend to use 587 as a part of your troubleshooting process and reach out to your email service provider if they don't accept emails on that port - they're messing up.

Testing your setup

Once all your environment variables are set correctly and they're displaying in the Env Var section in your Grafana OnCall, you'll need to test the setup.

1. Navigate to the Users section and find a user you wish to send a notification to. Select Edit and set Default Notifications channel to Email. Ensure an email address is visible on the left hand side panel
2. Navigate to the Escalation Channels section and ensure you have an Escalation Channel with a Default notification to the aforementioned user. If no Escalation Channels exist, create one
3. Navigate to the Integrations section and ensure that you have at least one integration. Ensure that it has the aforementioned Escalation Chain assigned to the integration
4. Click on "Send demo alert" button
5. Navigate to the Alert groups and open your demo alert. On the right hand-side in Timeline you will see Grafana OnCall proceed through the Escalation Chain and attempt to send an email to the selected user
6. Observe the destination mailbox and see if the email arrives

If you did everything right, you should see the following message in the timeline:

sent email message to user.name

and an email should arrive in that user's mailbox.

1. Grafana cloud offering also integrates with other services, but I'm only going to talk about the open source version.
2. I'm not afiliated with mailbox.org, and this is not an advert or a particularly strong recommendation. It's just what I happened to use for this job. Other email hosting providers are available. YMMV.
3. This is per Grafana OnCall documentation.