Grafana OnCall outgoing email notifications
Grafana OnCall is a plugin for Grafana that allows to aggregate, group, and take action on alerts right in the Grafana user interface. It offers powerful functionality, but the open source version only released in June 2022 the documentation is not very detailed, and the community guides are thin on the ground; as such it can be challenging to configure.
Alert groups that are captured by Grafana OnCall can send out notifications via various channels, as configured in the settings, and in the escalation chains. These are email, Slack, telegram, and outgoing webhooks .
To send email you need to have an existing SMTP mail server, and an email user with a password. For the use case I was working on, there wasn't an existing dedicated email address for this, and relaying emails via the existing SMTP server as it was set up proved challenging. It also wasn't important what the alert's outgoing email address or domain was, so the cheapest, reliable, and fastest to set up option was signing up a mailbox.org account  .
Mailbox.org is an email hosting provider and you can get a mailbox for as little as €1 a month, with their standard tier clocking €3 a month, and premium €9 a month. You can see the features in each pricing plan here.
When you sign up for mailbox.org account they will provide you with the mail client configuration settings (in Connect your device section) which look like this:
We can ignore the entire part about the incoming server settings, as we are only using this mailbox to send email. Take note of the outgoing server settings.
This is where things can go terribly wrong, and did for me. Pay particular attention to the port number. We will get to this a little later.
Configuring the server settings in Grafana OnCall
Grafana OnCall fundamental configuration can be seen in the Env Variables section in the User Interface. As an admin you can reveal all the environment variables on that screen, except for the keys and passwords, and change or set new values. It's worth noting that there are three ways any of these values can be set:
- the default value set by Grafana OnCall
- the environment variable set at, or before the plugin start time
- the value set in the Grafana OnCall UI
Environment variables override defaults, and the value set in UI overrides environment variables.
If you set the value in the UI then Grafana OnCall will also provide a button to reset this value to the default. Note that this default will be the Grafana OnCall default only if there is no environment variable value; it will be the environment variable value if one has been set.
You should use environment variables wherever possible.
The following values can set to allow for sending of email :
EMAIL_HOST- SMTP server host
EMAIL_HOST_USER- SMTP server user
EMAIL_HOST_PASSWORD- SMTP server password
587) - SMTP server port
True) - To enable/disable TLS
EMAIL_FROM_ADDRESS(optional) - Email address used to send emails. If not specified,
EMAIL_HOST_USERwill be used.
For the mailbox.org configuration these values would be as follows:
EMAIL_HOST_USER- your email address
EMAIL_HOST_PASSWORD- your account password
EMAIL_FROM_ADDRESS- your email address
With the first four values being the minimum required, as the
EMAIL_USE_TLS is already enabled by default, and
EMAIL_FROM_ADDRESS being optional, and defaulting to the
- You should set
EMAIL_FROM_ADDRESSif your email provider login is not the full email address (for example just the local part before @ and domain) or if you are using an alias.
As of November 2022 (PR#759) Grafana OnCall also supports unauthenticated SMTP relays. This can be useful if your entire configuration is on an internal network. For such situations the minimal configuration required is
EMAIL_HOST_PASSWORDshould remain blank.
Here is where things can go terribly wrong.
While mailbox.org does accept emails on port 465, and Grafana will send emails to that port without displaying or logging an error, they will never arrive.
Given that as a user you will not have access to the mailbox.org SMTP server logs, and neither did I, I can't confirm what precise type of failure occurred, however I can highlight several things:
- in one yet unresolved GitHub issue there's a mention of port 465 no longer being supported
- IANA (Internet Assigned Numbers Authority) has reassigned this port to other services, with port 587 being used for message delivery
- unsetting the port number to default to port 587 solved the email delivery issue, therefore mailbox.org does allow message delivery on that port as an alternative to 465 - and it's likely they have failed to update their UI and/or documentation to indicate the port change
If you are having a similar issue, I recommend to use 587 as a part of your troubleshooting process and reach out to your email service provider if they don't accept emails on that port - they're messing up.
Testing your setup
Once all your environment variables are set correctly and they're displaying in the Env Var section in your Grafana OnCall, you'll need to test the setup.
- Navigate to the Users section and find a user you wish to send a notification to. Select Edit and set Default Notifications channel to Email. Ensure an email address is visible on the left hand side panel
- Navigate to the Escalation Channels section and ensure you have an Escalation Channel with a Default notification to the aforementioned user. If no Escalation Channels exist, create one
- Navigate to the Integrations section and ensure that you have at least one integration. Ensure that it has the aforementioned Escalation Chain assigned to the integration
- Click on "Send demo alert" button
- Navigate to the Alert groups and open your demo alert. On the right hand-side in Timeline you will see Grafana OnCall proceed through the Escalation Chain and attempt to send an email to the selected user
- Observe the destination mailbox and see if the email arrives
If you did everything right, you should see the following message in the timeline:
sent email message to user.name
and an email should arrive in that user's mailbox.
1. Grafana cloud offering also integrates with other services, but I'm only going to talk about the open source version.
2. I'm not afiliated with mailbox.org, and this is not an advert or a particularly strong recommendation. It's just what I happened to use for this job. Other email hosting providers are available. YMMV.
3. This is per Grafana OnCall documentation.