Inbound Email in Rails with Mailgun – Part One

One of the features I implemented recently at work was allowing users to create “cards” inside their workspace by sending an email to the workspace itself.  While not a revolutionary thing to do, it is a step towards giving users the ability to forward confirmation emails to their workspace and have it auto-generate data based on the email contents.

Some of this will be adapted from Mailgun’s documentation in their Quickstart Guide for Receiving Email.  Mailgun’s API documentation is pretty good at covering how to get started, so feel free to hop on over to their site if you need to dive a little deeper on any part of this initial setup.

Part One will cover getting your DNS and Mailgun account setup.  Part Two will go over setting up the endpoint in your Rails application to accept the forwarded message, validate the message, and then act upon it.

Getting Started – Mailgun Account and DNS Setup

The first place to get started with the process is to sign-up for an account on Mailgun.  Mailgun offers a variety of pricing tiers but you can sign-up for free for the first 10,000 emails per month.  You don’t have to use Mailgun, it was just the solution we already had in place for handling our outbound emails for account changes so it made sense to keep it all in one service.

If you want to explore other options, here are a few of Mailgun’s competitors:

I personally have no ties to any of the above companies, so your mileage may vary on how effective they may be or how easy they are to setup compared to Mailgun.

Once you have your account setup with Mailgun, you will be able to go to the Domains tab
mailgun domains tab

and then from there the Add Domain form itself.

mailgun_add_domain_form

Your domain name will largely depend on the application you’re building and what sort of UX/UI you have in mind.  For our purposes we went with the following domain conventions:

inbox.company_name.com

inbox.staging.company_name.com

After you’ve entered in the domain you’ll be taken to success page with some information on how to configure your DNS records.  Since I am only covering setting up inbound email in this post let’s skip down to the bottom of that page for the “optional” receiving MX records.

mailgun_add_domain_success

If you don’t know what an MX record is, don’t worry!  While I can’t cover the entirety of MX records here, you can take a peak at the Wikipedia entry for MX records or just think of them like a zip/postal code for the post office.  It doesn’t list exactly where your mailbox is, but it gets it to the right area for the local post office to handle from there.

Now if you’re using another service to handle your domain’s normal email (Gmail, etc.) then you’ll want to setup MX records using a sub domain like the suggested mg.company_domain.com.  However since we already know we want our domains to follow a certain format that we setup earlier (the inbox and inbox.staging) we can create entries for those instead that would mirror these:

Name:  inbox.company_name.com   Value:  10 mxa.mailgun.org , 10 mxb.mailgun.org

Name:  inbox.staging.company_name.com   Value:  10 mxa.mailgun.org , 10 mxb.mailgun.org

At work we currently use Ansible to handle our production and staging setup changes on Amazon Web Services (Route 53), but since you may be using a different setup you’ll have to consult your hosting service for more information if you aren’t sure where to set this up at for your site/application.

***Important Note:  DNS record changes can take upwards of 24-48 hours to fully complete and propagate (spread).  During this time you will likely not be able to reliably test the inbound mail system, so keep that in consideration if you try to do this all in one sitting.***

Once you have the MX records setup with your hosting service you will need to head back to Mailgun’s site.  If you take a look at the Domains tab you’ll see that your new domain(s) that we created will show as unverified.  When you setup a new domain for receiving email this verification typically completes during that 24-48 hour window once the DNS records are fully propagated.  However, in this case the verification will not complete because we’re only setting up the receiving side of things.

If you just can’t stand to see that icon saying unverified you can set up the DNS records for sending mail but you’ll have to reference Mailgun’s documentation on that for those instructions.

Congratulations you’re <insert arbitrary fraction here> of the way there!

Mailgun Routes

Unlike DNS records, Mailgun Routes are less of a hassle to wrap your head around.  What routes allow you to do is set up special rules to filter what happens when an email is sent to a domain tied to your Mailgun account.

To get started, head to the Routes tab once you’re logged into Mailgun.

mailgun_routes_tab

The Routes screen allows you to:

  • See any routes you currently have defined or create new routes…

mailgun routes show

  • Test route filter expressions and message formatting (we’ll cover those in a bit)…

mailgun routes test and post actions

When creating a new route you’re given several options:  Priority, Filter Expression, Actions, and Description.

Priority tells Mailgun which routes should be checked/applied first.  The Priority values are evaluated from lowest numeric value to the highest, meaning a Priority of 0 will be checked before a Priority of 10.

mailgun new route priority

The default priority is zero (0) when creating a new route, so we can just leave that setting as is for now.

Filter Expressions are the rules that tell Mailgun which things to take actions on.  The available options are catch_all(), match_recipient(""), and match_header("", "").

mailgun new route filter expressions

Catch_all, as you might expect, just tells Mailgun to assume any incoming message should be acted upon.  Match_recipient instructs the system to check the value provided in-between the quotation marks inside the parentheses and if the intended recipient of the email matches perform an action.  Match_header deals with checking certain aspects of the header.  The header of an email contains a lot of information about the email but for now let’s ignore this option for the sake of this post.

If you want more information on email headers, I like to direct people over to Media Temple’s community article.  Enjoy!

Actions are the things you want Mailgun to do to/with the messages the filter expressions match with.  Like Filter Expressions, there are three options but I will just be focusing on the first two options for this post.

mailgun new route actions

The first option, forward(""), has Mailgun send the email to either another email address or to a specified URL endpoint.  This ability to forward the message on to an endpoint will be integral to our inbound email feature.

The next option available is stop(), which causes the system to stop and not look at any other lower priority routes.  By default Mailgun is supposed* to continue processing and routes you’ve created even if the email matches and is acted upon by a higher priority route.  By using the stop() action you prevent Mailgun from continuing on processing other routes for that email.

*Now there is a slight difference between what the Mailgun User Manual says should happen by default and what I have actually witnessed.  Currently I have no stop() actions in any of my routes and a catch_all() set to a Priority of one while my match_recipient filters are set to a Priority of zero.  Even if I set all the Priorities to zero, if either of the match_recipient() filters are met the email is never evaluated (from what I can tell) at the catch_all() level.  No email is sent via a forward() action nor stored via the store() action.  I’m waiting to hear back from Mailgun Support on this one.

The final option available for actions is store(notify="") which temporarily stores the email at Mailgun and then provides a notification webhook.  Since we will not be using this option today you will need to reference the Mailgun User Manual for further information.

So for our needs we’re going to create three routes:  one for the production inbox, one for the staging inbox, and then a catch_all for any odd emails that get sent to the domains.

Production Inbox (inbox.company_name.com)

mailgun production route

Here we have the Priority set to zero as this is one of our main routes and should be evaluated first.  In the Filter Expression section you’ll see a match_recipient("") filter with a little bit of potentially odd looking formatting of the email domain.  What you’re seeing is a Python regular expression (regex for short) that tells Mailgun to match any recipient email address that ends in @inbox.company_name.com.  Since we want to catch-all emails for that domain and then handle the details of the exact recipient in the application, matching on the domain is the best way to handle it.  In the Actions you’ll see the expected forwarding of the email to the application and then the stop() action to prevent the system from trying to match other routes.  Right now we have not determined what our endpoint will be in our application, so I’m using a placeholder value of /endpoint.

Staging Inbox (inbox.staging.company_name.com)

mailgun staging route

Here you will see the exact same formatting as with the Production Inbox route with the only difference being the inclusion of the staging addresses for the domain and application.

Catch_All Route

mailgun catchall route

This route is designed to be lower priority than either of the other two routes and serves as a way to alert us if emails are coming in but not matching either other filter.  This should not happen since we are matching the email domains of production and staging at a higher priority (our only two domains), but it is always a good idea to have a backup to catch things that might slip through the cracks.  The forward(“”) action is set up to alert a placeholder email address which will likely be an email list that is monitored by support or other engineering staff.

Summary

As you can see, the process of getting inbound email setup through Mailgun isn’t necessarily difficult but there are a few options along the way that might trip you up if you aren’t prepared.  We went from briefly covering the initial Mailgun account setup to DNS settings (remember that 24-48 hour propagation window!) and finally to a few options in route configuration.

In Part Two I’ll cover how to set up the Rails endpoint, how we setup different “mailboxes” for each workspace, and some of the logic we use to verify the workspace and user when someone sends in an email.

As always if you have any questions or comments please share them and I will answer back as soon as possible!

 

3 Replies to “Inbound Email in Rails with Mailgun – Part One”

      1. My apologies for the delay in response! I’ll be finishing up and then adding more content related to this very soon. Let me know if you have any specific questions I can help answer.

Leave a Reply

Your email address will not be published. Required fields are marked *