Troubleshooting the red (Suspicious) Safety Tip for fraud detection checks

Introduction

It has now been about 8 months since we released our antispoofing protection in Office 365, a feature that defends against Business Email Compromise, where the From and To domains are the same. You can read more about that feature at http://aka.ms/AntispoofingInOffice365. To summarize, it defends against others spoofing your domain in the From: address – the one that appears in your email client – by figuring out if the sender is legitimate or malicious. It’s similar to how DMARC works, the key difference is that while DMARC looks for a DNS record published at _dmarc.<domain>, the antispoofing protection interpolates if it does not exist. That is, if the domain in the From: address does not publish a DMARC record, what would it say if it did?

It has also been a few months since we first started rolling out Safety Tips to Office 365 customers (everyone will get them by mid-November). Since that time, the question that has arisen the most is this: Why am I getting a red Safety Tip in my email? This is a legitimate sender!

.

Figuring out the base conditions of why it was stamped

There are 2 common reasons why a “legitimate” message gets a red Safety Tip about a message failing fraud detection checks when the From: domain and To: domain are the same, and 1 less common reason:
.

1. The sender is sending emails from an unauthorized source that is sending as your domain in the SMTP MAIL FROM, but not in your domain’s SPF record, and is also not signing the emails with DKIM
   This occurs when a department within your organization spins up a mail server, or outsources to a Software-as-a-Service provider, who sends email as your domain but they are not in your SPF record. To determine this, open up the message headers and look at the Authentication-Results header. Suppose the service that is sending emails as you (or in the example below, contoso.com) is HRnotifications.com and they are sending from the IP 1.2.3.4:

   Authentication-Results: spf=fail (sender IP is 1.2.3.4)
     smtp.mailfrom=contoso.com; dkim=none (message not signed)
     header.d=none; dmarc=none action=none
     header.from=contoso.com

   In this case, the SMTP MAIL FROM is contoso.com, and the From: domain is also contoso.com. Contoso.com also does not publish a DMARC record. A case like this will almost always get a red Safety Tip and mark as spam because even though a DMARC record is not published, it would have failed had it been.

   I say “almost always” because Office 365 does suppress the Safety Tip and mark as spam action if we have enough reputation data to determine that this is a valid message. It works most of the time, but not for small senders, nor for senders with insufficient reputation, or senders with poor reputation.

   Even if your domain does not have an SPF record, the antispoofing check will still apply.

   Authentication-Results: spf=none (sender IP is 1.2.3.4)
     smtp.mailfrom=contoso.com; dkim=none (message not signed)
     header.d=none; dmarc=none action=none
     header.from=example.com

   .

2. The sender is sending emails that pass SPF or DKIM, but neither the domain that passes SPF nor the domain the passes DKIM aligns with the From: domain
   Just passing SPF or DKIM is insufficient to suppress the Safety Tip or mark as spam. For example:

   Authentication-Results: spf=pass (sender IP is 1.2.3.4)
     smtp.mailfrom=HRnotifications.com; dkim=pass (signature was valid)
     header.d=HRnotifications.com; dmarc=none action=none
     header.from=example.com

   Even though this passed SPF and DKIM, HRnotifications.com != example.com. One of the SPF-passing or DKIM-signing domains must align with example.com. If it doesn’t, a Safety Tip will be inserted unless Office 365 determines that the sending IP or domain has enough reputation to suppress the Safety Tip.
   .

3. [In process of being fixed] The sender publishes a DMARC record, but Office 365 had a DNS lookup failure and DMARC TempError’ed
   A corner case we are working on fixing is when a message ought to have passed DMARC, but didn’t because of a DNS lookup failure:

   Authentication-Results: spf=pass (sender IP is 1.2.3.4)
     smtp.mailfrom=HRnotifications.com; dkim=pass (signature was valid)
     header.d=notifications.example.com; dmarc=temperror action=none
     header.from=example.com

   In this case, DMARC should have passed by aligning on the DKIM domain with the From: domain, but it couldn’t because of the DNS lookup failure. A case like this may fail the antispoofing check and insert the red Safety Tip.

   We are working on this to treat it as a DMARC BestGuessPass and supersede the TempError.
   .

Common scenarios when this occurs

When do the first two occur most often?

The IETF has published RFC 7960 – Interoperability Issues between DMARC and Indirection Email Flows. This document describes all the times when a valid message fails DMARC.

Since Office 365’s Exact-Domain antispoofing check is similar to DMARC, it is subject to some of the same problems. The difference between regular DMARC and Office 365’s antispoofing checks is that our checks will try to figure out automatically that the sender, though failing authentication, is legitimate so you don’t have to do anything; whereas DMARC makes you explicitly configure things to make it work. Office 365 doesn’t always figure it out automatically so the more you configure it, the lower the chances that legitimate email will get marked as spam.

RFC 7960 was written by some smart people, and have done a great job to inventory as many different scenarios that are possible. The ones that I see the most often are the following, along with their solutions (I discuss this more in http://aka.ms/AntispoofingInOffice365):
.

1. A third party is sending email, on your domain’s behalf, but is not in your SPF record or DKIM-signing as your domain
   This constitutes the majority of cases. Even we here at Microsoft see this all the time. Not a week goes by where someone doesn’t ping me asking why they are getting the red Safety Tip. It’s because Microsoft has clamped down on spoofing because the problem of exact-domain spear phishing is so prevalent.The fix for this, either:a) Add the sender’s IPs to your SPF record, or add the include:<their domain> to your SPF record. See Set up SPF in Office 365.

   OR

   
   b) Have the sender DKIM-sign as your domain. This requires them giving you to publish a public key at a selector you specify, e.g., myselector, and then publishing the key at myselector._domainkey.<yourdomain>.

   OR
   c) Add the sender’s IPs to your IP Allow list, see Configure the Connection Filter Policy in Office 365.
   OR

   
   d) Create an Exchange Transport Rule (ETR) allow rule:

   – Add the sender’s IPs to an ETR Allow rule
   – Or, if you want to be more secure, add the sender’s IPs + the sender domain to an ETR Allow rule
   – Or, if you want to impress your friends, create an ETR Allow rule that looks for “spf=pass” in the Authentication-Results header and the sending domain in the envelope is the domain sending on your behalf (e.g., HRnotifications.com). Alternatively, you look for “dkim=pass (signature was verified) header.d=<signing domain>” in the Authentication-Results header. You would use this ETR in the case that the sending domain passes SPF or DKIM, but is not the same as your domain (i.e., they are sending on your behalf and authenticating, but not aligning)Both option (c) and option (d) will skip filtering and suppress the Safety Tip. End user Safe Senders will skip filtering but will not suppress the Safety Tip.See Transport rules in Office 365.
   OR

   
   e) Use the Set-PhishFilterPolicy to specify senders who are allowed to send to you on your behalf. See Set-PhishFilterPolicy. This is only available to Advanced Threat Protection customers.
   .

2. You are sending to a mailing list which is modifying and replaying the message back into your environment 

   This occurs with the following routing path:

   Office 365 -> Mailing list (pass SPF, DKIM, DMARC... then modify the message) -> Office 365

   This is one of the hardest ones to solve. Section 4.1.3 of RFC 7960 talks about this. If you are in control of the mailing list, there are a couple of good options you can make to your mail server:

   a) Set it up to not modify the message (not the most popular option)

   b) Rewrite the From: address so that it doesn’t fail alignment

   Suppose the message you send looks like this when sending out:

   From: Example User <example@contoso.com>
   To: Mailing List <list@mailingList.org>

   When it goes to the list and gets replayed, it could be rewritten like this:

   From: Example User via Mailing list <list@mailingList.org>
   To: Example User <example@contoso.com>
   Reply-To: Example User <example@contoso.com>

   Not everyone in the email industry likes this solution, but I think it’s pretty good if the list server is under your control.

   c) Submit it as a false positive to Office 365 and we’ll see if we can add it to our list of “Do not enforce antispoofing checks from these senders” list. This is a manually maintained list and we only add mailing lists to it, and it takes a long time to update. But if the list is not under your control, this is an option.
   .

3. You are forwarding email into and out of Office 365 and modifying the message. Sometimes on purpose, but sometimes not.
   If your email path goes like this:

   Internet -> Office 365 -> on-prem mail server -> Office 365

   If the on-prem mail server modifies the message, it could fail antispoofing. This occurs if you are running Exchange on-prem which modifies messages (see here for details), or you have another appliance that inserts footers into a message or does some other modification.

   If you are running Exchange, the solution for this is to set up connectors so that the headers we stamp on the first pass through Office 365 are respected when you relay it back into the service. If you are not running Exchange, you will have to set up a few ETRs to respect the original spam or non-spam verdicts and route the message accordingly.

Those are the ones I see the most for customers of Office 365.
.

What about disabling Safety Tips?

As a last-ditch attempt to solve the problem, you may be tempted to disable Safety Tips altogether. My advice is this – Don’t do it!

Safety Tips are like a seat belt in your car. If it’s too tight, or sits too high, or scratches you, the solution is to adjust it or buy one of those thingies that adds padding to it. The solution is not to drive without it.

Safety Tips helps protect you against spoofing, but it also helps defend against phishing (another red safety tip), to tell you when a sender is trusted, when you’ve skipped filtering due to IP Allows or Safe Senders, and other types of impersonation (coming soon in 2017). So by turning off Safety Tips, you’ve disabled all that protection.

You are much better off working through the issues about by inventorying IPs, getting proper DKIM signing,  fixing mailing lists, and making sure connectors are set up properly. That ensures you have the best protection possible. It is a little inconvenient (it’s inconvenient here at Microsoft, too) but it is worth the effort.

But if you must turn off Safety Tips, you can do that here: Enable or disable Safety Tips in Office 365.
.

Conclusion

Hopefully this article helps you figure out why a message has that red Safety Tip inserted above. We do our best to make sure we only insert it in cases where we think the email is fraudulent, but sometimes legitimate email gets the tip. Fortunately, you can fix it yourself by following the steps above.

If you have questions, just let me know in the comments.