Over the past month, my team and I have been going over logs in our system, looking for SPF PermErrors and trying to figure out how many we had, and the root cause of them. As it turns out, there are lots of things that cause a permanent SPF failure. The most common examples are going over the 10 DNS lookup limit because a domain owner adds too many 3rd parties to their SPF record. Another reason is creating two SPF records (two TXT records in DNS) instead of putting them in as a nested include. However, there are tons of other errors like:

• Spaces after the ip4, e.g., ip4: 1.2.3.4
• No "4", e.g., ip:1.2.3.4
• Forgetting to include a policy, e.g., no -all, ~all, or ?all
• Putting quote marks around everything, e.g., "v=spf1" "ip4:1.2.3.4" "-all"
• And a whole bunch more

But the weirdest one we discovered was the case of a non-existent nested include. This is when a domain owner adds an include mechanism, but the domain does not exist. For example:

contoso.com IN TXT "v=spf1 ip4:1.2.3.4 include:_spf.fabrikam.com ~all"

When an email comes in from an IP, say 5.6.7.8, the email receiver looks up the SPF record for contoso.com. It sees that that 1.2.3.4 is not a match, and then sees the directive to lookup _spf.fabrikam.com.

What's supposed to happen is the email receiver does an SPF lookup on _spf.fabrikam.com, and follow that chain. But suppose that the TXT record doesn't exist, it returns NXDOMAIN. As it turns out, we were stamping that as a PermError. In other words, a non-existent include was being treated a syntax error, instead of being treated as a no-op.

"What?" we said. "A PermError?" That should be treated as a non-operation, and burn a DNS lookup limit (count once against the limit of 10) and assume that the IP could not be found in there. "It must be a bug in our code" we concluded.

As it turns out, it's not a bug in our code, it's how the specification works. From OpenSPF's syntax documentation:

The "include" mechanism

include:<domain>

The specified domain is searched for a match. If the lookup does not return a match or an error, processing proceeds to the next directive. Warning: If the domain does not have a valid SPF record, the result is a permanent error.

I disagree with the bolded part above. Rather than being a permanent error, it should be a non-match and the SPF check should continue.

In other words, the result of 5.6.7.8 for example.com against this:

contoso.com IN TXT "v=spf1 ip4:1.2.3.4 include:_spf.fabrikam.com ~all"

...should be a soft fail, and not a PermError.

Why?

For a few reasons:

1. Downgrading to a PermError introduces ambiguity when it could give a more authoritative result
   .
   The domain owner of example.com is taking a dependency on the domain owner of fabrikam.com. If fabrikam.com ever updates their own SPF record and breaks it, example.com shouldn't be punished. A potential SPF pass should still pass, a potential SPF fail should still fail and not give an otherwise ambiguous result.The breaking of an SPF record where the rest of it is okay should result in the okay parts being respected, and the broken parts disregarded. The one broken part should not break everything else.
   .
2. To avoid punishing the original domain owner
   .
   Look, we get it. It turns out that SPF is hard to get right. The domain owner of example.com is doing everything they can to do the right thing. They shouldn't be punished for making one mistake; instead, the mistake should be contained (kind of like a try/catch to handle exceptions when coding) in order to limit the damage. The maintainers of email filtering software know how to handle simple errors in SPF records better than domain owners understand how to maintain valid SPF syntax. Therefore, email filtering software should try harder.
   .
3. For practicality - apparently domain owners don't notice when they have permanent errors
   .
   If this were only a few problematic domains, I might shrug my shoulders and say "Meh, no big deal. The domain owner should be paying attention to these types of things. Surely they've noticed deliverability challenges due to invalid SPF records."Well, it turns out that there are hundreds, perhaps even thousands, of invalid SPF records. Clearly, domain owners are not paying attention. It's simply more practical to detect the domain owner's intent than to fail delivery because of it and hope they notice, and fix it.

I think this change to SPF would make it work better. While on the one hand it is more forgiving to the owners of SPF records (the ones with access to update DNS) and removes the burden from them, on the other hand it's not easy to get it right. As someone who is well-versed in SPF, and who has readers who are well-versed in SPF, you know what I'm talking about. Software designers need to be able to gracefully handle error conditions, and in Office 365 and Outlook.com, when it comes to DNS records we think it's more important to detect the user's intent (when it comes to email auth) than to pay strict attention to the letter of the law when it comes to record verification.

However, the SPF specification could probably stand to have an update, or perhaps a Best Common Practices for verifiers. Probably everyone who work with SPF has the same stories as me, so we may as well make our software more robust to predictable errors.

It would make stuff work better.

This is a rough description of how the Phishing Confidence Level (PCL) works in Office 365.

Way back in the olden days - 2007 or so - Exchange server used to have its own spam filter, Smartscreen. This was more-or-less the same spam filter running in Outlook.com. But whereas Smartscreen in Outlook.com (then known as Hotmail) updated multiple times per day, the spam filter in Exchange only updated periodically.

Smartscreen on Exchange was designed to communicate with Outlook the email client. When Smartscreen set the Spam Confidence Level (SCL), this was propagated to Outlook who would move messages SCL 5 and higher to the Junk Email folder. The SCL was stamped in the following header:

X-MS-Exchange-Organization-SCL: 5

Smartscreen could also detect if a message was phishing, and if so, it would set the Phishing Confidence Level (PCL) to 8. It would stamp this in the following header:

X-MS-Exchange-Organization-PCL: 8

Or, perhaps this:

X-MS-Exchange-Organization-PCL: Suspicious

That comes from Antispam Stamps, our documentation from Exchange on-prem's filtering (I'm also not totally sure on the format, I rarely work with the phishing filter for on-prem because I'm all about the Office 365 and Outlook.com services).

But it's not enough to store these values in headers, Exchange also stores them in MAPI properties. Not every X-MS-Exchange-Organization-* header corresponds to a MAPI property, but these ones do (MAPI is a database-backed mailstore for Exchange; I've been working with it for years and still don't fully understand it). So, Outlook has access to those headers but also to those MAPI properties.

Whenever Outlook sees PCL 8 in a MAPI property, it displays the following pink bar in the email client:

You may have seen this in Outlook; if you do, it's because PCL 8 has been set.

A couple years ago, we moved away from the native Outlook phishing bar because... well, it's aesthetically unpleasing and hard to update (we have to update the code and then get our users to update their software. That's an uphill battle.) In its place, we moved to Safety Tips. Where's what the new Safety Tip looks like for phishing:

This new-and-improved Safety Tip is only available in Outlook.com and Office 365. It is not available in Exchange on-prem. Doesn't this look way better? The "Learn more" link can be controlled dynamically, that is, we can change where it points to.

The problem is that all the code to show the pink bar still exists in Outlook, which means that if Office 365 (or Outlook.com) ever sees a phishing message and tries to set PCL 8, which is how we've defined phishing previously, that native pink bar will show up in the client. We don't want that, we only want to show the new Safety Tip.

To solve this problem, we decided that we would reserve a new value for phishing: -9990. That way, we would always know that it was Office 365 that identified a message as phishing and not some other service, and Outlook would never show the native pink bar. It allows us to instead show only the newest Safety Tip.

The PCL is supposed to be stamped in two places, the first is the X-Microsoft-Antispam header in the PCL property, and then additionally in the X-MS-Exchange-Organization-PCL header. But it also gets stamped sometimes in the X-Exchange-Antispam-CFA-Test header which is an internal header.

But sometimes a message still gets PCL 8, despite my best efforts to purge PCL 8 from the system.

1. Exchange on-prem still sets the PCL value

If you are running Exchange on-prem in front of Office 365, and it detects a message as phish and sets PCL 8, and you have a set of connectors to connect to Office 365 which propagates message properties forward, Office 365 will inherit and promote the previously existing PCL the same as it does a previously set SCL. In a case like this, PCL 8 will appear in the message headers. However, the X-MS-Exchange-Organization-PCL header will have a bunch of other properties that Office 365 does not use, and that allows us to differentiate between Exchange on-prem and Office 365's antiphishing.

2. Corner cases in our code

Despite my super-best efforts, sometimes messages get PCL 8 because they have a "Mark as phish" rule somewhere in the system that does not have PCL -9990 assigned, and instead gets a default PCL 8. This one drives me crazy. Usually, although not always, when a message is marked as phish in our system, it's because the SFTY value is 9.x, and 9.1 is the most common reason (this property is not documented). But sometimes even in a default case, 9.1 is stamped. Those are difficult to track down. Fortunately, this doesn't happen that much anymore.

The Phishing Confidence Level is a little more complicated than that, because we have machine learning and some other technologies, and I don't quite know how they interact with the SFTY and PCL. I haven't worked on those features and I'm not so familiar with them. Still, this description - as of this writing - covers the majority of PCL and phishing cases that I have seen.

I hope you find this useful.

Background

We sometimes see users creating allow rules, either through Exchange Transport Rules (ETRs), or Domain Allows, or Safe Senders, when they want to receive email from senders. However, they frequently do this insecurely; spammers then spoof the sending domain (or even the full email address) which skips all spam filtering and the message lands in a user's inbox. This causes significant user-dissatisfaction.

Or, on the flip side, organizations sometimes want to receive email from a particular sender (either their own domain, or an external one) and messages go to Junk because it's (legitimately) flagged as spam by our filters. This can occur a couple of ways:

1. The sender is spoofing (usually your own domain) and being marked as spam/phish

This occurs when you have some service, such as an on-premise mail server or a cloud service like AWS or Azure, send automated reports to yourself. These frequently get marked as spam and have the red fraud detection tip on them.

2. The sender is being flagged as spam because of bulk email filtering

It's common to send bulk email, such as marketing or transactional messages, through a bulk email service like MailChimp, SalesForce, SendGrid, SparkPost, SailThru, etc. These bulk email services often have the option for customers to send email through dedicated IPs, but they more frequently have customers send through shared IP pools. These bulk IPs are assigned a Bulk Confidence Level (BCL); because they come from shared IPs, some IPs have a high (more spammy) BCL and therefore get marked as spam. The net result is that the bulk email service you subscribe to can't send email to yourself (for more details, see Bulk Complaint Level values.)

This article describes how to fix this.

The wrong way to allow a sender domain

One of the most common methods to allow a sending domain is through ETRs. However, the most insecure way to do it is to create one that looks at the sending domain, and if it matches, sets SCL -1:

This is the least secure ETR you could possibly write. What happens is the email hits the Transport Rule engine, and then does a string match on the domain in the From: address. If it matches, it skips spam filtering and delivers to the Inbox.

This is insecure because it's trivially easy for a spammer or phisher to put anything they want into the From: address. So while you think you're allowing email from the sender, you're actually allowing email from anyone who puts that particular domain in the From: address. Because of the volume of spear-phishing attacks that is so prevalent today, it's fairly easy to guess whose sending you email legitimately that you might want to allow. Common examples are members of an organization's senior leadership team like the CEO, CFO, etc. Other guessable examples are companies you do business with. A large energy company probably talks to other large energy companies and its suppliers. A large (or medium, or small) technology company probably talks to other large (or medium, or small) technology companies. You maybe even have your own domain allowed so you get all sorts of external reports.

These are all easily guessable by spammers and phishers.

The secure way to allow a domain to send you email

The best way to create an ETR is to do it securely. You should allow a domain + something else:

• Use the sending domain plus authentication status. You can put the sending domain as one condition, and then the Authentication-Results header contains "dmarc=pass" or "dmarc=bestguesspass". This ensures that the sending domain has been authenticated and is not spoofing. The advantage of doing it this way is if you want to add more domains, you can simply add to the list of domains
• Or, if the sending domain does not authenticate, use the sending domain plus a sending IP (or IP range). However, be restrictive in the amount of IPs you add, we recommend going no wider than a /24. If you go wider than that, you may be allowing too many sending infrastructures that is not under control of the sender
• Optionally, add a condition that the message originates from outside the organization (this is implicit, but it's fine to add it as a condition to account for on-premise servers that may not be configured correctly)

The rule actions should be the following:

• Set the SCL to -1 (Bypass spam filtering)
• Add an x-header that says what the rule does. In the example below, I've added a simple header "X-ETR: Bypass spam filtering for authenticated sender contoso.com". If you have more than one domain in this rule, you can change the header text as appropriate.When a message skips filtering due to an ETR, it stamps SFV:SKN in the X-Forefront-Antispam-Report header (if it's on an IP Allow list, it also stamps IPV:CAL). However, there's no way to see if the ETR was secure or not without manually unwinding everything and viewing the ETR syntax. That's why it is important to add the additional header so you can figure what's going on in the event that you ever get a spam message that skipped filtering.

You should only include domains that are under the control of the sending organization, that is, don't add free consumer accounts like @gmail.com, @outlook.com, etc. If you want to add a free consumer account, add the entire email address.

What about Domain Allows, and Sender Allows?

There's another way to bypass filtering, and that's by adding a Domain Allow, or Sender Allow (full email address).

The problem with Domain Allows as currently implemented by Office 365 is that they are simple string matches, similar to insecure ETRs. You should only use them when you want to allow a particular sender that is not easily guessable, e.g., noreply-alerts@contoso.com (although even that is somewhat guessable).

When a message skips filtering due to a Domain Allow or Sender Allow, it stamps SFV:SKA in the X-Forefront-Antispam-Report header.

You shouldn't use these to add your own domain, or when the sending domain has no authentication, or when the sending domain is a free consumer email service. This is because the first two are easily spoofable, and the third one doesn't have enough sign-up roadblocks to prevent phishing.

The good news is that we are fixing the problem of authentication and spoofing with Domain Allows - if a message fails authentication, we will not respect the Domain or Sender Allow and instead send it through regular filtering. Until then, probably the best advice is to avoid this feature to allow sending domains unless you're really, really confident a spammer would not spoof it.

What about Safe Senders?

Safe senders are user-actions that your users can take to skip filtering from senders that are ending up in Junk (or they do it proactively to prevent the filter from ever doing it). Rather than get into the full details, please see How to set up safe senders and blocked senders in Office 365.

It is difficult to prevent your users from using Safe Senders, they are a valuable feature. However, as currently implemented by Office 365, they are vulnerable to spoofing because they are simple string matches. Fortunately, as per above, we are making a change to not respect a user's safe sender if it fails authentication. Our recommendation is for users to add to safe senders when they want to receive email from someone specific.

When a message skips filtering due to a user safe sender, it stamps SFV:SFE in the X-Forefront-Antispam-Report header.

Be aware that it's fairly common for executives in organizations to safe sender each other, and spammers/phishers guess this and send malicious email which skips filtering.

The option "Never block Sender's Domain (@example.com)" is never respected in Office 365. It is a no-operation.

Making sure your domain is secure... or at least, authenticating so you can securely allow it to bypass filtering

Okay, by now you've understood why you should only permit secure domains to skip filtering when sending into your organization (unless you know what you're doing and want to take a calculated risk). The next question is: How do I ensure my domain is secure?

1. First, if you're sending outbound email through Office 365, make sure you've set up SPF and DKIM so that it passes authentication when email originates outside of the service.
   .
   How Office 365 uses Sender Policy Framework (SPF) to prevent spoofing
   Use DKIM to validate outbound email sent from your custom domain in Office 365asdfasdf
   .asdfas.
2. Second, for each service you use that sends outbound email as your domain, make sure you add it either to your SPF record, or set up DKIM so that it signs as your domain.There is lots and lots of documentation on the Internet about how to go about doing this, and the best way to inventory them is to use DMARC (for example, see How we got to enforce DMARC for sub-domains of Microsoft’s largest consumer email brands and How we moved microsoft.com to a p=quarantine DMARC record).
   .
   However, there are a handful of large senders that a lot of large organizations use. Here's a quick list of their documentation on how to set up SPF and/or DKIM on them:
   .
   SalesForce
   SPF - What SPF records do I need to add to my domain for Salesforce?
   DKIM - DKIM Signing Outbound Email, Set up DKIM in SalesForce classic
   .
   SparkPost
   SPF - Sparkpost SPF builder
   DKIM - Sparkpost: DKIM explained
   .
   SendGrid
   SPF - SPF: Sendgrid documentation
   DKIM - DKIM: Sendgrid documentation
   .
   MailChimp
   SPF and DKIM - Set Up Custom Domain Authentication: DKIM and SPF
   .
   SailThru
   SPF and DKIM - Configure DNS records for your domain
   .
   This list is not meant to be exhaustive. If I haven't listed the service here, you can easily find it yourself by doing a search for "Set up SPF <service>" or "Set up DKIM <service>".

By setting up authentication, you can send email to yourself (or anyone else) and have it authenticate as your domain, and then it's safe to add to a domain allow list. However, you must do this for each service you send out of, otherwise if you don't, it will go through the regular spam filtering stack where it may get marked as spam.

Conclusion

We understand the need to sometimes bypass spam filtering; sometimes filters make mistake and legitimate email gets marked as spam. However, you need to do it securely so you don't accidentally allow spam/phish/malware into your organization.

We're making updates to the system so that we do the right thing even without you - as an administrator - having to specifically set it up. However, there will always be a place for admins to make some overrides.

This post should help you do it.

Appendix - IP Allows

There is another way to skip filtering in Office 365 and that's by setting up an IP Allow. There are advantages and disadvantages.

On the plus side, they are easy to add. You just navigate through the UX (or use Powershell), and add a sending IP and it then skips all spam filtering (malware filtering is still applied). No fancy ETR syntax required.

However, one drawback is that they are hard to manage. Whereas it's pretty easy to see what a domain is (e.g., microsoft.com is Microsoft, google.com is Google), it's difficult to look at an IP and see what it is without doing a reverse DNS lookup or a WHOIS lookup. "Just what was I thinking when I added 1.2.3.4 to my IP Allow?"

A second drawback of IP allows is that you don't always know who sends the IP. Is it the one sender who you got a false positive from? Or is it shared IP space (such as a bulk email provider, or even Office 365)? That is, will adding an IP Allow skip filtering from many different senders, most of whom you didn't intend? Before adding it, you should verify whether the IP belongs to a single domain, or many. If you don't know how to do that, use an ETR per above and allow the sending domain securely.

A third drawback of IP allows, related to the second drawback, is that you may add too many IPs. For example, if you add an entire /24 range, you may not know the ownership of that entire range. For many administrators who aren't technical enough to look up WHOIS records and interpret CIDR notation, this may end up permitting more than you thought.

So, exercise caution when using this feature.

When users get spam and phishing messages in the inbox, we ask users to submit them back to us, using the instructions here: Submit spam, non-spam, and phishing scam messages to Microsoft for analysis. I explain why this is important in Why does spam and phishing get through Office 365? And what can be done about it?

But sometimes, we get customer support tickets, or random questions, from customers or other people within Microsoft. Unfortunately, many times these submissions are not that useful. The following are examples that don't help us:

• Forwarding the original spam message (i.e., hitting "Forward" in your email client of the spam message to someone else who asks for a sample)
• Screenshots of a spam message
• Partial headers, and not the full set, in a text file of a message
• Partial headers, and not the full set, from the message header analyzer
• A verbal description with no example (e.g., "I'm having trouble delivering to Microsoft" or "I'm getting lots of spam messages")
• Message traces in a csv or Excel file without the originals attached
• Screenshots showing deliverability going down over time

None of these are helpful. Occasionally for the first one we can do a message trace and find a candidate message, and then go back to the logs and figure out what happened. That works, but it is time-consuming and doesn't scale beyond one or two messages.

All of those can help, but without the original message headers, it limits our ability to investigate.

So, for everyone who has ever asked for help with deliverability to Microsoft (or too much deliverability in the way of spam), here's a quick primer.

Getting message headers in Outlook

Getting message headers in Outlook Web Access (OWA, also referred to as Outlook on the web)

BTW, this also works in Outlook.com's web interface.

Attaching messages using drag-and-drop in Outlook (or attach item) sometimes drops important headers

Sometimes people will do the correct thing and forward a message as an attachment. That is, in Outlook, people will compose a new message and then drag-and-drop the spam message as an attachment. Or, they'll use the "Attach item" feature.

This is supposed to preserve the entire contents of the message, but sometimes it doesn't. Sometimes, for reasons I don't understand, when dragging-and-dropping a message from the Outlook list view into a new message as an attachment, a bunch of headers get stripped from the original message. This appears to be some header firewall issue between certain versions of Outlook and Exchange. You'll never notice when it happens, it's done secretly with no notification to the user.

So, to be safe, you may want to copy/paste the headers of a message into the message body. That's what I do, and it's what some versions of our spam reporting plug-in used to do - to avoid this very problem.

Basic troubleshooting

Sometimes, you can troubleshoot simple missed spam, or false positive, problems yourself. The key header to look for is the X-Forefront-Antispam-Report header, which is documented at Antispam message headers. If it's a user configuration, you can either adjust it yourself or contact an administrator.

Example 1 – Missed spam caused by SCL -1 and reason SKN

X-Forefront-Antispam-Report: CIP:104.47.32.118;IPV:NLI;CTRY:;EFV:NLI;SFV:SKN;SCL:-1;SFS:(These are spam rules);DIR:INB;SFP:;SRVR:CO2PR0801MB677;H:NAM01-SN1-obe.outbound.protection.outlook.com;FPR:;SPF:Pass;PTR:mail-sn1nam01on0118.outbound.protection.outlook.com;A:1;MX:1;LANG:en;

Example 2 – False positive with SCL 6 and Reason SKB

X-Forefront-Antispam-Report: CIP:104.47.32.118;IPV:NLI;CTRY:;EFV:NLI;SFV:SKB;SCL:6;SFS:(These are spam rules);DIR:INB;SFP:;SRVR:CO2PR0801MB677;H:NAM01-SN1-obe.outbound.protection.outlook.com;FPR:;SPF:Pass;PTR:mail-sn1nam01on0118.outbound.protection.outlook.com;A:1;MX:1;LANG:en;

Example 3 – Missed spam SCL -1 and reason IPV:CAL

X-Forefront-Antispam-Report: CIP:104.47.32.118;IPV:CAL;CTRY:;EFV:NLI;SFV:SKN;SCL:-1;SFS:(These are spam rules);DIR:INB;SFP:;SRVR:CO2PR0801MB677;H:NAM01-SN1-obe.outbound.protection.outlook.com;FPR:;SPF:Pass;PTR:mail-sn1nam01on0118.outbound.protection.outlook.com;A:1;MX:1;LANG:en;

Example 4 - Bulk email

To determine if a message is marked as spam because it's bulk email, you need to look at the X-Microsoft-Antispam header. This will contain the BCL value. In Office 365, the default value for BCL blocking is 7 (meaning 7-9 are blocked by default). Some customers will lower the BCL which will be more aggressive in terms of bulk email filtering, but this is usually associated with more false positives (see How to securely add a sender to an allow list in Office 365 for tips on how to avoid this).

X-Forefront-Antispam-Report: ...SFV:SPM; [Message is marked as spam, there is no special code]

X-Microsoft-Antispam: UriScan:;BCL:7;PCL:0;RULEID:(SpamRules);SRVR:BN6PR1101MB2113;

X-CustomSpam: Bulk mail [This is stamped when a message is spam due to bulk email filtering and BCL value is non-default]

Conclusion

This should help provide both you and us with the necessary information to troubleshoot spam problems when you experience them in Office 365. It lets you see if there's some configuration problems on your side, and if there isn't, then getting us data in a usable format reduces friction on side to quickly fix the problem so you can get on with your day.

If you're a user of Office 365 with a hosted mailbox, there may be times when a message ends up in your inbox despite the fact that it was marked as spam. When this occurs, it may be because you have (somehow) disabled junk mail filtering. When this occurs, email is still marked as spam, but instead of going to your Junk Email folder, it lands in your inbox.

How to tell something strange is going on

If something is clearly spam, and you never used to get lots of spam in your inbox, and now you are getting much more than normal, check the message headers. You can get the headers in Outlook Web Access (OWA) or Outlook using the instructions here: When creating support tickets about spam, be sure to include message headers.

Look for the X-Forefront-Antispam-Report header, specifically looking for the SFV property. If you see any of these:

• SFV:SPM (message marked as spam due to spam filter)
• SFV:BLK (message marked as spam because it's on your blocked senders list)
• SFV:SKS (message marked as spam because an Exchange Transport Rule [ETR] marked it as spam)

Then the message should have been delivered to your Junk Email folder.

How to fix this if you are a regular user

There are multiple reasons why this could occur, but the most common is that the setting that moves your spam to Junk Email has been disabled. To fix this: Settings/Options > Mail > Mail > Accounts > Block or allow > Automatically filter junk email

In most cases, this will resolve the problem. If it doesn't you will need to contact your local Help Desk.

How to fix this if you are an administrator

As an administrator, you can also see if the user has mailbox filtering enabled using Powershell. It's pretty easy:

$email= "user@example.com"

$junk = Get-MailboxJunkEmailConfiguration -Identity $email

$junk | fl Enabled

Enabled : False

You can then run Powershell to change this:

Set-MailboxJunkEmailConfiguration -Identity $email -Enabled $true

That will flip the user's setting to move spam to the Junk Email folder.

When would you not want spam moved to the Junk Email folder?

When do you intentionally want to turn this off?

In my personal domain (I get email both to Gmail and to Office 365 for personal email and newsletters), I have Junk Email filtering disabled on purpose. The reason is that I download my email via IMAP, although I also use the Outlook iOS email client as well. When I do that, the email clients don't download my Junk Email folder, and I get legitimate email in there - primarily newsletters (prompting me to write the article How to securely add a sender to an allow list in Office 365).

Here's what I did:

1. I changed my spam filter policy to modify the subject to prepend spam messages with [SPAM]
2. I disabled my junk email filtering on my personal mailbox
3. I created a folder just for Spam, calling it _Spam
4. I wrote an Inbox rule in Office 365 to move messages that are marked as spam (by looking through headers) to the _Spam folder

That way, my email clients will download the spam folder and I can look for false positives.

I don't recommend other people do this unless you're a power user like myself. I have reasons for doing this that the average user just doesn't have (I test our service all the time using myself as a guinea pig). I am not concerned about accidentally clicking on spam, nor on being fooled that a spam/phish/malware is actually legitimate.

But that's one reason why someone might not want to move spam to the Junk Email folder. It's why I do it.

But nobody else I know does. That's okay, too.

Disclaimer

I've been thinking about writing this blog post for a while now.

But I've deferred, thinking "I'm too much of a noob. What value can I possibly add to the discussion?"

But that's the point. When I'm not sure what I think, what I like to do is write things down to better organize my thoughts. And, it helps me remember. Then later on I can come back and see what I thought at a certain point in time, and clarify my points of view.

So when it comes to blockchain and cryptocurrencies, you should beware, readers of this blog. By no means am I an expert on blockchain or cryptocurrencies. Some of the things I say here will be wrong (hopefully not too many). Some of you reading this will be more in-tune with blockchain than I am, and will cringe at the very things I say. Sorry about that.

But I need to write about blockchain. It's the only way I will get better at it.

Introduction

If you've been paying attention to the news, or maybe even if you have, you've no doubt heard about the phenomenal rise of Bitcoin over the past 12 months. You may not be sure what Bitcoin is, or why it's popular, or if it's in a bubble, or if that bubble may pop. You may even have FOMO and are therefore looking to see how you can get some... just in case.

I don't have time on this blog to discuss what Bitcoin is or how blockchain works, except to say that Bitcoin is a cryptocurrency that is built on top of blockchain. If you're not sure what either of those two things are, there's a ton of resources available on the Internet. For example:

• Bitcoin: What is it? (Khan Academy's nine-part series)
• Beginner's Guide: What is Bitcoin? (CoinCentral)
• What is Bitcoin? (CoinDesk)
• Bitcoin on Wikipedia (Wikipedia)
• What is Bitcoin? (Bitcoin.com)

You can also do your own searches. And then for Blockchain:

• What is Blockchain? A step-by-step guide (BlockGeeks)
• What is Blockchain? (Business Insider)
• What is Blockchain? CNBC explains (CNBC; Youtube, 4:53)
• The Blockchain explained simply (TED Talks; Youtube, 15:32)
• Blockchain Tutorial (edureka; Youtube, 3:05)

If you've never heard of Bitcoin or blockchain before, and you've gone and watched those videos or read those articles, then congratulations! You've graduated to having zero-knowledge to being a noob, just like me (on a scale of 1-10, I consider myself a 2.5).

Now that we have that out of the way, let me answer a question you may be having: if this blog is all about security talk, why am I talking about cryptocurrenices and blockchain?

Well, I first heard about Bitcoin in 2012, and since then I've kept loose track of it, watching it go way up and way down over the years. But this past September I listened to a talk by a venture capitalist about blockchain, and after that - for reasons I can't really explain  - I became really excited about it. I don't often get excited about technology, but this did it.

Since that time, I've started to research Bitcoin and blockchain a lot more, really ramping up my efforts over the past 6 weeks which is what has prompted this blog post. And the things I want to talk about is not how it works, but how blockchain will shape the future and whether or not it's worth me investing in a particular cryptocurrency.

This blog post does not contain investment advice, you should do your own research and evaluate your own financial position before making determination.

What's the brou-ha-ha about blockchain and cryptocurrencies?

The cryptocurrency aspect of blockchain is one thing, but to me it's not the most interesting thing. I think what is more alluring is the inversion of the value that protocols bring - from thin protocols with fat clients, to fat protocols with thin clients. These are not my terms, but it's part of what I've read.

If you think about the protocol that defines email, SMTP, it's a very "thin" protocol. It just defines the rules about how to exchange messages over the web. But nobody ever made any money by inventing SMTP. Instead, it's the ISPs, email service providers, and endpoints that have extracted the overwhelming value from email. SMTP sits on top of TCP/IP, and nobody made money by defining TCP/IP either.

Indeed, this is how a lot of Internet protocols work - HTTP defines how to exchange data over the web, but it's applications like Google, Facebook, Amazon, and Twitter that have created value out of HTTP, rather than the inventor of HTTP (Tim Berners Lee).

Blockchain inverts that model. Rather than a thin protocol like SMTP or HTTP that only defines the rules of data exchange, Blockchain lets you, as a protocol creator, define the rules and derive a ton of value right in the protocol itself. The application layer is much less important:

The applications are thin, and must compete with each other in order to compete for users and drive value for their creators. But these applications are distinct from the Blockchain protocol itself.

And what speaks to me about this model, especially as a software geek who cares so much more about building cool stuff than driving shareholder value [2], is that the Blockchain is decentralized. Whereas Facebook, Twitter, Google, etc. all own your data and it's hard or impossible to move your data out of those services, the Blockchain works by being replicated on many people's machines, so by definition you really can't centralize your data in any particular service.

Wait, that's not the interesting part to me.

No, what I like is that Blockchains create value by inventing protocols that solve a problem, and as they gain more widespread adoption, the protocol itself becomes more valuable. It means the geeks and nerds who designed it, wrote the code, and made it useful are extracting value out of it. As more blocks are mined by others on the network, the network becomes more valuable in and of itself - the Network Effect.

For example, Ethereum is a platform for creating Smart Contracts. That solves a problem. The creators of Ethereum will themselves have Ethereum tokens which would be useless by themselves; but as Ethereum is used as a platform to solve more and more problems, more and more Ethereum tokens are mined and drives up their value. And the creators of Ethereum - assuming they still have their tokens - see the value of their tokens go up which is the right thing because they invented it. You are rewarded for driving value, rather than needing a corporation to come in and drive value.

Corporations, of course, can do that to. And they should; and, they will derive value by building applications on top of the Blockchain. But the creators of the Blockchain are also rewarded for creating the protocol upon which many things are built.

This is where I differ from a blockchain being a "fat protocol, thin client". Whereas the reverse is true for the current web, I think for Blockchain it will need to be "fat protocol, fat or thin client" in order to drive value both for the protocol and for the application layer, otherwise no one will build applications. Or, if they do, they may have margins too thin to incentivize people to move to blockchain.

Categorizing the various types of cryptocurrencies

In doing my research, I've mentally divided cryptocurrencies into two different types:

1. True cryptocurrencies
   .
   These are designed to act like money, and the most obvious example is Bitcoin [3]. More examples are altcoins (other cryptocurrencies that are not Bitcoin) like DashCoin, LiteCoin, and Monero.
   .
2. Cryptocurrencies that aren't trying to be currencies
   .
   These are platforms upon which you can build apps, or blockchains that solve other problems. You can mine them, or you can speculate on them. Examples are Ethereum, Ripple, Cardano, and NEO.It's a misnomer to call them a cryptocurrency because they are not trying to function as a currency. However, you can buy some of them with US dollars, and others can only be exchanged from another cryptocurrency (i.e., you can't buy them [yet] with US dollars). So in that regard, the fact that you can buy them (and speculate on them) makes them feel like a cryptocurrency, but their main purpose is to act as a protocol that can be used to solve problems.Even among these cryptocurrencies, there are some that are more advanced than others. I'm still working my way through them, it's a tough slog figuring out which ones are good and which ones will bust.

I'm not sure that this is the correct way to think about cryptocurrencies. There are coins and then there are tokens. I'm not even sure what those are yet.

Thus, when it comes to distinguishing the hype around blockchain and cryptocurrencies, there are several questions to ask:

- Is it trying to solve a problem? In Bitcoin's case, yes. It's trying to be a decentralized currency that is resistant to manipulation by central banks, and it is trying to act as a store of value similar to the way that gold does. And unlike gold, it is easier to conduct transactions.

- In the case of Bitcoin, it seems like the only way to "play" it is to speculate in it, that is, buy and pray that it goes up in value at which point you can either sell it back into your local currency to exchange for goods and services, or purchase good or services from a retailer that accepts Bitcoin.

- For other cryptocurrencies (I will call them cryptos 2.0, even though some of them are probably cryptos 3.0), what problem are they trying to solve? Is it a legitimate problem? Or is it likely to be just another one of the tech world's false prophets? Cryptos 2.0 seem to offer opportunities to speculate, or build apps on them. Speculating in these gives a better dopamine rush, but building apps on them is better for the advancement of humanity.

Conclusion

I'm going to end this blog post here even though I have plenty more to say on it. How can I know which cryptocurrencies will go up [4]? How do I hedge my risk? How do I know which blockchain is worthwhile, and which other ones are fluff? What does it take to make one succeed? What resources can I use to better educate myself?

There really is a lot to take in. But truthfully, I haven't been this intrigued by a new technology in a long time. Let's hope it doesn't let me down.

[1] Don't get me wrong. Corporations play an important part of our market-oriented economy. There's no better mechanism that private enterprise to deliver a large percentage of goods and services. But they are not the best at all things, and the mantra that corporations should exist only to drive shareholder value seems very 1980's to me.

[2] I'm probably using the term "token" here incorrectly. Sorry.

[3] In the United States, the IRS disagrees that Bitcoin is a currency and instead treats it like a capital asset. What does that mean? It means it treats it like a stock or bond (actually, more like a stock that doesn't pay dividends). For example, say you have $10 and bought 1 bitcoin (assume 1 bitcoin = $10 USD). The price of bitcoin then goes up to $20 USD per bitcoin. You've doubled your money!

Now suppose you want to buy a popsicle at BitcoinPopsicles.com, and they are going for 1/2 bitcoin each, or $10 USD (they're good popsicles). You may think "Sweet. I've doubled my money in bitcoin, so I'll send over 1/2 a bitcoin which is $10 US, and I'll still have 1/2 bitcoin left over." The way I understand it, that's not what happens for tax purposes.

No, what happens is you first "sell" your bitcoins and realize a capital gain, and then send them over to BitcoinPopsicles. So, you sell 1/2 bitcoin and realize a gain of $10, and then spend that $10 on a popsicle. On your next year's taxes, you must report a capital gain of $10 and pay your normal rate of capital gains taxes. So, if you are in the 25% tax bracket, and your held your bitcoin for 30 days, then your tax liability is $2.50 US.

Thus, when you pay for stuff with bitcoin, you need to realize that it's going to cost you more than you think. You'll want to talk to an accountant. Also, see https://www.irs.gov/pub/irs-drop/n-14-21.pdf

[4] Another disclaimer: as of the time of this writing, I own a little bit of Bitcoin and Ethereum.

Disclaimer - If you haven't read my disclaimer yet, make sure you do so here. TL;DR version - Buyer beware, I am not an expert, I am fumbling my way through this like the rest of you.

Also, I hold a little bit of Bitcoin and Ethereum.

As I've started learning more about blockchain and cryptocurrencies, and wanting to start speculating in some of them, a question naturally arises - how do I know which ones will go up? I don't want to buy something only to see it go to zero. After all, I've read that somewhere between 90% and 99% of cryptocurrencies are, or will turn out to be, worthless. A 1 in 100 chance of being right is a roulette wheel I don't want to spin.

Buying stocks and bonds is hard, too, but at least there you can buy a mutual fund and diversify your risk. With cryptocurrencies, you don't have that luxury. You either have to mine it yourself, or buy it yourself through an exchange and pay all the fees.

So, if I choose to speculate in a cryptocurrency, it better be worthwhile.

The frustrating part is that while I've come up with a method to research, I don't have a good way to determine if it will be successful. Here's what I do.

First, a cryptocurrency has to solve a problem

It can't just be a "Me, too! Me, too!" thing. For example, Bitcoin was invented to act as a decentralized digital currency not controlled by any government (because of tax rules in the United States, I view it more as a store-of-value than as a currency). If there were someone who were copy/paste the Bitcoin code and modify it slightly and give it a name like ByteDollar, I would consider that a "Me, too! Me, too!" cryptocurrency. It doesn't solve a problem beyond what Bitcoin has already solved; because it doesn't fundamentally change the game with respect to Bitcoin, it doesn't seem like a good investment.

That may not be sound logic, but it's the logic I use to simplify my strategy.

Second, I need to classify it

I need to determine if the cryptocurrency is trying to be a currency (store-of-value), or a platform. Examples of currencies are Monero, DashCoin, ZCash, and PIVX. Platforms are Ethereum, Cardano, NEM, and IOTA. I usually refer to both as cryptocurrencies.

I may not understand cryptocurrencies but I do understand diversification of risk. Rather than hold only cryptocurrencies or only platforms, I would prefer to limit my exposure by holding both sets, although not necessarily in equal proportions. Since there is no cryptocurrency mutual fund, that means I have to buy them each individually (or mine them myself).

That may not be sound logic, but it's the logic I use to hedge my risk.

Third, I need a way to evaluate one vs. the other

Where in the world do I even begin? There's well over a thousand cryptocurrencies out there. How can I possibly research even 10% of them? I simply do not have time for that.

I tried searching the web and found some threads on Reddit, Quora, etc. for "good" cryptocurrency investments. There are lots and lots (and lots and lots!) of recommendations about how 'x' is going to explode, or how 'y' is super interesting. Sometimes 'z' is promising, but not as much as 'a'.

How am I supposed to keep track of all that? I am an amateur. I can't do this full time. It's really frustrating to me that I don't have the time or energy to all the research; and even if I did, I can still guess wrong.

I ultimately settled on a strategy that many people reading this will think is a dumb strategy: I go to CoinMarketCap and sort by market capitalization.

I look at the top 10-15 cryptocurrencies and single out those for further research. The reason I sort by market cap is because it means that there is at least some uptake by other people. Other people are expressing confidence in it, which means that cryptocurrency is benefiting from a network effect. I know I'm supposed to think for myself, and this is a follow-the-herd approach. And, I know that there are a lot of other cryptocurrencies that are potential up-and-comers that are buried further down the stack that I will miss out on. I'm not saying that I won't take a look at those.

But this is a starting point. I can keep track of the top 10-15 cryptocurrencies.

Besides which, the whole point of tracking cryptocurrencies is not necessarily to speculate, it's also to see which one (or ones) might be fun to join the community and start developing for. Cryptocurrency's entire value proposition is that it solves a problem with blockchain. Speculating for speculation's sake is not sustainable, nor does it deserve our respect.

There are other places to find new cryptocurrencies to invest in, but when I discover them I head over to CoinMarketCap to see what the market capitalization is. If it's too small, it's risky; if it goes up it can go way up, but without having proven itself to others, I personally need to stay away.

This may not be sound logic because it necessarily means I will miss out on gains on some revolutionary new blockchain products that are just beginning. But, it's the logic I use to turn a huge problem into a manageable one.

Fourth, there's the heavy-lifting of doing the actual research

The first three steps above are a starting point to find something potentially interesting. Next comes the hard part - doing the research.

1. I start by going to the website to read the whitepapers and poke around with the documentation. With any luck, it will be readable in plain English
   .
2. Next, I head over to YouTube to see if there are any videos on it. Hopefully, there's a few
   .
3. Next, I search the web for "what is <cryptocurrency>?" I'll read five to ten articles. When you've read that many, they start saying the same thing. When every new article you read starts adding no value, you know you've probably gotten all you need to know
   .
4. I head back to the website and read up on the founders to see what their background is
   .
5. User hype is often the least genuine feedback because you can't be sure if people are excited about a cryptocurrency because it's revolutionary, or if they are pumping-with-the-intent-of-dumping once the price runs up. It's kind of like how you can't always trust a company's management to tell you the truth when it comes to investing in their organization because they have conflicting interests when talking about it.
   .
6. I don't read the code, though. Some people do and it's a good idea, but I'm not good at reading other people's code

If this all sounds like a lot of work, that's because it is a lot of work. And even after doing all that work, there's still no guarantee that if you speculate in a cryptocurrency that it will go up. Many things can go wrong like there's massive bugs in the code (I doubt I could spot them even if I did read the code); another cryptocurrency comes along that does it better; lack of developer interest; an overly complicated platform that does too much and is hard to get right; government regulation that strangles a particular cryptocurrency; misinterpreting what something does and failing to get involved for the right reasons; buying into a bubble right before it bursts; and I'm sure you can think of others.

I literally came up with all that off the top of my head in the past 3 minutes.

But at least if you do your homework, you can hedge your risk. This may not be the best logic, but it's the logic I use to separate the wheat from the chaff.

Taking the plunge

As you can see, this is the process I use to take a big problem and try to whittle it down to a manageable one. Getting involved in blockchain is not just about speculating, but about trying to use the technology to innovate. There's more than one way to make money in blockchain than just buying them, of course. You could actually build applications. Or devices that use the blockchain to communicate with other devices... securely.

By no means am I an expert. By no means will I be able to predict what will take off and what won't. But I can improve my odds of making a correct guess.

At least it's fun to try.

Disclaimer - If you haven't read my disclaimer yet, make sure you do so here. TL;DR version - Buyer beware, I am not an expert, I am fumbling my way through this like the rest of you.

Also, I hold a little bit of Bitcoin and Ethereum.

The promise certainly sounds good

Blockchain promises us something absolutely amazing - the decentralization of everything:

• Whereas all currencies today are run by central governments, Bitcoin is not controlled by anybody because it is decentralized and therefore nobody can manipulate it
• Whereas platforms are built on private corporations' servers, Ethereum is not controlled by anybody because it is a decentralized platform that nobody can manipulate
• Whereas all of our data is controlled by a handful of big companies, files stored on the blockchain are distributed everywhere so nobody can control it

This promise of decentralization is key to the entire value proposition of blockchain.

But tech has touted the benefits of decentralization before, and it didn't live up to the hype.

The idea behind networks and the network effect is that you have a bunch of nodes that are more-or-less evenly connected, so that everyone can learn from everyone else. This is the grand bargain behind social networks - by sharing information, the best and most correct ideas will bubble to the top while the wrong and worst ideas will fade away. Information wants to be free - and shared - and therefore reason and logic will win the day because of the free flow of information. Good ideas survive (eventually), bad ideas are disproven (eventually) .

That's what we were promised.

The reality looks way different

But it's not how it turned out. Social networking, far from having a unifying affect on people, has turned out to have the opposite effect. It's made us more divided than ever. And, far from weeding out the bad ideas, it turns out that wrong ideas flourish in social networks.

The reason is (partly) because in a social network, nodes are not evenly distributed where information flows freely amongst everyone. Instead, it turns out that there are bottlenecks everywhere. Groups are localized among each other, and connected via super-connectors (pictured in blue in the diagram below). We stay by ourselves in our groups and only occasionally interact with the outside world, and it's done via the super-connectors.

This ends up suppressing the free flow of information, and reinforcing our own particular points of view, since information depends upon whether or not they can get through the chokepoints.

The tech industry seems baffled about why this occurred. Wasn't information supposed to be free? Didn't it want to be free? Wasn't free expression and debate supposed to sift out the bad ideas? How could this be? [1]

Yet this was entirely foreseeable. The decentralization of information has occurred before, the Internet is not the first time. The most obvious example is the Protestant Reformation. Whereas Christian theology used to be centralized in the Catholic church, the Protestant Reformation's rallying cry was that all people could be their own priests, and everyone could interpret holy scripture for themselves. The interpretation of divine revelation was meant to be free, not centralized.

While the Reformation succeeded in reforming the Church, it also caused a lot of divisiveness, wars of religion, and tens of thousands of splinter denominations. That's what happens when you decentralize, anyone can come up with a particular view; and then if you have enough skillz to make a particular outlook popular, you can get people to join along with you and embiggen your group. I suspect Martin Luther did not foresee that people, upon interpreting scripture for themselves, would come up with such different interpretations and would form splinter groups as much as they did.

In the same way, tech did not foresee how the decentralization of information on the Internet would lead to the insulation of people seeking to reinforce their own particular viewpoints and biases, rather than seeking diversity of opinion in a quest for truth. Yes, that exists. I do it, you do it, we all do it. But we also prefer to reinforce our own viewpoints in many areas of our lives.

As a representative of the tech industry who was caught off guard by this, I plead guilty. I don't know whether or not I should have known better back then... but I certainly do now. The results of decentralization can be mixed [2].

What happens when we apply it to new technology?

Which brings me back to Blockchain. As I said earlier, its biggest premise is decentralization. Yet decentralization is a mixed bag. Will it deliver on its promises? Or will it fall flat? Or worse, will the results be mixed such that we become so dependent on the good that we can't roll back the bad, lest we simultaneously rollback the good?

Bitcoin has been a roller coaster since its inception. It's up bigly since it was created but has undergone massive corrections of 30-60% every few months (it recently corrected 45% from its peak this past December 2017, and only took a week to do it). Is this massive volatility a temporary thing until Bitcoin stabilizes as it gets more adoption? Or, is volatility a feature of Bitcoin because of its decentralized nature, where anyone can come in and buy/sell with no financial oversight?

All this volatility is not a good thing; between buyers and sellers, if the medium of exchange is likely to fluctuate wildly, then what you pay is likely to be far different than what you agreed upon. That makes it less likely you will want to use it as a medium of exchange which was one of Bitcoin's original benefits - to act as a decentralized currency (although as I have said before, it behaves more as a store-of-value since the IRS treats the dispersion of your Bitcoins as a taxable event).

Thus, when it comes to using blockchain as a currency, decentralization has the benefits of being a deflationary currency that is resistant to manipulation by central banks, but it's still potentially vulnerable to manipulation by the millions of end users who have their own ideas of what digital currency is, and what it should be, and then shunt it off from its original purpose.

I don't know whether or not that's a good thing.

What about decentralized applications? Those act as a currency (sort of) and as a platform to build stuff on. I don't know about those, maybe they are useful and maybe they aren't. For sure, blockchain promises to be disruptive. There are some like Ripple and IOTA that solve real problems; Ripple is criticized for being a private implementation of a blockchain, which conceptually it isn't but in practice it kind of is. But even if Ripple is pseudo-private, is that a bad thing (because it undermines the concept of blockchain as a decentralized application) or a good thing (since it means it can't be accosted by "do it yourselfers")?

I think that this is one of the fundamental known-unknowns with blockchain. Decentralization may be awesome.

But then again, it may be yet another false prophet.

[1] This blog post is inspired by an article I read in Foreign Affairs magazine (I am a subscriber), see The False Prophecy of Hyperconnection - How to survive in a networked age. It's behind a paywall, though.

[2] For more on the Reformation, see Majority believe Reformation was divisive, but justified

Even though it's not a recommend configuration for our customers (in terms of spam filtering), some customers of Office 365 route their email through a competing spam filtering service in the cloud, or through an on-prem server. That is, the mail flow looks like this:

I've written previously about the problems this can cause, see Hooking up additional spam filters in front of or behind Office 365. However, if you must do it, you may want to ensure that you force all email to go through the 3rd party server. If you hook up a 3rd party server, email can be delivered to your organization in Office 365 through that server or by connecting directly to EOP (Exchange Online Protection). This is not good because our spam filters don't understand that email was sent directly to the service and not through a gateway; if your MX record does not point to Office 365 or EOP, some spam filtering checks are suppressed automatically to avoid false positives.

Therefore, to get the fullest protection possible, I recommend relying upon the 3rd party service, and then maybe or maybe not doing double-filtering in EOP (accepting the fact that there will be false positives and false negatives). But, don't just rely on EOP.

So, to force email through your on-prem server, you will need to install a TLS cert in your on-prem server, and then always ensure that it's used when connecting from the server to Office 365. Then, create a partner connecter using a seldom used attribute which isn’t exposed via the mainstream UX but only through cmdlet. It’s called AssociatedAcceptedDomains.

To do this with TLS-cert-based connectors using cmdlets:

New-InboundConnector
–Name "OnlyAcceptEmailFrom<OnPremServer>"
-ConnectorType Partner
-SenderDomains *
-RestrictDomainsToCertificate $true
-TlsSenderCertificateName <Full set of TLS cert names>
-AssociatedAcceptedDomains <full list of accepted domains that belong to your organization>

For more information on TLS connectors and how to set up a partner connector, see Set up connectors for secure mail flow with a partner organization.

You may have to tweak this a bit to get it right, so you may want to experiment with some smaller domains before enabling it for every domain in your organization.

What this does is reject messages that don't come over the TLS cert; so long as your on-prem server is correctly configured, any email that tries to connect directly to Office 365/EOP should be rejected.

You would want to use this when you are connecting through an on-premise mail server and you can control the certificate. However, if you are connecting through a shared service and cannot specify the TLS-cert, then this would probably not be appropriate without some modifications to the connector.

Hopefully, you find this useful.

Disclaimer - If you haven't read my disclaimer yet, make sure you do so here. TL;DR version - Buyer beware, I am not an expert, I am fumbling my way through this like the rest of you.

Also, I hold a little bit of Bitcoin and Ethereum.

Way back in the fall of 2012, I attend the Virus Bulletin conference in Dallas, TX. While I was there, I remember either attending, or hearing about a session, entitled Malware taking a bit(coin) more than we bargained for.

The presentation was by a researcher at Microsoft, and they talked about how bitcoin was a new digital currency just starting to gain traction. In response, new malware families were arising that would either take over user's computers to mine bitcoin (this was back in the day when a single computer still had a reasonable change of actually mining one), or try to steal users' bitcoins. I think that may have been my first introduction to Bitcoin, and I remember at the time that it was interesting, but wasn't sure whether or not it would catch on as a digital currency. If the malware creators succeeded in mining Bitcoin, they would have seen it go up in value by 100x.

Fast forward several years, to 2017, with the WannaCry malware outbreak. Malware hurts, and ransomware is even more painful as you're locked out of your system, but the market incentive to pay the fine is enticing if you can be certain that it will unlock your system; the drawback is that it incentivizes bad behavior for the malware author.

Both cases are examples of malware creators looking toward alternative payment methods to make themselves less trackable.

But what's interesting is how malware writers have stayed with that principle but have switched out cryptocurrencies. Whereas before they were mining bitcoin, now they are mining Monero:

• Hackers stealing users computing power to mine malware [Sept 2016]
• How fans of TV show 'Homeland' help hackers mine Monero [Sept 2017]
• Android malware miners Monero, can destroy phones [Dec 2017]

These are just a few snippets of articles I found, and you can see they span 15 months. So, while it's a newer thing, it's not totally brand new. But the point is: Hackers are diversifying into alt-coins (an alt-coin is anything that is not a bitcoin).

As I say in some of my other cryptocurrency articles, the value of a digital currency built on blockchain is how many users believe in it, build on top of it, and start using it. Hackers and malware authors were early adopters of Bitcoin, and they seem to be proven right (so far... barring a collapse of Bitcoin). Do they have any special insight into whether or not Monero will eventually be successful?

You can do your own research into what Monero is and how it differs from Bitcoin. My own quick summary is that it's a digital currency like Bitcoin, but it's not built on the Bitcoin code like how a lot of other cryptocurrencies are. And whereas Bitcoin is pseudonymous, all transactions are public. If you observe enough patterns, you can see that randomNumber ID #1 that sends 0.5 btc to randomNumber ID #2 is a transaction. You don't have the identities of everyone yet, but with enough observations you may be able to figure out some of the identities. Bitcoin leaves a trail that is reversible back to its original transaction participants (in some cases, depending upon how many resources the investigator wants to spend).

Monero is different because it is much more private. Instead of this:

A sends xx bitcoins to B

You get this:

? sends ? to ?

You can see that's more private and not trackable.

There are some legitimate use cases of hiding your financial transactions from all viewing eyes. Using regular cash is kind of like this. But on the other hand, one of those use cases is criminal activity; if you're exchanging illegal goods or services, you want that to be hidden from everyone. Thus, if Bitcoin had a reputation as being useful for underground transactions, Monero could market itself the same way. No doubt cyber criminals already do, as that's why they are mining Monero using other people's machines.

I sympathize with the solutions to problems that altcoins are trying to solve. But, by introducing stronger privacy, they also set themselves up as a magnet for criminal activity. The maintainers of the code may say that they are building a platform and are not responsible for its usage. I'm not so sure about that.

Just ask Facebook.

Disclaimer - If you haven't read my disclaimer yet, make sure you do so here. TL;DR version - Buyer beware, I am not an expert, I am fumbling my way through this like the rest of you.

Also, I hold a little bit of Bitcoin, Ethereum, and Cardano.

So, over the past couple of weeks I've been doing a lot of research into cryptocurrency. I've spotted a few more I'd like to speculate on, identified why people complain about (it's slow to verify), different algorithms for verification (Proof-of-work vs. Proof-of-stake), and tried-but-failed to understand new twists on blockchain (side chains).

However, I want to come back to logistics and talk about cryptocurrency's liquidity problem and how to minimize it.

There are four main sources of friction that I see when it comes to trading:

1. Trading fees, which I'll talk about in this post
   .
2. Capital gains taxes, which threatens to be a monstrous problem, which I won't talk about in this post (or possibly ever because it's so complicated and I am not qualified to talk about) [1]
   .
3. Slippage, which is the different between the price you place an order at, and the price your order is filled
   .
4. The bid/ask spread; this is more of an issue when buying stocks where if a stock is trading at $4/share, you would buy at the Ask price which might be $4.02, and sell at the Bid price, which might be $3.98. That's a spread of 4 cents per share, so even if the price of $4 never changed, buying and then immediately selling would still result in a net loss (ignoring trading fees and possible slippage).

If you're going to trade, you must reduce friction because it will eat you alive.

I use Coinbase to buy Bitcoin and Ethereum, and then if I want to buy an altcoin, I have to send Bitcoin or Ethereum to a second exchange and then convert it into that altcoin. This is a multi-step process and it's a pain compared to trading traditional securities.

Through my regular brokerage:

1. I transfer money into my brokerage and it's there the same day, I can usually trade with it right away (or within a minute or two)
   .
2. I search for the stock, bond, or ETF I want to buy, and then I buy it. It's deposited into my account within seconds. I am charged a fee of $7.

From start to finish, this would take me a few minutes with minimal clicks.

For buying cryptocurrency - specifically an altcoin:

1. I can pick multiple methods for purchasing through Coinbase - either by hooking up to my bank account directly, or by hooking up a debit/credit card. Both methods... are not great.

a) If I choose my bank account, the trading fee earlier today was $14, twice the fee of buying stocks. This deducts the money from my account, less the trading fee, and then purchases the equivalent amount of Bitcoin or Ethereum. However, I'm informed that because of the amount of time it takes for the bank transaction to clear, this will take 11 days. 11 days! Are. You. Kidding. Me.

This is going to introduce a great deal of slippage. 11 days in cryptocurrency-land is like 12 months in stocks/bonds/ETFs-land with regard to price stability.

My trading costs are double and the slippage is (potentially) gigantic. But, at least the price I bought my Bitcoin or Ethereum is locked in today, I don't have to wait for the transaction to clear before acquiring them.

b) If I choose my debit/credit account, the transaction is instant and I'll get my Bitcoin or Ethereum right then and there. But, the trading fee is $38 [2], almost three times the cost of doing it by bank account.

Three times the trading fee! Are. You. Kidding. Me.

2. Next, to buy the alt-coin, I have to send the Bitcoin or Ethereum to my second exchange. There's a fee for this, I think it's a flat fee of the cryptocurrency I am trading (or maybe part of the USD... I forget).

I send it over and it takes somewhere between 15 and 30 minutes to transfer over.

3. Finally, I head over to the second exchange and I convert the Ethereum or Bitcoin to the altcoin. There's a trading fee for that, too.

Start to finish, the I've been hit with three trading fees, and it takes either 7-10 days (option 1a) or 30-40 minutes (option 1b).

As fascinated as I am with cryptocurrency, that end-to-end buying experience sucks compared to buying traditional securities. (Oh, yeah, one more thing - I accidentally clicked "bank account transfer" earlier today to buy some cryptocurrency with the express purpose of buying an altcoin, which I wanted to do today. I can't cancel this trade in progress to do the faster one, and I won't get it until 11 days from now. You might say "So go back and do the instant one." But I can't! I can't pay that 3x fee, I just can't do it...).

Since I need to minimize friction, the only way to do it is to plan in advance. I have to buy some Ethereum or Bitcoin to always have on hand in my Coinbase account on the foreknowledge that I will want to exchange it to another altcoin 7-10+ days from now. Then, while I am sitting around waiting for the transaction to clear, do all my research and pick the ones I am planning to acquire. Then, the day the transaction clears, do the transfer to exchange to an altcoin.

In other words, always keep some funds in hand to give you some flexibility. This helps reduce trading fees.

But it does it at the risk of introducing serious slippage, and also potentially hitting you with taxes.

First, re: slippage, altcoins bounce around a lot. So, if you buy $1000 USD of Ethereum or Bitcoin on Day 1 which has an equivalent price of $1000 USD for the altcoin, on day 10 that same amount of Ethereum may only be able to purchase the equivalent of $900 or $800 of the altcoin due to its appreciation. It could also work the other way. So, there's a lot of instability and unpredictability if you're using this particular strategy.

Second, re: taxes, whenever you transfer any cryptocurrency, that's a taxable event. You buy $1000 worth of Ethereum on Day 1, and on Day 10 you exchange it to an altcoin. Suppose that your $1000 of Ethereum is now worth $1100; the way I understand the IRS tax code is you now have a capital gain of $1100-$1000 = $100 that you must track so you can pay taxes on it. If you're in the 25% tax bracket, it's like you are now buying $1075 worth of the altcoin. Of course, you've bought $1100 USD of the altcoin, but even if you never buy or sell anything for the rest of the year, you'll have a tax liability of $100 which will be treated as a short-term capital gain.

And you're responsible for tracking it all. Good luck!

This is way worse than trading stocks; whenever you sell a stock because you want to buy something else, you have to realize a capital gains similar to the above (ignoring FIFO, LIFO, or choose-your-stocks-to-sell). But the big difference is that you can convert US dollars to any stock, bond, or ETF whereas with the majority of cryptocurrencies force you to convert them from another cryptocurrency first and realize a taxable event. It's not like that buying stocks or bonds.

So no matter what strategy you use, the end-to-end experience sucks.

As a result, because of the risks involved in cryptocurrency speculation, they had better have outsized returns. But of course, the true value in cryptocurrency is not in speculation, but in driving value and using blockchain to solve problems that either couldn't be solved without it, or are much better solved with it.

That's much where I'd prefer to be.

[1] Yes, I realize that after saying I wouldn't talk about taxes, I ended up talking about taxes. So there's no need to bring that up.

[2] Just think about what this means for a second. If you are a small-time speculator in both stocks/bonds/ETFs, and also in cryptocurrencies, a $1000 purchase has a fee of 0.7% for the stock vs. 3.8% for the cryptocurrency. If there's anything I've learned about investing over the past 15 years, it's "Minimize fees!" If you had a mutual fund with a fee of 3.8%, it'd be one of the highest in the industry; only hedge funds can get away with charging that much.

People sometimes ask me "How do I, as a sender into Office 365, get images to load by default? Every time I send, the images are blocked."

I've decided to finally answer that question so I don't need to keep typing my response.

1. Images in Outlook.com load by default if you're a good sender, but as a user you can modify this behavior

By default, in Outlook.com, if you're a good sender (determined by a bunch of back-end algorithms) and the message makes it to the inbox, images will be displayed. Here's an example from Groupon, I did nothing to ensure that these images are loaded:

However, if I wanted to, I could flip a setting under Options (gear icon in top right) > Mail > Junk email > Filters and reporting. As you can see, the default behavior is to Show attachments, pictures, and links for senders with good reputation. If I flipped it to the other setting, all images would be suppressed for senders not in my Safe senders list.

(In the image above, I have removed some other sections that are not relevant to this discussion).

2. In Office 365, images do not load by default even if you're a good sender

By contrast, in Office 365, if you're viewing your email in Outlook Web Access (OWA), also known as Outlook on the Web, images do not load by default.

As a user, you can choose to either (a) show the images within the message for the current session (if you navigate away from the message and come back later, images will be suppressed); or, (b) you can choose to always show content from that sender at which point it adds to your Safe senders list. You can also manually populate your Safe senders list.

For example:

To populate your Safe senders list manually (and thereby bypass spam filtering as well), select Options (gear icon in top right > Mail > Accounts > Block or allow, and then add to your Safe Senders list. You can also Trust email from your contacts. If you do, it will skip spam filtering and I think load images as well.

You may be wondering "Okay, so how do I get all legitimate senders to load images by default, the way I can in Outlook.com?" The answer is - You can't. There is no equivalent option in Office 365.

That's one of the differences between the Consumer and Enterprise versions of the web interface for checking your email.

BTW, images in the Junk folder don't load at all, neither in Consumer nor in Enterprise.

3. In Outlook desktop client, images also do not load by default even if you're a good sender but there is a little more configurability

Outlook desktop client, when working together with Office 365 (I'm not sure how it behaves when used standalone) behaves similar to OWA for Office 365. However, there are some key differences.

Just like OWA, external senders do not load images by default unless they are in your Safe senders list. However, unlike OWA:

a) If you choose to show images on that particular session, your selection will be remembered if you navigate away and then come back

b) There is no one-click option to add the sender to your safe senders list from the infobar. Instead, you can right-click on a message and add them to your Safe senders list, or you can do it manually by right clicking and selecting Junk Email Options > Safe senders.

When it comes to downloading and showing images, these settings are controlled via the Outlook Trust Center. You can access this in the newest version of Outlook via File > Options > Trust Center > Trust Center settings > Automatic Download. Outlook actually suppresses all images for downloading, but by default enables the option to download for safe senders, and the trusted zones (published by your organization or perhaps manually populated, I don't recommend playing around with the Trusted Zone).

So you see that Outlook lets you suppress images from everyone except your Safe senders list, whereas OWA for Office 365 shows you images from your Safe senders list without giving you a way to change it. However, just like OWA for Office 365, there's no way for a sender to get images to load by default.

If you're a sender sending email to Office 365, you may be thinking "Are you telling me that I can't have images load by default in the absence of being on a user's Safe sender list?"

Correct. You have to be on the user's Safe sender list.

"But what if I'm on a good sender's list? Like the green shield list?"

That doesn't matter.

"How can I get myself added to all your users' safe senders list so that images will show in Office 365 users' mailboxes?"

You can't, that is not a service we provide.

That's a quick overview of how loading images works in our various email interfaces. I know that I haven't included Outlook mobile, nor the native Windows mail clients. Perhaps one day I will add those to this list.

But for now, this is how images load in the most used email clients that I regularly interact with.

One of the most common questions people ask me is "How do you get Office 365 to send out DMARC aggregate and forensic reports?" This is followed by "When is Office 365 going to send out DMARC aggregate and forensic reports?"

Office 365 doesn't send out DMARC reports, nor is it on our public roadmap. However, there's a hacky way you can kind of approximate aggregate reports. This is how I would do it.

Many people don't know about this, but there's basic reporting for spoofed messages flowing into your organization. This is so that you can see who is sending you spoofed messages and create allow/block lists as required (that is, for a message that doesn't authenticate and is marked as spam by our service because of it, you can allow that domain-pair [1] to go through regular filtering; conversely, if Office 365 thinks a domain-pair is legitimate, you can override it and say it's not).

Anyhow, the Get-PhishFilterPolicy gives you an overview of everything sending into the service that doesn't authenticate, so you can actually extract this information and send it off to a DMARC reporting address. You can do this via the following:

1. Connect to Exchange Online using Powershell

2. Run the following script, updating this value for your own domain:

What this does is get all the spoofed email sending as you into your organization, and then send a copy to the address in the DMARC's DNS record.

There are some things to be aware of:

a) Whoever you send this DMARC "aggregate" report to needs to understand that you are not sending a DMARC aggregate report which is in xml format. Instead, it is the Office 365 phish filter policy for antispoofing reporting and the fields do *not* map 1:1 for a regular DMARC aggregate report. The report has to be parsed and the necessary data extrapolated. And this is the best I can do at the moment. Please don't come back to me saying "But this is not a DMARC report! When are we getting DMARC reports?"

b) This script sends your organization's data to a third party outside of the Office 365 compliance boundary. It includes what we classify as End User Identifiable Information because it contains an email address in clear text. When you run this script, be aware of data that you are sending. The script above strips off the localpart of the email address, although there may be other email addresses it doesn't match. So, beware.

c) When you run this script, make sure you run it from a machine with an IP address that is in your organization's SPF record. If it isn't, you will need to do some customization.

d) I've omitted several fields from the Get-PhishFilterPolicy when sending the "aggregate" report, but IMHO they aren't required

e) This cmdlet gets all of your data, whereas you might want to only pull data every 24 hours. If it were me, I would automate it and add the following line:

Then replace the corresponding line above with:

f) I'm not a Powershell expert, there are many of you out there who are much better than me and can better optimize the script above. This script is verbose so you can see what's going on.

So, this script isn't perfect, and it excludes some key information. But it does give some useful data.

[1] A domain-pair is a combination of the From: + the sending infrastructure. If the From domain is internal to your organization (one of your org's accepted domains), the SpoofedUser is the exact email address; if the From domain is external, then the SpoofedUser is only the From: domain. The sending infrastructure (Sender) is the organizational domain of the PTR record of the sending IP. If the sending IP has no PTR record, then the Sender is the /24 range of the sending IP, e.g., 1.2.3.4 would be 1.2.3.0/24.

One of the questions that has come up recently, especially as a lot of customers migrate over from an existing spam filtering solution to Office 365, is how to force Office 365 to rely upon the spam/non-spam verdict of the service that's sitting in front of Exchange Online and not double-filter.

Office 365 already has a way to do this via connectors if the email server in front is an on-premise Exchange server. It does this by using TLS certs to "promote" the pre-existing properties of a message stamped by the Exchange server so they are re-used by Office 365. But what if your server in front is not Exchange? Then what?

There are a few ways to do this, but the key thing is that there is no simple way to disable spam filtering in Office 365. The option for "Do not filter spam" does not exist at an organizational level. Instead, you have to do a few tricks depending on the configuration.

1. You want the service in front of Office 365 to get rid of spam, such as sending it to a spam quarantine 

   In this case, the spam is taken care of somewhere else, and all email going to Office 365 is non-spam (according to the upstream filter). Therefore, you can create a set of IP Allow List entries corresponding to the relay IPs into the service. This will set SCL -1 and send all email into your users' inbox, bypassing spam scanning, and stamp IPV:CAL and SFV:SKN in the X-Forefront-Antispam-Report header.Alternatively (even preferably) you can create an Exchange Transport Rule (ETR) for those connecting IPs that (a) sets the SCL to -1, and (b) sets an x-header:

   X-Relay-IP-for-service: Allow email from <name of service>

   ... as it is already filtered for spam.

   This way, when someone in your organization gets a message that is spam, and you decide to escalate to Office 365 for missing spam even though we didn't filter it because you said not to, we'll be able to quickly look at the headers and see that's why the message was delivered to the user's inbox.

2. You want to use Office 365 to manage both spam (in the Junk Email folder) and non-spam 

   In this case, you want non-spam to end up in users' inboxes, and spam to go to the Junk Email folder the way it would for something like an Outlook.com or Gmail account.This requires you to configure the upstream mail service to configure two headers: one that is stamped when the service marks a message a spam, and a second one that is stamped when the service marks a message as non-spam. For example, Office 365 uses the X-Forefront-Antispam-Report header to stamp SCL -1 or 1 for non-spam, and SCL 5 or 6 when it is spam, and SCL 9 when it is high confidence spam.Then, in the Office 365 platform, you would write an ETR that sets SCL -1 when it sees the stamp for non-spam, and SCL 6 (or 9, depending on what you want the action to be) when the message has the spam stamp.

   You can extend this for safe and blocked senders, safe and blocked domains, etc. Whatever stamp the upfront service stamps in each case should be consumed downstream.

There is another method such as disabling junk mail filtering on a per-mailbox basis which will force messages to go through filtering but still land in the inbox (this is what I do on my own personal email, although my MX points to Office 365). However, this is an advanced scenario and causes customers a lot of trouble.

Putting additional services in front of Office 365 is what we call "complex routing"; there are several scenarios for complex routing, and the MX pointing to another service is the simplest case.

Finally, if you do use complex routing like this, there are some things to be aware of:

• You lose a big chunk of the benefits of pointing your MX to Office 365 as the first hop in the path. Make sure you read the first article in the Related Posts section at the bottom of this article
• Outlook Web Access (OWA) has some client-side integration that lights up when hooked together with Office 365, and some of those features depend on Office 365 being the first hop in the path (e.g., showing a '?' in the sender photo). When these behaviors come to Outlook desktop client, it will have the same requirement. That's another feature loss for complex routing.
• Additional services in front of Office 365 results in additional layers of management complexity
• Advanced Threat Protection works much better when Office 365 is the first hop in the path, we usually recommend putting other services behind us, and then looping the email back in via a connector. You'll have to use ETRs again and they are more complex, but that way you get the full protection
• If you get missed spam or false positives because of this configuration, it is challenging for us to do much with them because parts of our filter have been disabled

Related posts

• Hooking up additional spam filters in front of or behind Office 365
• If you use Office 365 but your MX record doesn’t point to Office, you may want to close down your security settings
• Proofpoint - Configure mail routing with Office 365
• MimeCast - Configure inbound delivery with Office 365

One of the requests that frequently crosses my desk (computer screen) is a vulnerability claim that a certain domain that is owned by Microsoft is prone to spoofing because it does not have email authentication records - neither SPF, DKIM, nor DMARC. Because this can be used to spoof, it is a vulnerability.

Microsoft Corporation owns many domains, from microsoft.com to skype.com to xbox.com to live.com. Some of these domains have strong auth records (DMARC reject or quarantine) and some have nothing published at all (e.g., microsoftstore.com).

We used to go and chase down the most commonly spoofed domains and published SPF, DKIM, and DMARC records for them; monitor the domains; and then move to a DMARC quarantine/reject record. That would lock down the domains.

However, that is no longer our strategy because it doesn't scale.

• Within Office 365 Advanced Threat Protection [1], we stopped relying upon SPF and DMARC records exclusively as a means to stop spoofing. For our ATP customers, they have antispoof protection available to them, see http://aka.ms/LearnAboutSpoofing. We also added some additional protection in the event the From domain has "microsoft" anywhere in it to ensure it gets marked as spam/phish (I call this "substring spoofing", or sometimes "brand impersonation" which by itself has multiple categories which includes putting the brand name into the Display Name, rather than the domain).
  .
• The simple addition of a hard fail to the SPF record to any of Microsoft's domains does not solve the spoofing problem for all variants of Microsoft properties – outlook.com, Hotmail.com, Hotmail.ca, msn.fr, etc., so we’d have to protect those as well. That is, we could protect one domain, but unless we went and protected everything we own, another claim could come up that yet another domain isn't protected. The locking down of all these domains is time consuming.
  .
• Furthermore, and most importantly due to the way attacks are sent these days, it doesn’t solve the spoofing problem of variants that Microsoft does not own like msn.com.com, my-msn.com, etc. Those are also damaging to our brand and can be used to spoof, but since we don't own them, we can't update those DNS records. We'd have to buy them, or figure out their current owner and shut them down. But we can't stop any and all variants, it's a nearly infinite long tail, especially as Email Address Internationalization ramps up.

Our approach to solving the problem of spoofing is to on-board customers to ATP/E5  [2] where they can get antispoofing protection regardless of the sending domain; if/when we expand Antispoofing to the rest of the customer base, those variants of the Microsoft brand will also be covered. We are also looking to expand the way we protect Top Brands like MSN by checking to see if the domain name contains a Top Brand, or a variant of it, and using other heuristics to detect it as a spoof.

In the diagram below, the green part represents authenticated email (SPF, DKIM, DMARC), the blue represents unauthenticated email that is not marked as spam, and the red is everything else.

Thus, we won't need to publish any auth records because we are flipping the model on its head - rather than relying upon a domain to publish auth records and defaulting back to assuming something is good, we instead default back to assuming something is bad. We do this because we try super hard to implicitly authenticate a message. Despite our best efforts, this can still result in false positives; but all false positives can be resolved by either publishing auth records by the domain owner, or our customer creating their own local overrides.

Publishing an SPF hard fail has its drawbacks – especially when email is forwarded. As I say above, we are looking to move protection away from the SPF/MailFrom domain to the From domain, and then use better heuristics built upon the principles of DMARC rely upon that instead. We think the industry is better served by treating unauthenticated email as suspicious regardless of whether or not it publishes auth records, and then using signals to rescue it - such as authentication or sender reputation - in the event the message isn’t malicious.

That’s how we are planning to scale up protection of our brands; we still publish auth records opportunistically, but not necessarily for all of our domains [3]. And the ones that do have SPF records may not necessarily have SPF Hard Fail. Instead, we encourage others to add their own intelligence on top of what is published to repudiate the ones that don't auth. That's what we do.

It's the only way to make this scale.

[1] And eventually the rest of the Office 365 customer base, and eventually our consumer email platform

[2] Same as [1]

[3] We will do this to improve delivery, such as a new domain is being spun up and wants better delivery because we are treating unauthenticated email as spam/phish/spoof. Adding SPF, DKIM, and DMARC will help in self-identifying that the domain is authorized to send email. It's kind of (?) like a Spamhaus PBL for domains

About a year ago, in Office 365, we released the feature that - similar to Gmail - Outlook Web Access stamps a '?' in the sender photo when the message is not authenticated with either SPF or DKIM.

Well, I was pleasantly surprised to see this morning it's also in Outlook (for Windows, not sure about other clients):

This is great to see, we're now managed to unify the behavior between both Outlook and Outlook Web Access, at least in Office 365.

Woot!

Some of you may have noticed that the amount that I post on this blog has lessened over the years, and especially this year. This is not for lack of ideas, but simply lack of time to write good, in-depth articles.

However, irrespective of that, this is my last blog post on this blog.

After spending 14 years at Microsoft (which includes my time with Frontbridge) in the antispam business - fighting the spam, malware, and phishing, and talking about reputation and everything related to email - I’ve decided change is in the air and I’ve made the difficult decision to move on from Microsoft to the next chapter in my professional life.

I have thoroughly enjoyed my time at Microsoft, having been given the opportunity to work on world class projects and features. It is satisfying to see so many of the things we’ve worked on together getting used by millions of people around the world. Yet the key piece in that sentence is the word “together” because beyond working on great software, the most rewarding piece is working with so many great, hard-working, and talented people. There really is no better reward than that. I know we all get heads-down with developing the things we need to get done, but when I look around and see the things that everyone else (other than me) has gotten done, it makes me proud to say “Hey, we’ve all done something great here!”

I have also enjoyed writing this blog. It has given me the time to practice my writing skills, helped me organize my thoughts out-loud, given me a log to refer back to, and made it possible for me to refer back to features I worked on years ago. If any of you reading this found it valuable, I am pleased I could be of service.

You can still find me at various places around the web:

I haven't decided whether or not I will start another blog. Maybe I will, maybe I won't. It depends on where I end up, and those other platforms have a bigger audience anyway.

I thought it would be appropriate to share the oath we here in the antispam team who have been here a little longer had to swear by when we first joined. It’s quite techno-geeky, but they make us do it in a secret ceremony. The part in bold is my own addition:

Spam fighters, and now my watch begins,
It shall not end until my metaphorical death (that is, when I leave the company).
I shall send no unwanted email, hold no typo-squatted domains, populate no spamtraps.
I shall wear no black hats and win no glory. I shall learn how to send email via Telnet and Powershell.
I am rod that reels in the lures of phish. I am the spreader of the net. I am the shield that guards the realms of email servers.
I pledge my honor to the Spam Fight, for this night and all the nights to come.

And now my watch is over.

Thank-you for reading this blog.

This is a rough description of how the Phishing Confidence Level (PCL) works in Office 365.

Way back in the olden days - 2007 or so - Exchange server used to have its own spam filter, Smartscreen. This was more-or-less the same spam filter running in Outlook.com. But whereas Smartscreen in Outlook.com (then known as Hotmail) updated multiple times per day, the spam filter in Exchange only updated periodically.

Smartscreen on Exchange was designed to communicate with Outlook the email client. When Smartscreen set the Spam Confidence Level (SCL), this was propagated to Outlook who would move messages SCL 5 and higher to the Junk Email folder. The SCL was stamped in the following header:

X-MS-Exchange-Organization-SCL: 5

Smartscreen could also detect if a message was phishing, and if so, it would set the Phishing Confidence Level (PCL) to 8. It would stamp this in the following header:

X-MS-Exchange-Organization-PCL: 8

Or, perhaps this:

X-MS-Exchange-Organization-PCL: Suspicious

That comes from Antispam Stamps, our documentation from Exchange on-prem's filtering (I'm also not totally sure on the format, I rarely work with the phishing filter for on-prem because I'm all about the Office 365 and Outlook.com services).

But it's not enough to store these values in headers, Exchange also stores them in MAPI properties. Not every X-MS-Exchange-Organization-* header corresponds to a MAPI property, but these ones do (MAPI is a database-backed mailstore for Exchange; I've been working with it for years and still don't fully understand it). So, Outlook has access to those headers but also to those MAPI properties.

Whenever Outlook sees PCL 8 in a MAPI property, it displays the following pink bar in the email client:

You may have seen this in Outlook; if you do, it's because PCL 8 has been set.

A couple years ago, we moved away from the native Outlook phishing bar because... well, it's aesthetically unpleasing and hard to update (we have to update the code and then get our users to update their software. That's an uphill battle.) In its place, we moved to Safety Tips. Where's what the new Safety Tip looks like for phishing:

This new-and-improved Safety Tip is only available in Outlook.com and Office 365. It is not available in Exchange on-prem. Doesn't this look way better? The "Learn more" link can be controlled dynamically, that is, we can change where it points to.

The problem is that all the code to show the pink bar still exists in Outlook, which means that if Office 365 (or Outlook.com) ever sees a phishing message and tries to set PCL 8, which is how we've defined phishing previously, that native pink bar will show up in the client. We don't want that, we only want to show the new Safety Tip.

To solve this problem, we decided that we would reserve a new value for phishing: -9990. That way, we would always know that it was Office 365 that identified a message as phishing and not some other service, and Outlook would never show the native pink bar. It allows us to instead show only the newest Safety Tip.

The PCL is supposed to be stamped in two places, the first is the X-Microsoft-Antispam header in the PCL property, and then additionally in the X-MS-Exchange-Organization-PCL header. But it also gets stamped sometimes in the X-Exchange-Antispam-CFA-Test header which is an internal header.

But sometimes a message still gets PCL 8, despite my best efforts to purge PCL 8 from the system.

1. Exchange on-prem still sets the PCL value

If you are running Exchange on-prem in front of Office 365, and it detects a message as phish and sets PCL 8, and you have a set of connectors to connect to Office 365 which propagates message properties forward, Office 365 will inherit and promote the previously existing PCL the same as it does a previously set SCL. In a case like this, PCL 8 will appear in the message headers. However, the X-MS-Exchange-Organization-PCL header will have a bunch of other properties that Office 365 does not use, and that allows us to differentiate between Exchange on-prem and Office 365's antiphishing.

2. Corner cases in our code

Despite my super-best efforts, sometimes messages get PCL 8 because they have a "Mark as phish" rule somewhere in the system that does not have PCL -9990 assigned, and instead gets a default PCL 8. This one drives me crazy. Usually, although not always, when a message is marked as phish in our system, it's because the SFTY value is 9.x, and 9.1 is the most common reason (this property is not documented). But sometimes even in a default case, 9.1 is stamped. Those are difficult to track down. Fortunately, this doesn't happen that much anymore.

The Phishing Confidence Level is a little more complicated than that, because we have machine learning and some other technologies, and I don't quite know how they interact with the SFTY and PCL. I haven't worked on those features and I'm not so familiar with them. Still, this description - as of this writing - covers the majority of PCL and phishing cases that I have seen.

I hope you find this useful.