People sometimes run into the "too many DNS lookups" error when rolling out SPF (Sender Policy Framework). It doesn't help that there is a lot of bad guidance on the Internet. This article describes how to fix this issue. Read the rest of this entry »
How can you eventually move to a p=reject policy when third parties are unable to send email properly on your behalf?
In many cases, a DMARC compliant SMTP relay server can be used to do the trick. In this article, we’ll explore some of the facets of sending DMARC compliant email from third parties, what to look for, and how common hosted solutions such as Google Apps, Office 365, Amazon Simple Email Service (SES), can be leveraged as SMTP relays. Read the rest of this entry »
We've put together a short overview video about SPF:
This video is part of a larger video series on all things DMARC.
The transcript follows:
Read the rest of this entry »
SPF looks at the bounce address of an email when doing its check. (The bounce address is also referred to as the Return-Path, the MAIL FROM address, the envelope address, and in some circles, the RFC5321.MailFrom address.) When SPF does its check, it will look for an SPF record using the domain found in the bounce address.
DMARC attempts to correlate the results of checking SPF and DKIM with the domain found in a message's From: header. It's the domain of the From: header that ends up tying together all things DMARC.
Companies often misunderstand how SPF works Read the rest of this entry »
People sometimes write in and ask "what is the impact of a broken SPF record"?
The net effect of a broken SPF record is that receivers can't reliably use SPF to determine the legitimacy of the domain's email. *Some* receivers might ignore the broken parts of an SPF record and keep checking, but out of the box all SPF implementations will barf, and you'll be left with a record that is introducing uncertainty into email performance.
We've discovered an odd story related to SPF deployment. When delivering a lot of email, you need an SPF record just to get to the welcome mat. What happens is that new senders go through their pre-flight check list, start sending with working SPF records, and things operate in a nice steady state for a while.
Over time, the servers that are authorized using SPF earn enough good reputation so that senders get through the front door and into the content-based/user-engagement world. At any time, SPF records may get extended until they break or become inaccurate, but since the original set of authorized servers already has a good enough reputation, the inaccurate/broken SPF records do not cause too much trouble. This is how SPF has traditionally been used prior to DMARC. In this way, SPF might have been a "reputation bootstrapping tool".
Unfortunately, when SPF breaks, the newly authorized servers are left out in the cold. People then hire deliverability experts, build volume ramps, and chase symptoms as most of their email is doing OK. Worse is the resulting notion that email infrastructure is a heavy weight and cannot be replaced or upgraded without serious investment.
DMARC changes the above story in a very significant way. First is visibility — you can actually see what is going on and where SPF isn't performing like it should. Second is that SPF results now matter in terms of email content and identifier alignment. In sum, accurate SPF records are no longer a "reputation bootstrapping tool", but necessary for reliable email delivery.
If PTR mechanisms are detected, the current diagnostic output is:
Warning: PTR mechanisms SHOULD NOT be used and cannot be resolved using this diagnostic tool. More info at <this page!>.
What does the PTR mechanism mean? When an email receiver gets a piece of email and the PTR mechanism is in the sender's SPF record, the receiver will look at the incoming IP address and do a "PTR" lookup. For example, if the sender is sending email from IP address 22.214.171.124, the receiver will perform a PTR lookup of 126.96.36.199 to attempt to retrieve a hostname. Lastly, if a hostname is discovered for IP address 188.8.131.52, then that hostname's domain is compared to the domain that was originally used to lookup the SPF record.
3 important things about the above:
- The PTR mechanism has been deprecated. See the relevant RFC for more info.
- The SPF Surveyor cannot resolve PTR mechanisms because a real connection from a real sender is necessary to complete the lookup.
- MOST IMPORTANTLY: Some large receivers will skip the mechanism – or worse they'll skip the entire SPF record – because such mechanisms cannot be easily cached. Imagine a large receiver doing a PTR lookup for millions of different connections... the size of the local cache explodes.
SPF is all about publishing a list of servers that are authorized to send on behalf of a domain.
After writing out a list of servers in the form of an SPF record, the right thing to do is to end an SPF record with something that says "and everything else on the Internet is NOT authorized".
The way the above is written is to use the "all" mechanism. This mechanism matches everything. By adding a prefix of "~" or "-", the meaning of the mechanism is changed to be:
- "softfail" in the case of "~"
- "fail" in the case of "-"
Both mean "NOT PASS", but there is a subtle difference, and it has to do with history.
Before DMARC came along, SPF tried to allow its users to express policy -- that is, what should be done if SPF fails. "softfail" was largely interpreted to mean "NOT PASS". "fail" was interpreted by a few operators to mean "NOT PASS AND DISCARD THIS FAILING EMAIL".
Discarding email based on SPF results ended up causing too much legitimate email to be dropped (because of improperly configured SPF records, or vendors who didn't understand how to send SPF-compliant email), and almost all receivers ended up using SPF as input for anti-spam engines. However, a few operators still interpreted "fail" as meaning "NOT PASS AND DISCARD".
Fast forward to the world of DMARC. Now receivers are using DMARC to find any positive assertion that a domain is associated with a piece of email. In this context, ~all is the same thing as -all... "NOT PASS".
However, if you operate with "-all" in your SPF record, you might run into an operator (once in a blue moon) that discards otherwise legitimate email. Debugging this issue can be difficult. This issue could be sidestepped by using "~all" instead of "-all". Alternatively, use of "-all" can still be quite valuable in conveying confidence in the correctness of your SPF record; with that flag it is more likely that a failed match will result in fraudulent messages being discarded by MTAs which do not yet support DMARC.