From Wikipedia, the free encyclopedia - View original article
Sender Policy Framework (SPF) is an email validation system designed to prevent email spam by detecting email spoofing, a common vulnerability, by verifying sender IP addresses. SPF allows administrators to specify which hosts are allowed to send mail from a given domain by creating a specific SPF record (or TXT record) in the Domain Name System (DNS). Mail exchangers use the DNS to check that mail from a given domain is being sent by a host sanctioned by that domain's administrators.
The Simple Mail Transfer Protocol permits any computer to send email claiming to be from any source address. This is exploited by spammers who often use forged email addresses, making it more difficult to trace a message back to its sender, and easy for spammers to hide their identity in order to avoid responsibility. It is also used in phishing techniques, where users can be duped into disclosing private information in response to an email purportedly sent by an organization such as a bank.
SPF allows the owner of an Internet domain to specify which computers are authorized to send mail with sender addresses in that domain, using special Domain Name System (DNS) records (SPF, type 99). Receivers verifying the SPF records may reject messages from unauthorized sources before receiving the body of the message. Thus, the principles of operation are similar to those of DNS-based blackhole lists (DNSBL), except that SPF uses the authority delegation scheme of the Domain Name System. Early implementations used TXT records for implementation before the new record type was commonly available in DNS software. Use of TXT records for SPF was intended as a transitional mechanism. However, according to the current RFC, RFC 4408, section 3.1.1, "An SPF-compliant domain name SHOULD have SPF records of both RR types. A compliant domain name MUST have a record of at least one type," and as such, TXT record use is not deprecated.
The sender address is transmitted at the beginning of the SMTP dialog. If the server rejects the sender, the unauthorized client should receive a rejection message, and if that client was a relaying message transfer agent (MTA), a bounce message to the original sending address may be generated. If the server accepts the sender, and subsequently also accepts the recipients and the body of the message, it should insert a Return-Path field in the message header in order to save the sender address. While the address in the Return-Path often matches other originator addresses in the mail header such as From or Sender, this is not necessarily the case, and SPF does not prevent forgery of these other addresses.
Spammers can send email with an SPF PASS result if they have an account in a domain with a sender policy, or abuse a compromised system in this domain. However, doing so makes the spammer easier to trace.
The main benefit of SPF is to the owners of e-mail addresses that are forged in the Return-Path. They receive large amounts of unsolicited error messages and other auto-replies. If such receivers use SPF to specify their legitimate source IP addresses and indicate FAIL result for all other addresses, receivers checking SPF can reject forgeries, thus reducing or eliminating the amount of backscatter.
SPF has potential advantages beyond helping identify unwanted mail. In particular, if a sender provides SPF information, then receivers can use SPF PASS results in combination with a white list to identify known reliable senders. Scenarios like compromised systems and shared sending mailers limit this use.
If a domain publishes an SPF record, spammers and phishers are less likely to forge e-mails pretending to be from that domain, because the forged e-mails are more likely to be caught in spam filters which check the SPF record. Therefore, an SPF-protected domain is less attractive to spammers and phishers. Because an SPF-protected domain is less attractive as a spoofed address, it is less likely to be blacklisted by spam filters and so ultimately the legitimate e-mail from the domain is more likely to get through.
SPF breaks plain message forwarding. When a domain publishes an SPF FAIL policy, legitimate messages sent to receivers forwarding their mail to third parties may be rejected and/or bounced if all of the following occur:
Publishers of SPF FAIL policies must accept the risk that their legitimate emails are being rejected or bounced. They should test (e.g., with a SOFTFAIL policy) until they are satisfied with the results. See below for a list of alternatives to plain message forwarding.
For an empty Return-Path as used in error messages and other auto-replies, an SPF check of the HELO-identity is mandatory.
With a bogus HELO identity the result NONE would not help, but for valid host names SPF also protects the HELO identity. This SPF feature was always supported as an option for receivers, and later SPF drafts including the final specification recommend to check the HELO always.
This allows receivers to white list sending mailers based on a HELO PASS, or to reject all mails after a HELO FAIL. It can also be used in reputation systems (any white or black list is a simple case of a reputation system).
Compliance with SPF consists of three loosely related tasks:
551 User not local; please try <firstname.lastname@example.org>,
Thus, the key issue in SPF is the specification for the new DNS information that domains set and receivers use. The records laid out below are in typical DNS syntax. Note that RFC 4408 recommended that both SPF and TXT records be used (during the transitional period), although either by itself was acceptable:
example.com. IN TXT "v=spf1 ip4:192.0.2.0/24 ip4:198.51.100.123 a -all" example.com. IN SPF "v=spf1 ip4:192.0.2.0/24 ip4:198.51.100.123 a -all"
"v=" defines the version of SPF used. The following words provide mechanisms to use to determine if a domain is eligible to send mail. The "ip4" and "a" specify the systems permitted to send messages for the given domain. The "-all" at the end specifies that, if the previous mechanisms did not match, the message should be rejected.
Eight mechanisms are defined:
|ALL||Matches always; used for a default result like -all for all IPs not matched by prior mechanisms.|
|A||If the domain name has an address record (A or AAAA) that can be resolved to the sender's address, it will match.|
|IP4||If the sender is in a given IPv4 address range, match.|
|IP6||If the sender is in a given IPv6 address range, match.|
|MX||If the domain name has an MX record resolving to the sender's address, it will match (i.e. the mail comes from one of the domain's incoming mail servers).|
|PTR||If the domain name (PTR record) for the client's address is in the given domain and that domain name resolves to the client's address (forward-confirmed reverse DNS), match.|
|EXISTS||If the given domain name resolves to any address, match (no matter the address it resolves to). This is rarely used. Along with the SPF macro language it offers more complex matches like DNSBL-queries.|
|INCLUDE||If the included (a misnomer) policy passes the test this mechanism matches. This is typically used to include policies of more than one ISP.|
Each mechanism can be combined with one of four qualifiers:
The modifiers allow for future extensions to the framework. To date only the two modifiers defined in the RFC 4408 have been widely deployed:
As soon as SPF implementations detect syntax errors in a sender policy they must abort the evaluation with result PERMERROR. Skipping erroneous mechanisms cannot work as expected, therefore include:bad.example and redirect=bad.example also cause a PERMERROR.
Another safeguard is the maximum of ten mechanisms querying DNS, i.e. any mechanism except from IP4, IP6, and ALL. Implementations can abort the evaluation with result SOFTERROR when it takes too long or a DNS query times out, but they must return PERMERROR if the policy directly or indirectly needs more than ten queries for mechanisms. Any redirect= also counts towards this processing limit.
A typical SPF HELO policy v=spf1 a -all may execute up to three DNS queries: (1) SPF, (2) TXT (for backwards compatibility during the transition), and (3) A or AAAA. This last query counts as the first mechanism towards the limit (10). In this example it is also the last, because ALL needs no DNS lookup.
SPF FAIL policies can be an effective but problematic tool. A typical example is a user that wishes to send an email from a private PC or a mobile phone: the user uses their corporate email address but may use a different outgoing SMTP server which is not listed in the SPF record. The corporate domain may therefore be secure by blocking all email that does not originate from themselves, but have thereby limited some of their own users. Many organizations consider this compromise acceptable and even desirable.
SPF PASS is useful for authenticating the domain for use as a parameter to a spam classification engine. That is, the domain in the sender address can be considered to be authentic if the originating IP yields an SPF PASS. The domain can then be referenced against a reputation database.
SPF results other than PASS (used in combination with a reputation system) and FAIL cannot be meaningfully mapped to PASS and FAIL. However, a reputation system can easily track independent reputations for each SPF result, i.e. example.com:PASS and example.com:NEUTRAL would have different reputations, and ditto for the other results. This approach is useful even without whitelisting plain forwarders, because the FAIL results from the plain forwarders simply accrue an independent reputation.
The meaning of PASS, SOFTFAIL, FAIL is sometimes incorrectly interpreted to mean "not-spam", "maybe-spam", "spam" respectively. However SPF does nothing of the sort. SPF merely offers an organization firstly the means to classify emails based on their domain name instead of their IP address (SPF PASS); and secondly, the means to block unauthorized use of their domain (SPF FAIL).
In a naive implementation, SPF does not prevent a user with the same domain sending an email on behalf of another user because only the domain part of the address is used to locate the SPF policy record. In more sophisticated implementations, the domain owner can specify separate policies for each user by means of SPF "macros" that reference the "localpart" (user) as defined in RFC 4408, or simply require all mail submissions for the domain to use SMTP AUTH (RFC 4954). The latter is highly recommended anyway for many reasons.
SPF needs to operate on the host indicated by the receiving domain's MX record. This means the host(s) that are the direct recipient of remote TCP connections, because such a host can easily deduce the originating IP address from the TCP session. These hosts are able to block the email during the SMTP session, avoiding the necessity to generate bounce messages which could be backscatter. (As the SPF RFC 4408 says, "Generating non-delivery notifications to forged identities that have failed the authorization check is generally abusive and against the explicit wishes of the identity owner.")
Other downstream hosts, for instance in a forwarding scenario, can only perform SPF checks based on "Received" headers. This is cumbersome and error-prone. A better approach is for the MX host to check SPF without blocking any email, and then add a "Received-SPF" header field as specified in RFC 4408 or the newer "Authentication-Results" header of RFC 7001. Downstream hosts can then look at these trace headers and set their own policy of whether to reject, accept, or quarantine based on the SPF result and other factors.
An Internet draft discussed concerns related to the scale of an SPF answer leading to network exploits as a means to corrupt the DNS. This issue is also covered in the security considerations of the SPF RFC. The SPF project did a detailed analysis of this draft and claimed that SPF does not pose any unique threat of DNS DoS, citing example attacks using NS and MX records and identifying void DNS lookups (negative caching) as the key DNS weakness.
An SPF based attack can generate more than 40KB of traffic per message originating completely from recipient resources once a spam campaign containing MAILFROM with unique local-parts exceed the recipient's negative caching limits commonly imposed. Often Windows based services impose rather low limits and other resolver offerings also permit low negative caching limits to mitigate access problems following service disruptions. SPF includes the "l" macro able to combine components of the email-address local-parts to construct unique DNS requests generated by recipient resources. The SPF result for "email@example.com" may not be the same as that for "firstname.lastname@example.org" which always requires repeating potentially long sequences of more than 100 DNS transactions based upon the same cached SPF record.
SPF provides several advantages for malefactors:
SPF validates the message envelope (the SMTP bounce address), not the message contents (header and body) – this is the distinction between SMTP (as specified in STD 10 or RFC 5321) and Internet Message Format (as specified in STD 11 or RFC 5322). It is orthogonal and complementary to DomainKeys Identified Mail (DKIM), which signs the contents (including headers).
In brief, SPF validates MAIL FROM vs. its source server; DKIM validates the "From:" message header and a mail body by cryptographic means.
Sender ID RFC 4406, is a parallel solution to the problem of message validation, and defines a pair of closely related tests. One validates a message's Purported Responsible Address (PRA) as defined in RFC 4407. The other validates a message's Reverse-Path (also known as MAIL-FROM address) as defined in RFC 4408.
Quoting from RFC4407:
"Note that the Sender ID experiment may use DNS records that may have been created for the current SPF experiment or earlier versions in this set of experiments. Depending on the content of the record, this may mean that sender-ID heuristics would be applied incorrectly to a message. Depending on the actions associated by the recipient with those heuristics, the message may not be delivered or may be discarded on receipt."
Those publishing SPF DNS records should consider the advice given in section 3.4 of RFC 4406 and may wish to publish both v=spf1 and spf2.0 records to avoid the conflict. However, note that despite the version label used, Sender ID is not technically a second revision of SPF.
Some spammers use SPF to decrease spam-rating by specifying wide mask in valid server address, so any spam from botnets becomes spf-valid and the probability to pass spam-filters increases.
seminar-for-you.ru. 14400 IN TXT "v=spf1 a mx ip4:18.104.22.168/2 ip4:22.214.171.124/2 ip4:126.96.36.199/2 ip4:188.8.131.52/2 -all"
worldwidemail.ru. 13733 IN TXT "v=spf1 a mx ip4:184.108.40.206/2 ip4:220.127.116.11/2 ip4:18.104.22.168/2 ip4:22.214.171.124/2 -all"
example.com. 21600 IN SPF "v=spf1 +all"
This last record says that any host on the Internet may send mail on behalf of the domain/hostname example.com. Although syntactically valid, "+all" is indicative of an administrator who does not care about SPF or the mail forgeries it detects.
For stable domains, this simply means that any reputation attached to the domain is the same with or without SPF and such spam domains are easily learned and rejected. The real value of wide-mask SPF policies to spammers is with "throw-away" domains that are registered, used to send spam from botnets for a few days, and then abandoned.
The idea of limiting by IP address who can send mail using a given sender domain may date back as far as 1997. The first public mention of the concept was in 2000 but went mostly unnoticed. No mention was made of the concept again until a first attempt at an SPF-like specification was published in 2002 on the IETF "namedroppers" mailing list by Dana Valerie Lank (previously D. Green), who was unaware of the 2000 mention of the idea. The very next day, Paul Vixie posted his own SPF-like specification on the same list. These posts ignited a lot of interest, and eventually led to the forming of the IETF Anti-Spam Research Group (ASRG) and their mailing list, where the SPF idea was debated among a subscriber base that seemed to grow exponentially day by day. Among the proposals submitted to the ASRG were "Reverse MX" by Hadmut Danisch, and "Designated Mailer Protocol" by Gordon Fecyk.
In June 2003, Meng Weng Wong merged the RMX and DMP specifications and solicited suggestions from other programmers. Over the next six months, a large number of changes were made and a large community had started working on SPF.
Originally SPF stood for Sender Permitted From and was sometimes also called SMTP+SPF, but its name was changed to Sender Policy Framework in February 2004.
After the collapse of MARID the SPF community returned to the original "classic" version of SPF. In July 2005 this version of the specification was approved by the IESG as an IETF experiment, inviting the community to observe SPF during the two years following publication. On April 28, 2006, the SPF RFC was published as experimental RFC 4408.
There are other concerns about the impact of widespread use of SPF, notably the impact on various legitimate forms of email spoofing, such as forwarding services, SMTP use by people with multiple identities, etc. (For example, a person who uses their home ISP's SMTP servers to send mail with their work email as the address.) On the other hand, many of these uses may be "expected" yet not "legitimate". To a certain extent this is more a question of ownership and expectations than a technical question.
The IETF spfbis working group, tasked with reworking the SPF specification aiming for "Proposed Standard" status in a new RFC, during April 2013 appeared to have reached consensus around deprecating SPF type 99 in favour of continued TXT record usage. People from the DNSEXT working group opposed this strongly in a series of email threads on spfbis, dnsext, and IETF general discussion mailing lists. The spfbis working group chair requested to stop that torrent of protest, since the discussion on the resource record type (RRTYPE) in the working group was terminated long time before  a move that was seen as trying to silence the protests by some fierce DNS purists. An independent draft was proposed later, documenting how the spurious recursion to TXT records is characterized in the current Internet.
Anti-spam software such as SpamAssassin version 3.0.0 and ASSP implement SPF. Many mail transfer agents (MTAs) support SPF directly such as Courier, CommuniGate Pro, Wildcat, MDaemon, and Microsoft Exchange, or have patches/plug-ins available that support SPF, including Postfix, Sendmail, Exim, qmail, and Qpsmtpd. As of 2013, more than seven million domains publish SPF FAIL -all policies.
In a survey published in 2007, 5% of the .com and .net domains had some kind of SPF policy. In 2009, a continuous survey run at Nokia Research reports that 51% of the tested domains specify an SPF policy. These results can include trivial policies like v=spf1 ?all. In April 2007, BITS, a division of the Financial Services Roundtable, published e-mail security recommendations for its members including SPF deployment.
In 2008, the Messaging Anti-Abuse Working Group (MAAWG) published a paper about email-authentication covering SPF, Sender ID, and DKIM. In their "Sender Best Communication Practices" the MAAWG stated: "At the very least, senders should incorporate SPF records for their mailing domains".