SPF and DMARC are standards that describe how the origins of email messages should be verified, to prevent email spoofing. I spent some free time over the past few weeks creating
checkdmarc , a Python 3 module and command-line interface that can validate and troubleshoot SPF and DMARC records across multiple domains, with the intent of building it into a web application that will process DMARC reports. The Department of Homeland Security recently launched an initiative to deploy SPF, DMARC, and other best practices on most federal agency domains by issuing BOD 18-01. This created the perfect case study of common challenges and mistakes when deploying SPF and DMARC across very large organizations, and even a few small ones.
2018-01-30 update: I have made many improvements to my script, corrected a few of my own misconceptions about DMARC I had in this post, and switched to updated results from 2018-01-28.
DHS Binding Operational Directive 18-01 (BOD 18-01)
On October 17, 2017 the US Department of Homeland Security (DHS) issued Binding Operational Directive 18-01 (BOD 18-01). This directive requires most federal agencies to deploy cyber hygiene best practices to secure their websites and email services, including SPF, DMARC, STARTTLS, and HTTPS with HTS:
- “Within 30 days of BOD issuance (November 16th, 2017), submit an “Agency Plan of Action” to FNR.BOD@hq.dhs.gov and begin implementing the plan.
- Within 90 days (January 15, 2018) of BOD issuance:
- Configure all internet-facing mail servers to offer STARTTLS.
- Configure all second-level domains to have valid SPF/DMARC records, with at minimum a DMARC policy of “p=none” and at least one address defined as a recipient of aggregate and/or failure reports.
- By 120 days (February 13, 2018) after BOD issuance:
- Ensure all publicly accessible Federal websites and web services provide service through a secure connection (HTTPS-only, with HSTS).
- Identify agency second-level domains that can be HSTS preloaded, and provide a list to DHS at FNR.BOD@hq.dhs.gov.
- Disable SSLv2 and SSLv3 on web and mail servers.
- Disable 3DES and RC4 ciphers on web and mail servers.
- Within 15 days of the establishment of a centralized NCCIC reporting location, add DHS as a recipient of DMARC aggregate reports.
- Within one year (October 16, 2018) of BOD issuance, set a DMARC policy of “reject” for all second-level domains and mail-sending hosts.”
Unfortunately, Intelligence Community and/or National Security systems are not required to comply with BODs, although they would still benefit from applying them.
Enabiling STARTTLS on mail servers allows senders to establish an encrypted connection before sending email. Enabling HTS on a web server tells clients to only use an encrypted (HTTPS) connection whenever they are accessing the site in the future. Disabling obsolete, weak cypher suites prevents easy exploits such as POODLE from working. SPF and DMARC are standards that work in tandem to prevent email spoofing. This blog post will focus on these standards. However, all of the best practices are key parts of defense in depth.
What is email spoofing?
Emails are spoofed when the from field of the message appears to be from someone other than the actual sender. For example, an attacker sending a phishing email can set the
from address to match the target’s CEO. In many organizations, the target’s mail client will even helpfully display a known email address as a name and picture from their internal profile, which makes the phishing email look more legitimate.
Spoofing is a powerful, simple tool for attackers, but there are legitimate reasons for spoofing email from addresses. If an organization is using an outside email marketing service, helpdesk, or any other outside service that sends email, there is a need to make the items appear they are coming from their trusted domain, and not the third party, so recipients know who is responsible for the messages.
So how can a recipient mail server differentiate from legitimate and illegitimate spoofing? SPF, DKIM, and DMARC, of course!
What is SPF?
The Sender Policy Framework (SPF) is a standard for listing mail servers that are authorized to send email from your domain. It is defined in RFC 7208. The domain owner publishes a specially formatted
TXT record in the DNS zone containing a list of authorized servers.
Details about the exact record format can be found here.
When a receiver mail server receives an email, it checks to see if the domain in the
MAIL FROM part of the SMTP envelope has published a SPF record. If a record is found, the IP address of the sending mail server is compared to the list. Note that SPF does not check the from header that the user sees, and the SMTP envelope is not part of the message headers. so it is easy to bypass SPF and still spoof email to the user. The best illustration I have found of this comes from the great guide at o365info.com that explains exactly how to bypass SPF:
Also, It is up the receiving mail server to decide what to do with an email if the SPF check fails. SPF does not specify an action for the recipient mail server to take in the event of a validation failure. DMARC fills these gaps, and more.
Common mistakes when deploying SPF and how to avoid them
Not deploying SPF (and DMARC) on domains that do not send email
An attacker can still spoof email to make it look as if it is coming from a domain even if it doesn’t have MX records and/or you don’t use it for email at all; these domains are the easiest to deploy SPF on: just add a
TXT record containing
Including the same servers more than once
There are multiple mechanisms to allow a server, but a server only needs to be allowed by one of them. For example, if you use the ipv4 mechanism to state that a whole netblock is allowed allowed to send email, you don’t need to use an a mechanism to include a sever by hostname.
Using the wrong mechanism
The most common mechanisms in use are
ipv4: used to specify a single IPv4 address or CIDR netblock
ipv6: used to specify a single IPv6 address or CIDR netblock
mx: used to specify the servers listed in a domain’s
MXresource records, not for allowing individual mail servers by hostname. If a
mxmechanism includes a value (e.g.
mx:mail.example.com), the MX records located at that value are queried; if a value is not included with the mechanism (e.g.
mx), the domain’s own records are queried
a: used to specify a server by hostname, or a domain’s own
AAAArecords if no hostname is included with it; this is frequently used to allow web servers to send mail, since they are not included in MX records
include: used to include SPF mechanisms from another domain. This is frequently used to include lists of
ipv6mechanisms from third party services that send email as the domain including cloud email and marketing services, such as Office 365, Google, or Salesforce; multiple include statements can bu used (e.g.
includemechanisms do not include the
allvalue; to include mechanisms and the
allvalue (e.g. to have a standard configuration for all of your domains), use the
redirect=modifier at the end of the record – this modifier can only be used once per record
Using too many mechanisms that trigger DNS lookups
RFC 7208, section 4.6.4 limits the combined number of
redirect mechanisms and modifiers used in a SPF record to 10, including any mechanisms from included records – the total number of
MX records returned per
mx mechanism is also cannot exceed a separate limit of 10; this is done to prevent DNS-based denial-of-service (DoS) attacks
To reduce the number of DNS lookup mechanisms in a record:
- If you have sending server IP addresses that won’t change often, consider using
ipv6mechanisms, which don’t trigger DNS lookups, instead of
- Have third party services send emails from dedicated subdomains; for example, if a billing services vendor currently sends emails as
firstname.lastname@example.org, have them send from
email@example.com, and make that email address an alias if you want to receive replies at
firstname.lastname@example.org; this way
billing.yourdomain.comgets its own SPF record and limits. Having MX records for these subdomains improves en emails trustworthiness to many gateways, even if you don’t actually have mailboxes for that subdomain.
According to RFC 7208, section 5.5, the
ptr mechanism should not be used
What is DKIM?
DKIM is a stronger method of validating that an email was sent from an authorized server than SPF. The sending server signs outgoing mail with a private key, and places the signature in a message header and the receiving server validates the signature using a public key published in DNS, as defined in RFC 6376 – DomainKeys Identified Mail (DKIM) Signatures. Messages may have multiple DKIM signature headers if they have been forwarded through multiple gateways.
Take a look at this example, lightly modified from Wikipedia:
DKIM-Signature: v=1; a=rsa-sha256; d=example.net; s=brisbane; c=relaxed/simple; q=dns/txt; l=1234; t=1117574938; x=1118006938; h=from:to:subject:date:keywords:keywords; bh=MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=; b=dzdVyOfAKCdLXdJOc9G2q8LoXSlEniSbav+yuU4zGeeruD00lszZ VoG4ZHRNiYzR
A verifier queries the
TXT resource record type of
Here, example.net is the author domain to be verified against (in the
brisbane is a selector given in the
s tag, and
_domainkey is a fixed part of the protocol.
There are no CAs or revocation lists involved in DKIM key management. The selector is a straightforward method to allow signers to add and remove keys whenever they wish. Long lasting signatures for archival purposes are outside DKIM’s scope. Other tags are visible in the example are:
- v – version
- a – signing algorithm
- d – domain
- s – selector
- c – canonicalization algorithm(s) for header and body
- q – default query method
- l – length of the canonicalized part of the body that has been signed
- t – signature timestamp
- x – expire time
- h – list of signed header fields, repeated for fields that occur multiple times
What is DMARC?
DMARC is the Domain-based Message Authentication, Reporting, and Conformance standard, as described in RFC 7489. It consists of a specially-formatted DNS
TXT resource record published by a Domain Owner, that tells mail servers what to do when they encounter a message that appears to be from that domain, but fails SPF and/or DKIM. It also provides methods for mail receivers to notify Domain Owners when mail fails the DMARC check.
DMARC records are
TXT resource records of a subdomain of a second-level domain called
_dmarc.example.com. Unlike SPF, DMARC options apply to all subdomains, under the base domain (i.e.
example.com). DMARC records are composed of tags and values that are joined by
=, and delimited by
;. All DMARC records start with
The most important and commonly used DMARC tags are
p tag sets the DMARC policy, which specifies the policy to be enacted by the Receiver at the request of the Domain Owner on mail that is supposedly from the Domain Owner’s domain fails the DMARC check. The policy applies to the domain and to its subdomains, unless subdomain policy is explicitly described using the
sp tag. Possible policy values include:
|The Domain Owner requests no specific action be taken regarding delivery of messages. This is the implicit default policy.|
|The Domain Owner wishes to have email that fails the DMARC mechanism check be treated by Mail Receivers as suspicious. Depending on the capabilities of the Mail Receiver, this can mean “place into spam folder”, “scrutinize with additional intensity”, and/or “flag as suspicious”.|
|The Domain Owner wishes for Mail Receivers to reject email that fails the DMARC mechanism check. Rejection should occur during the SMTP transaction.|
pct tag value is an integer percentage of messages from the Domain Owner’s mail stream to which the DMARC policy is to be applied. However, this must not be applied to the DMARC-generated reports, all of which must be sent and received unhindered. The purpose of the
pct tag is to allow Domain Owners to enact a slow rollout of enforcement of the DMARC mechanism. The implicit default value is 100.
I strongly recommend leaving
pct set at 100 (ideally by not setting it at all), and using the policies to slow the rollout of DMARC instead. That way, the same policy is applied to all mail, not an arbitrary percentage.
rua tag specifies comma-separated delivery locations for aggregate reports, which are typically daily emailed reports of compressed XML files that describe how many emails a Receiver observed from a given mail server IP address that failed the DMARC check.
ruf tag specifies comma-separated delivery locations for forensic reports, which are emails with copies of the original emails that failed the DMARC check attached. These are particularly useful for allowing Domain Owners to examine phishing emails sent third parties as well as their own organization, that were made took look like they came from their domain. However, many Receivers only send aggregate reports, if they send DMARC reports at all; others may only provide the email headers, and not the full content.
ruf tags are expected to have a value of a list of comma-separated DMARC URIs, the only URI type currently supported in the DMARC specification is
mailto. Each email address in the list must be prefixed with
mailto: in order to be a valid DMARC URI.
Additionally, if an email address in the
ruf tag values has a domain with a base domain that is different than the base domain that _dmarc record is a subdomain of, an authorization record must be added on the base domain of the email address. For example, if
example.com had a DMARC record like this:
_dmarc.example.net TXT "v=DMARC1; p=none; rua=mailto:email@example.com"
example.net does not match the base domain of
mail.example.com. receivers will check for the existence of the following record, before sending a report to
example.net._report._dmarc_example.com TXT "v=DMARC1"
example.com was using a third party service like Agari to receive, store, and process DMARC reports:
_dmarc.example.com TXT "v=DMARC1; p=none; rua=mailto:firstname.lastname@example.org ruf=mailto:email@example.com"
Then the following DNS records need to be added to the
agari.com DNS zone (which they do when a domain is added to an account):
example.com._report._dmarc_rua.agari.com TXT "v=DMARC1" example.com._report._dmarc_ruf.agari.com TXT "v=DMARC1"
These records are defined in RFC 7489, section 7.1. and are required when reports are directed external domains, so that someone can’t flood your DMARC inbox with reports for a domain that you do not operate by setting up a domain like
totallylegit.com, adding the record:
_dmarc.totallylegit.com TXT "v=DMARC1; p=none; rua=mailto:firstname.lastname@example.org; ruf=mailto:email@example.com"
And sending a ton of emails as
totallylegit.com that purposefully fail the DMARC check to many different organizations.
Return Path has an great series of blog posts of how to read DMARC reports. Check it out once you start getting reports.
What is DMARC alignment?
Passing a DMARC check requires passing SPF or DKIM checks, but DMARC goes beyond by also requiring that the identifiers for at least one of these checks are aligned, so spoofing is prevented.
|Passing||The signature in the DKIM header is validated using a public key that is published as a DNS record of the domain name specified in the signature||The mail server’s IP address is listed in the SPF record of the domain in the SMTP mail from envelope header|
|Alignment||The signing domain aligns with the domain in the message’s from header||The domain in the SMTP mail from envelope header aligns with the domain in the message’s from header|
DMARC without DKIM will break (some of) your email delivery
You must sign outgoing email at your gateway with a unique DKIM key before publishing a DMARC policy oa DMARC quarantine or reject policy. Simply allowing a sending server via SPF in not enough.
Why? Forwarded emails. Not the kinds of forwards that users send from their mail clients or webmail, but the forwards from mailbox rules. These forwards take place during SMTP delivery, and are common when someone changes addresses, or when a company gets acquired.
When forwards happen at the SMTP level, the envelope mail from will not match the header from, so the SPF indicators are not aligned, and the DMARC check will fail, unless the original sending mail server signed the message with DKIM, and the forwarding server(s) did not tamper with it. DKIM signatures are included in forwarded messages, because they are in the message headers.
The state of .gov in early 2018
I made some improvements to my scripts, and reran the tests on 2018-01-28 to see how the federal government had improved, now that the deadline to at least have a
p=none DMARC policy had passed.
Unfortunately, due to the bug fixes I’ve made in between tests, the results are not comparable, but these results are more accurate.
The generated files can be downloaded here.
|DMARC Policy (p value)||Count of Domains|
|Missing/Invalid DMARC record||645|
High profile .gov domains
Rather than trying to examine all 1,310 .gov domains in a single blog post, let’s focus on a much smaller subset of 44 domains from agencies of political, social, and/or bureaucratic significance:
|archives.gov||National Archives and Records Administration||Valid||p=reject|
|cbo.gov||Congressional Budget Office||Uses 13/10 DNS lookups||Missing record|
|cdc.gov||Centers for Disease Control||Valid||p=none|
|census.gov||US Census Bureau||Valid||doc.gov does not indicate that it accepts DMARC reports about census.gov|
|cms.gov||Centers for Medicare and Medicaid Services||Valid||p=none|
|commerce.gov||Department of Commerce||Valid||doc.gov does not indicate that it accepts DMARC reports about commerce.gov|
|consumerfinance.gov||Consumer Financial Protection Bureau||Missing record||cfpb.gov does not indicate that it accepts DMARC reports about consumerfinance.gov|
|doc.gov||Department of Commerce||Valid||p=none|
|doi.gov||Department of the Interior||Valid||p=none|
|dot.gov||Department of Transportation||Valid||p=none|
|ed.gov||Department of Education||Valid||p=none|
|epa.gov||Environmental Protection Agency||Valid||p=none|
|faa.gov||Federal Aviation Administration||Valid||p=none|
|fcc.gov||Federal Communication Commission||Valid||p=none|
|fda.gov||Food and Drug Administration||Valid||p=none|
|fdic.gov||Federal Deposit Insurance Corporation||Valid||p=reject|
|fema.dhs.gov||Federal Emergency Management Agency||Valid||p=none|
|fema.gov||Federal Emergency Management Agency||Valid||fema.dhs.gov does not indicate that it accepts DMARC reports about fema.gov|
|gsa.gov||General Services Agency||Uses 13/10 DNS lookups||pct value is less than 100. This leads to inconsistent and unpredictable policy enforcement.|
|healthcare.gov||Federal Healthcare Exchange||Valid||p=reject|
|hhs.gov||Department of Health and Human Services||The ptr mechanism should not be used – https://tools.ietf.org/html/rfc7208#section-5.5||p=none|
|hud.gov||Department of Housing and Urban Development||Valid||p=none|
|ice.gov||Immigration and Customs Enforcement||Missing record||Missing record|
|interior.gov||Department of the Interior||Valid||p=none|
|irs.gov||Internal Revenue Service||DNS servers fail to respond to query for include:qai.irs.gov||p=none|
|loc.gov||Library of Congress||Valid||Missing Record|
|mail.nasa.gov||National Aeronautics and Space Administration||Valid||p=none|
|nara.gov||National Archives and Records Administration||Valid||p=none|
|nasa.gov||National Aeronautics and Space Administration||Valid||p=none|
|ncua.gov||National Credit Union Administration||Valid||p=none|
|nhtsa.gov||National Highway Traffic Safety Administration||Valid||dot.gov does not indicate that it accepts DMARC reports about nhtsa.gov|
|nps.gov||National Park Service||Valid||p=none|
|ofdp.irs.gov||Internal Revenue Service||Valid||p=none|
|omb.gov||Office of Management and Budget||Valid||Invalid fo tag: Values 0 and 1 are mutually exclusive|
|sec.gov||Securities and Exchange Commission||Valid||p=none|
|ssa.gov||Social Security Administration||Valid||p=quarentine|
|uscis.gov||US Citizenship and Immigration Services||Missing record||Missing record|
|usda.gov||US Department Department of Agriculture||Valid||Missing record|
|uspto.gov||US Patent and Trademark Office||Valid||p=none|
|va.gov||Department of Veteran’s Affairs||Valid||p=none|
The US Intelligence Community
Some very significant agencies are missing from the list above, because they are members of the Intelligence Community (IC). DHS Binding Operational Directives do not apply to IC and/or National Security systems. However, SPF and DMARC would still be great security controls for them to deploy, as they are much more likely to be targets of spoofed phishing email than other agencies. At the very least, it could provide counterintelligence on phishing attempts sent to Unclassified email systems, because nothing like that has ever happened before. So, let’s check on the deployment of SPF and DMARC in the IC too.
|af.mil||US Air Force||Missing record||Missing record|
|army.mil||US Army||Missing record||Missing record|
|cia.gov||Central Intelligence Agency||Unnecessary SPF mechanisms: mx:cia.gov mx:ucia.gov||Missing record|
|dea.gov||Drug Enforcement Agency||Valid||p=none|
|dhs.gov||Department of Homeland Security||Valid||p=none|
|dia.mil||Defense Intelligence Agency||Valid||Missing record|
|dni.gov||Director of National Intelligence||Missing record||Missing record|
|dod.gov||Department of Defense||Missing record||Missing record|
|energy.gov||Department of Energy||Valid||p=none|
|fbi.gov||Federal Bureau of Investigation||Valid||p=reject|
|hq.dhs.gov||Department of Homeland Security||Valid||p=none|
|justice.gov||Department of Justice||Valid||p=none|
|mail.mil||Shared Department of Defense email system hosted by DISA||Valid||Unrelated TXT records were discovered. These should be removed, as some receivers may not expect to find unrelated TXT records at _dmarc.mail.mil|
v=spf2.0/pra include:_spf.eemsg.mail.mil ~all
|marines.mil||US Marine Corps||Missing record||Missing record|
|navy.mil||US Navy||Valid||Missing record|
|nga.mil||National Geospacial-Intelligence Agency||Missing record||Missing record|
|nro.gov||National Reconnaissance Office||Missing record||Missing record|
|nro.mil||National Reconnaissance Office||Missing record||Missing record|
|state.gov||Department of State||Valid||p=none|
|treasury.gov||Department of the Treasury||Valid||p=none|
|uscg.mil||US Coast Guard||Valid||Missing record|
|usdoj.gov||Department of Justice||Valid||p=none|
|usmc.mil||US Marine Corps||Valid||Missing record|
|whitehouse.gov||The White House||Valid||Missing record|
Shout out to the FBI for still being the only IC member with a DMARC policy currently set to
IC Support Agencies
These agencies provide services to IC agencies:
|disa.mil||Defense Information Systems Agency||Missing record||Missing record|
|dla.mil||Defense Logistics Agency||Valid||Missing record|
|opm.gov||Office of Personnel Management||usalearning.net has multiple spf1 TXT records||p=none|
Let’s check the major defense contractors for comparison. I also threw in Leidos, a spin-off of Lockheed Martian that provides information security consulting services to a variety of industries, ans SpaceX, to see how newer, companies compare to the established giants.
Several of these companies have separate domains for their email and web services. The email domains are protected by SPF and DMARC, but the web domains are not. Although those web domains do not host email services, those domains could still be useful to spoof for phishing.
|baesystems.com||BAE Systems||Missing record||Missing record|
|bah.com||Booz Allen Hamilton (Email domain)||Valid||p=none|
|boozallen.com||Booz Allen Hamilton||Missing record||Missing record|
|gd.com||General Dynamics||Valid||Missing record|
|l3-com.com||L3 Communications||Valid||Missing record|
|l3t.com||L3 Technologies||Valid||Missing record|
|lmco.com||Lockheed Martin (Email domain)||Valid||Missing record|
|lockheedmartin.com||Lockheed Martin||Missing record||Missing record|
|ngc.com||Northrop Grumman (Email domain)||Valid||Missing record|
|northropgrumman.com||Northrop Grumman||Missing record||Missing record|