Lessons Learned from the US Federal Government’s Ongoing Deployment of SPF and DMARC

SPF and DMARC are standards that describe how the origins of email messages should be verified, to prevent email spoofing. I spent some free time over the past few weeks creating checkdmarc , a Python 3 module and command-line interface that can validate and troubleshoot SPF and DMARC records across multiple domains, with the intent of building it into a web application that will process DMARC reports. The Department of Homeland Security recently launched an initiative to deploy SPF, DMARC, and other best practices on most federal agency domains by issuing BOD 18-01. This created the perfect case study of common challenges and mistakes when deploying SPF and DMARC across very large organizations, and even a few small ones.

2018-01-30 update: I have made many improvements to my script, corrected a few of my own misconceptions about DMARC I had in this post, and switched to updated results from 2018-01-28.

DHS Binding Operational Directive 18-01 (BOD 18-01)

On October 17, 2017 the US Department of Homeland Security (DHS) issued Binding Operational Directive 18-01 (BOD 18-01). This directive requires most federal agencies to deploy cyber hygiene best practices to secure their websites and email services, including SPF, DMARC, STARTTLS, and HTTPS with HTS:

  • “Within 30 days of BOD issuance (November 16th, 2017), submit an “Agency Plan of Action” to [email protected] and begin implementing the plan.
  • Within 90 days (January 15, 2018) of BOD issuance:
    • Configure all internet-facing mail servers to offer STARTTLS.
    • Configure all second-level domains to have valid SPF/DMARC records, with at minimum a DMARC policy of “p=none” and at least one address defined as a recipient of aggregate and/or failure reports.
  • By 120 days (February 13, 2018) after BOD issuance:
    • Ensure all publicly accessible Federal websites and web services provide service through a secure connection (HTTPS-only, with HSTS).
    • Identify agency second-level domains that can be HSTS preloaded, and provide a list to DHS at [email protected].
    • Disable SSLv2 and SSLv3 on web and mail servers.
    • Disable 3DES and RC4 ciphers on web and mail servers.
    • Within 15 days of the establishment of a centralized NCCIC reporting location, add DHS as a recipient of DMARC aggregate reports.
  • Within one year (October 16, 2018) of BOD issuance, set a DMARC policy of “reject” for all second-level domains and mail-sending hosts.”

Unfortunately, Intelligence Community and/or National Security systems are not required to comply with BODs, although they would still benefit from applying them.

Enabiling STARTTLS on mail servers allows senders to establish an encrypted connection before sending email. Enabling HTS on a web server tells clients to only use an encrypted (HTTPS) connection whenever they are accessing the site in the future. Disabling obsolete, weak cypher suites prevents easy exploits such as POODLE from working. SPF and DMARC are standards that work in tandem to prevent email spoofing. This blog post will focus on these standards. However, all of the best practices are key parts of defense in depth.

What is email spoofing?

Emails are spoofed when the from field of the message appears to be from someone other than the actual sender. For example, an attacker sending a phishing email can set the from address to match the target’s CEO. In many organizations, the target’s mail client will even helpfully display a known email address as a name and picture from their internal profile, which makes the phishing email look more legitimate.

Spoofing is a powerful, simple tool for attackers, but there are legitimate reasons for spoofing email from addresses. If an organization is using an outside email marketing service, helpdesk, or any other outside service that sends email, there is a need to make the items appear they are coming from their trusted domain, and not the third party, so recipients know who is responsible for the messages.

So how can a recipient mail server differentiate from legitimate and illegitimate spoofing? SPF, DKIM, and DMARC, of course!

What is SPF?

The Sender Policy Framework (SPF) is a standard for listing mail servers that are authorized to send email from your domain. It is defined in RFC 7208. The domain owner publishes a specially formatted TXT record in the DNS zone containing a list of authorized servers.

Details about the exact record format can be found here.

When a receiver mail server receives an email, it checks to see if the domain in the MAIL FROM part of the SMTP envelope has published a SPF record. If a record is found, the IP address of the sending mail server is compared to the list. Note that SPF does not check the from header that the user sees, and the SMTP envelope is not part of the message headers. so it is easy to bypass SPF and still spoof email to the user. The best illustration I have found of this comes from the great guide at o365info.com that explains exactly how to bypass SPF:

An illustration of email spoofing that bypasses SPF checks
An illustration of email spoofing that bypasses SPF checks
Credit: o365info.com
Used under fair use for educational purposes

Also, It is up the receiving mail server to decide what to do with an email if the SPF check fails. SPF does not specify an action for the recipient mail server to take in the event of a validation failure. DMARC fills these gaps, and more.

Common mistakes when deploying SPF and how to avoid them

Not deploying SPF (and DMARC) on domains that do not send email

An attacker can still spoof email to make it look as if it is coming from a domain even if it doesn’t have MX records and/or you don’t use it for email at all; these domains are the easiest to deploy SPF on: just add a TXT record containing "v=spf1 -all"

Including the same servers more than once

There are multiple mechanisms to allow a server, but a server only needs to be allowed by one of them. For example, if you use the ipv4 mechanism to state that a whole netblock is allowed allowed to send email, you don’t need to use an a mechanism to include a sever by hostname.

Using the wrong mechanism

The most common mechanisms in use are ipv4, ipv6, mx, a, and include

  • ipv4: used to specify a single IPv4 address or CIDR  netblock
  • ipv6:  used to specify a single IPv6 address or CIDR  netblock
  • mx: used to specify the servers listed in a domain’s MX resource records, not for allowing individual mail servers by hostname. If a mx mechanism includes a value (e.g. mx:mail.example.com), the MX records located at that value are queried; if a value is not included with the mechanism (e.g. mx), the domain’s own records are queried
  • a: used to specify a server by hostname, or a domain’s own A/AAAA records if no hostname is included with it; this is frequently used to allow web servers to send mail, since they are not included in MX records
  • include: used to include SPF mechanisms from another domain. This is frequently used to include lists of ipv4/ipv6 mechanisms from third party services that send email as the domain including cloud email and marketing services, such as Office 365, Google, or Salesforce; multiple include statements can bu used (e.g. include:spf.protection.outlook.com include:_spf.elasticemail.com) include mechanisms do not include the all value; to include mechanisms and the all value (e.g. to have a standard configuration for all of your domains), use the redirect= modifier at the end of the record – this modifier can only be used once per record

Using too many mechanisms that trigger DNS lookups

RFC 7208, section 4.6.4 limits the combined number of a, mx, include, exists,  and redirect mechanisms and modifiers used in a SPF record to 10, including any mechanisms from included records –  the total number of MX records returned per mx mechanism is also cannot exceed a separate limit of 10; this is done to prevent DNS-based denial-of-service (DoS) attacks

To reduce the number of DNS lookup mechanisms in a record:

  • If you have sending server IP addresses that won’t change often, consider using ipv4 or ipv6 mechanisms, which don’t trigger DNS lookups, instead of mx or a.
  • Have third party services send emails from dedicated subdomains; for example, if a billing services vendor currently sends emails as [email protected], have them send from [email protected], and make that email address an alias if you want to receive replies at [email protected]; this way billing.yourdomain.com gets its own SPF record and limits. Having MX records for these subdomains improves en emails trustworthiness to many gateways, even if you don’t actually have mailboxes for that subdomain.

Using the ptr mechanism

According to RFC 7208, section 5.5, the ptr mechanism should not be used

What is DKIM?

DKIM is a stronger method of validating that an email was sent from an authorized server than SPF.  The sending server signs outgoing mail with a private key, and places the signature in a message header and the receiving server validates the signature using a public key published in DNS, as defined in RFC 6376 – DomainKeys Identified Mail (DKIM) Signatures. Messages may have multiple DKIM signature headers if they have been forwarded through multiple gateways.

Take a look at this example, lightly modified from Wikipedia:

DKIM-Signature: v=1; a=rsa-sha256; d=example.net; s=brisbane;

     c=relaxed/simple; q=dns/txt; l=1234; t=1117574938; x=1118006938;

     h=from:to:subject:date:keywords:keywords;

     bh=MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=;

     b=dzdVyOfAKCdLXdJOc9G2q8LoXSlEniSbav+yuU4zGeeruD00lszZ

              VoG4ZHRNiYzR

A verifier queries the TXT resource record type of brisbane._domainkey.example.net.

Here, example.net is the author domain to be verified against (in the d tag), brisbane is a selector given in the s tag, and _domainkey is a fixed part of the protocol.

There are no CAs or revocation lists involved in DKIM key management. The selector is a straightforward method to allow signers to add and remove keys whenever they wish. Long lasting signatures for archival purposes are outside DKIM’s scope. Other tags are visible in the example are:

  • v – version
  • a – signing algorithm
  • d – domain
  • s – selector
  • c – canonicalization algorithm(s) for header and body
  • q – default query method
  • l – length of the canonicalized part of the body that has been signed
  • t – signature timestamp
  • x – expire time
  • h – list of signed header fields, repeated for fields that occur multiple times

What is DMARC?

DMARC is the Domain-based Message Authentication, Reporting, and Conformance standard, as described in RFC 7489. It consists of a specially-formatted DNS TXT resource record published by a Domain Owner, that tells mail servers what to do when they encounter a message that appears to be from that domain, but fails SPF and/or DKIM. It also provides methods for mail receivers to notify Domain Owners when mail fails the DMARC check.

DMARC records are TXT resource records of a subdomain of a second-level domain called _dmarc, i.e. _dmarc.example.com. Unlike SPF, DMARC options apply to all subdomains, under the base domain (i.e. example.com). DMARC records are composed of tags and values that are joined by =, and delimited  by ;. All DMARC records start with v=DMARC1.

The most important and commonly used DMARC tags are p, sp, pct, rua, and ruf.

The p tag sets the DMARC policy, which specifies the policy to be enacted by the Receiver at the request of the Domain Owner on mail that is supposedly from the Domain Owner’s domain fails the DMARC check. The policy applies to the domain and to its subdomains, unless subdomain policy is explicitly described using the sp tag. Possible policy values include:

Policy Description
none The Domain Owner requests no specific action be taken regarding delivery of messages. This is the implicit default policy.
quarantine The Domain Owner wishes to have email that fails the  DMARC mechanism check be treated by Mail Receivers as suspicious. Depending on the capabilities of the Mail Receiver, this can mean “place into spam folder”, “scrutinize with additional intensity”, and/or “flag as suspicious”.
reject  The Domain Owner wishes for Mail Receivers to reject email that fails the DMARC mechanism check. Rejection should occur during the SMTP transaction.

The pct tag value is an integer percentage of messages from the Domain Owner’s mail stream to which the DMARC policy is to be applied. However, this must not be applied to the DMARC-generated reports, all of which must be sent and received unhindered. The purpose of the pct tag is to allow Domain Owners to enact a slow rollout of enforcement of the DMARC mechanism. The implicit default value is 100.

I strongly recommend leaving pct set at 100 (ideally by not setting it at all), and using the policies to slow the rollout of DMARC instead. That way, the same policy is applied to all mail, not an arbitrary percentage.

The rua tag specifies comma-separated delivery locations for aggregate reports, which are typically daily emailed reports of compressed XML files that describe how many emails a Receiver observed from a given mail server IP address that failed the DMARC check.

The ruf tag specifies comma-separated delivery locations for forensic reports, which are emails with copies of the original emails that failed the DMARC check attached. These are particularly useful for allowing Domain Owners to examine phishing emails sent third parties as well as their own organization, that were made took look like they came from their domain. However,  many Receivers only send aggregate reports, if they send DMARC reports at all; others may only provide the email headers, and not the full content.

Both rua and ruf tags are expected to have a value of a list of comma-separated DMARC URIs, the only URI type currently supported in the DMARC specification is mailto. Each email address in the list must be prefixed with mailto: in order to be a valid DMARC URI.

Additionally, if an email address in the rua and ruf tag values has a domain with a base domain that is different than the base domain that _dmarc record is a subdomain of, an authorization record must be added on the base domain of the email address. For example, if example.com had a DMARC record like this:

_dmarc.example.net TXT "v=DMARC1; p=none; rua=mailto:[email protected]"

because example.net does not match the base domain of mail.example.com. receivers will check for the existence of the following record, before sending a report to [email protected]:

example.net._report._dmarc_example.com TXT "v=DMARC1"

Likewise, if example.com was using a third party service like Agari to receive, store, and process DMARC reports:

_dmarc.example.com TXT "v=DMARC1; p=none; rua=mailto:[email protected] ruf=mailto:[email protected]"

Then the following DNS records need to be added to the agari.com DNS zone (which they do when a domain is added to an account):

example.com._report._dmarc_rua.agari.com TXT "v=DMARC1"
example.com._report._dmarc_ruf.agari.com TXT "v=DMARC1"

These records are defined in RFC 7489, section 7.1. and are required when reports are directed external domains, so that someone can’t flood your DMARC inbox with reports for a domain that you do not operate by setting up a domain like totallylegit.com,  adding the record:

_dmarc.totallylegit.com TXT "v=DMARC1; p=none; rua=mailto:[email protected]; ruf=mailto:[email protected]"

And sending a ton of emails as totallylegit.com that purposefully fail the DMARC check to many different organizations.

Return Path has an great series of blog posts of how to read DMARC reports. Check it out once you start getting reports.

What is DMARC alignment?

Passing a DMARC check requires passing SPF or DKIM checks, but DMARC goes beyond by also requiring that the identifiers for at least one of these checks are aligned, so spoofing is prevented.

DKIM SPF
Passing The signature in the DKIM header is validated using a public key that is  published as a DNS record of the domain name specified in the signature The mail server’s IP address is listed in the SPF record of the domain in the SMTP mail from envelope header
Alignment The signing domain aligns with the domain in the message’s from header The domain in the SMTP mail from envelope header aligns with the domain in the message’s from header

DMARC without DKIM will break (some of) your email delivery

You must sign outgoing email at your gateway with a unique DKIM key before publishing a DMARC policy oa DMARC quarantine or reject policy. Simply allowing a sending server via SPF in not enough.

Why? Forwarded emails. Not the kinds of forwards that users send from their mail clients or webmail, but the forwards from mailbox rules. These forwards take place during SMTP delivery, and are common when someone changes addresses, or when a company gets acquired.

When forwards happen at the SMTP level, the envelope mail from will not match the header from, so the SPF indicators are not aligned, and the DMARC check will fail, unless the original sending mail server signed the message with DKIM, and the forwarding server(s) did not tamper with it. DKIM signatures are included in forwarded messages, because they are in the message headers.

The state of .gov in early 2018

I made some improvements to my scripts, and reran the tests on 2018-01-28 to see how the federal government had improved, now that the deadline to at least have a p=none DMARC policy had passed.

Unfortunately, due to the bug fixes I’ve made in between tests, the results are not comparable, but these results are more accurate.

The generated files can be downloaded here.

A pie chart of SPF deployment on .gov domains as of 2018-01-28
A pie chart of SPF deployment on .gov domains as of 2018-01-28
A pie chart of DMARC deployment on .gov domains as of 2018-01-28
A pie chart of DMARC deployment on .gov domains as of 2018-01-28
A column chart of DMARC deployment by policy on .gov domains as of 2018-01-28; Note: “none” is a valid DMARC policy – the “(empty)” chart value represents domains that have missing or invalid DMARC records
A column chart of DMARC deployment by policy on .gov domains as of 2018-01-28;
Note: “none” is a valid DMARC policy – the “(empty)” chart value represents domains that have missing or invalid DMARC record
Value SPF DMARC
Invalid/Missing 429 645
Valid 881 665
DMARC Policy (p value) Count of Domains
none 463
quarantine 10
reject 192
Missing/Invalid DMARC record 645

High profile .gov domains

Rather than trying to examine all 1,310 .gov domains in a single blog post, let’s focus on a much smaller subset of 44 domains from agencies of political, social, and/or bureaucratic significance:

Domain Agency SPF DMARC
archives.gov National Archives and Records Administration Valid p=reject
cbo.gov Congressional Budget Office Uses 13/10 DNS lookups Missing record
cdc.gov Centers for Disease Control Valid p=none
census.gov US Census Bureau Valid doc.gov does not indicate that it accepts DMARC reports about census.gov
cms.gov Centers for Medicare and Medicaid Services Valid p=none
commerce.gov Department of Commerce Valid doc.gov does not indicate that it accepts DMARC reports about commerce.gov
consumerfinance.gov Consumer Financial Protection Bureau Missing record cfpb.gov does not indicate that it accepts DMARC reports about consumerfinance.gov
doc.gov Department of Commerce Valid p=none
doi.gov Department of the Interior Valid p=none
dot.gov Department of Transportation Valid p=none
ed.gov Department of Education Valid p=none
epa.gov Environmental Protection Agency Valid p=none
faa.gov Federal Aviation Administration Valid p=none
fcc.gov Federal Communication Commission Valid p=none
fda.gov Food and Drug Administration Valid p=none
fdic.gov Federal Deposit Insurance Corporation Valid p=reject
federalreserve.gov Federal Reserve Valid p=reject
fema.dhs.gov Federal Emergency Management Agency Valid p=none
fema.gov Federal Emergency Management Agency Valid fema.dhs.gov does not indicate that it accepts DMARC reports about fema.gov
gsa.gov General Services Agency Uses 13/10 DNS lookups pct value is less than 100. This leads to inconsistent and unpredictable policy enforcement.
healthcare.gov Federal Healthcare Exchange Valid p=reject
hhs.gov Department of Health and Human Services The ptr mechanism should not be used – https://tools.ietf.org/html/rfc7208#section-5.5 p=none
hud.gov Department of Housing and Urban Development Valid p=none
ice.gov Immigration and Customs Enforcement Missing record Missing record
interior.gov Department of the Interior Valid p=none
irs.gov Internal Revenue Service DNS servers fail to respond to query for include:qai.irs.gov p=none
loc.gov Library of Congress Valid Missing Record
mail.nasa.gov National Aeronautics and Space Administration Valid p=none
medicaid.gov Medicaid Valid p=reject
medicare.gov Medicare Valid p=none
nara.gov National Archives and Records Administration Valid p=none
nasa.gov National Aeronautics and Space Administration Valid p=none
ncua.gov National Credit Union Administration Valid p=none
nhtsa.gov National Highway Traffic Safety Administration Valid dot.gov does not indicate that it accepts DMARC reports about nhtsa.gov
nps.gov National Park Service Valid p=none
ofdp.irs.gov Internal Revenue Service Valid p=none
omb.gov Office of Management and Budget Valid Invalid fo tag: Values 0 and 1 are mutually exclusive
sec.gov Securities and Exchange Commission Valid p=none
ssa.gov Social Security Administration Valid p=quarentine
uscis.gov US Citizenship and Immigration Services Missing record Missing record
usda.gov US Department Department of Agriculture Valid Missing record
usmint.gov US Mint Valid p=none
uspto.gov US Patent and Trademark Office Valid p=none
va.gov Department of Veteran’s Affairs Valid p=none

The US Intelligence Community

Some very significant agencies are missing from the list above, because they are members of the Intelligence Community (IC). DHS Binding Operational Directives do not apply to IC and/or National Security systems. However, SPF and DMARC would still be great security controls for them to deploy, as they are much more likely to be targets of spoofed phishing email than other agencies. At the very least, it could provide counterintelligence on phishing attempts sent to Unclassified email systems, because nothing like that has ever happened before. So, let’s check on the deployment of SPF and DMARC in the IC too.

Domain Agency SPF DMARC
af.mil US Air Force Missing record Missing record
army.mil US Army Missing record Missing record
cia.gov Central Intelligence Agency Unnecessary SPF mechanisms: mx:cia.gov mx:ucia.gov Missing record
dea.gov Drug Enforcement Agency Valid p=none
dhs.gov Department of Homeland Security Valid p=none
dia.mil Defense Intelligence Agency Valid Missing record
dni.gov Director of National Intelligence Missing record Missing record
dod.gov Department of Defense Missing record Missing record
energy.gov Department of Energy Valid p=none
fbi.gov Federal Bureau of Investigation Valid p=reject
hq.dhs.gov Department of Homeland Security Valid p=none
justice.gov Department of Justice Valid p=none
mail.mil Shared Department of Defense email system hosted by DISA Valid Unrelated TXT records were discovered. These should be removed, as some receivers may not expect to find unrelated TXT records at _dmarc.mail.mil

v=spf2.0/pra include:_spf.eemsg.mail.mil ~all

marines.mil US Marine Corps Missing record Missing record
navy.mil US Navy Valid Missing record
nga.mil National Geospacial-Intelligence Agency Missing record Missing record
nro.gov National Reconnaissance Office Missing record Missing record
nro.mil National Reconnaissance Office Missing record Missing record
state.gov Department of State Valid p=none
treasury.gov Department of the Treasury Valid p=none
uscg.mil US Coast Guard Valid Missing record
usdoj.gov Department of Justice Valid p=none
usmc.mil US Marine Corps Valid Missing record
whitehouse.gov The White House Valid Missing record

Shout out to the FBI for still being the only IC member with a DMARC policy currently set to reject!

IC Support Agencies

These agencies provide services to IC agencies:

Domain Agency SPF DMARC
disa.mil Defense Information Systems Agency Missing record Missing record
dla.mil Defense Logistics Agency Valid Missing record
opm.gov Office of Personnel Management usalearning.net has multiple spf1 TXT records p=none

Defense contractors

Let’s check the major defense contractors for comparison. I also threw in Leidos, a spin-off of Lockheed Martian that provides information security consulting services to a variety of industries, and SpaceX, to see how newer, companies compare to the established giants.

Several of these companies have separate domains for their email and web services. The email domains are protected by SPF and DMARC, but the web domains are not. Although those web domains do not host email services, those domains could still be useful to spoof for phishing.

Domain Company SPF DMARC
baesystems.com BAE Systems Missing record Missing record
bah.com Booz Allen Hamilton (Email domain) Valid p=none
boeing.com Boeing Valid Missing record
boozallen.com Booz Allen Hamilton Missing record Missing record
gd.com General Dynamics Valid Missing record
l3-com.com L3 Communications Valid Missing record
l3t.com L3 Technologies Valid Missing record
leidos.com Leidos Valid Missing record
lmco.com Lockheed Martin (Email domain) Valid Missing record
lockheedmartin.com Lockheed Martin Missing record Missing record
ngc.com Northrop Grumman (Email domain) Valid Missing record
northropgrumman.com Northrop Grumman Missing record Missing record
palantir.com Palantir Valid Missing record
raytheon.com Raytheon Valid p=none
spacex.com SpaceX Valid p=none
utc.com United Technologies Valid p=none

6 thoughts on “Lessons Learned from the US Federal Government’s Ongoing Deployment of SPF and DMARC”

  1. Great post! Very interested in seeing update since Jan 15 BOD 18-01 deadline for DMARC records & aggregate reporting. Could you shoot me an email….would like to discuss this further with you. Thanks!

    Reply
  2. Would you please post an update page to your blog so we can see the results to date. It is almost a year later, and it would be useful to see how the effort to implement DHS Binding Operational Directive 18-01 (BOD 18-01) is going.

    Thank you.

    Reply
  3. very helpful post !! but how the directive force US agencies to implement SPF and DMARC( with p= reject) with DKIM instructions, as you mention that “DMARC without DKIM will break (some of) your email delivery”, could you explain for us this misunderstanding.
    Thanks a lots

    Reply
  4. updated comment:

    very helpful post !! but how the directive force US agencies to implement SPF and DMARC( with p= reject) without DKIM instructions, as you mention that “DMARC without DKIM will break (some of) your email delivery”, could you explain for us this misunderstanding.
    Thanks a lots

    Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.