Back to Learning CenterEmail Authentication

Secure email server: the standards that define one

By Samuel ChenardAugust 11, 202612 min read
Secure email server: the standards that define one

A secure email server is one that implements the standards controlling SMTP transport and sender identity: TLS negotiated with STARTTLS (RFC 3207), an enforceable transport policy through MTA-STS (RFC 8461), delivery visibility through TLS-RPT (RFC 8460), and sender authentication through SPF, DKIM, and DMARC. IT teams, MSPs, and anyone else running a mail transfer agent are affected, since none of these six controls is optional if the goal is a server that resists downgrade attacks and address spoofing.

At a glance

Quick takeaways

  • STARTTLS (RFC 3207, February 2002) negotiates TLS inside an existing SMTP session, but a compliant server must not require it for local delivery, which makes plain STARTTLS opportunistic and downgradable.
  • MTA-STS (RFC 8461, September 2018) lets a domain publish a policy telling senders to skip delivery when TLS negotiation, MX matching, or certificate validation fails, with three modes: none, testing, and enforce.
  • TLS-RPT (RFC 8460, September 2018) reports successful and failed TLS session counts back to the sending domain through a separate TXT record.
  • SPF (RFC 7208), DKIM (RFC 6376), and DMARC (RFC 7489) authenticate who may send as a domain; none of the three encrypts the SMTP transport or the message body.
  • DMARC is published as an Informational RFC, not a full Internet Standard; DKIM carries Internet Standard status as STD 76.

Who is affected?

This framework affects anyone who operates the server side of SMTP: internal IT teams running their own mail transfer agent, MSPs managing infrastructure for client domains, and administrators of a hosted or self-managed mail platform. It governs the server and the DNS records tied to a sending domain, not end-user encryption tools such as S/MIME, which NIST SP 800-177 Rev. 1 treats as a separate control layer for message-level confidentiality. A team looking for that narrower, client-side question should see how to send secure email in Outlook instead.

The standards apply at the domain level regardless of who operates the sending software. A domain that outsources outbound mail to a third-party service still needs its own SPF, DKIM, DMARC, and, where the provider does not already publish one, MTA-STS and TLS-RPT records, because these are DNS declarations tied to the domain rather than settings inside a vendor's dashboard. STARTTLS is different: it is a property of whichever mail transfer agent answers connections on port 25 or the submission ports 587 and 465, so an organization that still receives mail on its own server has to configure STARTTLS on that server directly.

None of RFC 3207, RFC 8461, RFC 8460, RFC 7208, RFC 6376, or RFC 7489 names a required vendor, operating system, or mail server product. Each is protocol level: it defines what a compliant server or DNS zone does, not which software runs it. A secure email gateway can implement several of these controls on a team's behalf, but the underlying DNS records still have to be published for the sending domain either way.

What are the requirements?

A secure email server, in verifiable protocol terms, implements six controls: negotiated transport encryption, an enforceable transport policy, transport reporting, and three sender-authentication records. Each comes from its own RFC.

Six standards a secure email server implements, the record each one publishes, and what it does
Source: Palisade.

Negotiate TLS with STARTTLS before message content moves

RFC 3207 defines STARTTLS as an SMTP service extension that lets a server and client "use TLS... to provide private, authenticated communication over the Internet." The EHLO keyword and the command are both STARTTLS, and the server signals readiness with the reply "220 Ready to start TLS":

Technical exampletext
EHLO client.example.com
250-mail.yourdomain.com Hello client.example.com
250-STARTTLS
250 OK
STARTTLS
220 Ready to start TLS

Illustrative session only; hostnames are examples. After negotiation, both sides discard everything learned in the plaintext phase, and the client should send EHLO again so the session restarts cleanly inside TLS, per RFC 3207 section 4.2.

RFC 3207 also states that a publicly referenced SMTP server "MUST NOT require use of the STARTTLS extension in order to deliver mail locally." Baseline STARTTLS is opportunistic by standard, not by accident, which makes it downgradable: an attacker who strips the "250 STARTTLS" line from the plaintext response, or rewrites the resolved MX record, can force a fallback to plaintext or redirect delivery entirely. RFC 8461 names that exact scenario as its motivation.

Publish an MTA-STS policy so senders can enforce TLS instead of trusting it

MTA-STS lets a mail service provider declare that it can receive TLS-secured SMTP and tell sending systems what to do when TLS cannot be negotiated. The policy has two published parts: a TXT record at _mta-sts. and a policy file fetched over HTTPS at a fixed path.

Technical exampletext
_mta-sts.yourdomain.com.  IN TXT  "v=STSv1; id=20260101000000Z"
Technical exampletext
version: STSv1
mode: testing
mx: mail.yourdomain.com
max_age: 604800

Illustrative record and policy file; the id value and mx host are examples, not values to publish as shown. MTA-STS defines exactly three modes. In enforce mode, RFC 8461 states that senders "MUST NOT deliver the message to hosts that fail MX matching or certificate validation or that do not support STARTTLS." In testing mode, failures are reported through TLS-RPT but mail still delivers. In none mode, the domain is treated as having no active policy.

MTA-STS relies on certificates from a public certificate authority and does not require DNSSEC. DANE covers the same goal through DNSSEC instead, and RFC 8461 is explicit that MTA-STS validation must never override a failing DANE validation.

Publish a TLS-RPT record to see which connections actually used TLS

RFC 8460 defines TLS-RPT as "a reporting mechanism and format by which sending systems can share statistics and specific information about potential failures" with recipient domains. A domain enables reporting with a TXT record whose rua tag names the report destination:

Technical exampletext
_smtp._tls.yourdomain.com.  IN TXT  "v=TLSRPTv1; rua=mailto:tlsrpt@yourdomain.com"

Illustrative record; the reporting mailbox is an example address. Aggregate reports carry per-policy counts, including total-successful-session-count and total-failure-session-count, so an operator can see both working and failing TLS delivery instead of assuming STARTTLS worked because nothing bounced.

Publish SPF, DKIM, and DMARC records so receivers can verify the sender

RFC 7208 has domains publish DNS records authorizing which hosts may use their name in the "MAIL FROM" or "HELO" identities; receivers test that authorization during the SMTP transaction. RFC 6376, Internet Standard STD 76, "permits a person, role, or organization to claim some responsibility for a message by associating a domain name with the message," validated through a cryptographic signature and a public key published at the signer's domain. RFC 7489 lets a domain owner "express domain-level policies and preferences for message validation, disposition, and reporting" on top of SPF and DKIM, published as a DNS TXT record at _dmarc.. RFC 7489 is Informational, not a full Internet Standard; keep that distinction when citing it.

Technical exampletext
yourdomain.com.                        IN TXT  "v=spf1 include:_spf.yourdomain-provider.com -all"
selector1._domainkey.yourdomain.com.   IN TXT  "v=DKIM1; k=rsa; p=MIIBIjANBgkqh..."
_dmarc.yourdomain.com.                 IN TXT  "v=DMARC1; p=none; rua=mailto:dmarc-reports@yourdomain.com"

Illustrative records only. The sending platform generates the real DKIM selector and key, and a DMARC report processor generates the real rua address; do not publish these example values on a live domain.

When does the requirement take effect?

None of these documents ships with a sender-specific enforcement deadline. Mailbox-provider deadlines and volume thresholds are set independently by each provider and change on their own schedule, which is a separate, provider-specific check rather than a fact fixed by the protocol itself. What each RFC does fix is its own publication date and standards status:

  • RFC 3207 (STARTTLS): Standards Track, February 2002.
  • RFC 6376 (DKIM): Internet Standard STD 76, September 2011.
  • RFC 7208 (SPF): Standards Track, April 2014.
  • RFC 7489 (DMARC): Informational, March 2015.
  • RFC 8460 (TLS-RPT): Standards Track, September 2018.
  • RFC 8461 (MTA-STS): Standards Track, September 2018.
  • NIST SP 800-177 Rev. 1: Final publication, February 2019.
A server built today still has to satisfy all six regardless of publication order. MTA-STS and TLS-RPT were designed specifically to close the gap STARTTLS's opportunistic design leaves open; SPF, DKIM, and DMARC address a separate problem, message origin, that transport encryption does not touch. For the transport-encryption side of this list in more depth, see email transport security.

How do I implement the requirement?

The following is implementation guidance built from the standards above, not a requirement any single RFC states in this order.

Five-step implementation for email server security standards
Source: Palisade.

1. Enable STARTTLS and install a valid certificate

On the mail transfer agent, turn on STARTTLS support and install a certificate issued by a publicly trusted certificate authority on ports 25, 587, and 465 as applicable. Confirm the server responds with "220 Ready to start TLS" and that it discards the prior plaintext session state, restarting with a fresh EHLO inside the encrypted channel, per RFC 3207 section 4.2.

2. Publish the MTA-STS TXT record and HTTPS policy file, starting in testing mode

Publish the _mta-sts TXT record and the policy file at the fixed .well-known path, with mode: testing first. Testing mode reports failures through TLS-RPT without blocking delivery, so a misconfigured policy does not silently drop mail.

3. Publish a TLS-RPT record before changing the MTA-STS mode

Add the _smtp._tls TXT record with a working rua address before moving MTA-STS out of testing. Reports arrive as aggregated counts, so allow enough sending volume to accumulate meaningful successful and failed session totals.

4. Publish SPF, DKIM, and DMARC records for the sending domain

Publish SPF at the domain apex, DKIM keys under the selector the sending platform assigns, and a DMARC record at _dmarc with a reporting address, starting DMARC at p=none so aggregate reports arrive before any policy change can affect delivery.

5. Move MTA-STS to enforce once TLS-RPT reports come back clean

After TLS-RPT aggregate reports show consistent successful-session counts with no unexplained failures, change the MTA-STS mode to enforce. From that point, RFC 8461 requires that compliant senders not deliver to hosts that fail MX matching, certificate validation, or STARTTLS.

How do I validate compliance?

Confirm each record resolves the way the standard requires before assuming the server behind it is compliant. Look up the DNS records for the sending domain to see the current SPF, DKIM, DMARC, MTA-STS, and TLS-RPT TXT records exactly as a public resolver returns them, and compare that output against the record shapes above.

A resolved TXT record only proves the record is published. It does not prove that a specific message used STARTTLS, matched the MTA-STS policy, or passed SPF and DKIM alignment. For that, pull the raw headers from an actual delivered message and check the TLS version reported in the Received header chain along with the Authentication-Results line the receiving server adds for SPF, DKIM, and DMARC. Then watch the TLS-RPT and DMARC aggregate reports over at least one full reporting cycle, since RFC 8460 aggregate reports carry separate successful and failed session counts and a single day's data can hide an intermittent downgrade.

A green certificate indicator on a vendor dashboard is not the same as a delivered-message check. Confirm TLS and authentication on a real message before treating a server as compliant.

Confirm what the sending domain actually publishes

Working through six RFCs does not show what is live on a specific domain today. Look up the DNS records for your sending domain to see the current SPF, DKIM, DMARC, MTA-STS, and TLS-RPT TXT records as public resolvers return them, before changing anything on the mail server itself.

A DNS lookup confirms what is published. It does not show which real sending sources are failing SPF or DKIM alignment against a published DMARC record, and that gap only closes once aggregate reports accumulate. Start with Palisade to have DMARC aggregate reports parsed into which sources are aligned and which are not, so a move from p=none toward enforcement is based on that evidence rather than a guess. Palisade analyzes DMARC aggregate-report data and proposes the next policy step; it does not configure STARTTLS, MTA-STS, or TLS-RPT on the mail server itself, and a human still reviews and applies any policy change it proposes.

Evidence

Sources and further reading

Questions readers ask

Frequently asked questions

Make email authentication easier to manage

Start in Palisade.

Get started

Share this article

Samuel Chenard

Written by

Samuel Chenard

CEO & Co-Founder, Palisade

Samuel Chenard is the CEO and co-founder of Palisade, AI-first DMARC software for IT teams and MSPs, from one domain to thousands.

More from Samuel

Related articles