How to deploy DANE for SMTP with DNSSEC and TLSA without creating mail black holes

danednssectlsasmtptutorial

DANE can make inbound SMTP resistant to certificate substitution and STARTTLS downgrade attacks. It can also turn a small DNS or certificate mistake into a growing queue of delayed mail.

That is not a reason to avoid DANE. It is a reason to deploy it as a mail-routing change, not as a single TLS checkbox. The safe approach is to inventory every MX path, sign and validate the DNS zone, publish TLSA associations for the certificates actually served, roll out in stages, and monitor delivery queues as carefully as DNS and TLS.

This guide uses example.com and mx1.example.com as documentation-only names. Replace them with the real names in your environment.

What DANE changes in SMTP

Normal SMTP TLS is opportunistic. The sender connects to an MX host, sees whether that host advertises STARTTLS, and tries to encrypt the connection. An attacker who can interfere with the connection may remove the STARTTLS capability. An attacker who can tamper with unsigned DNS may also redirect the sender to a different MX host.

RFC 7672 uses DNSSEC-validated TLSA records to publish two related facts:

  1. The receiving service is committed to supporting TLS on that port.
  2. The certificate or public key that the service presents can be authenticated through DNSSEC.

DANE for SMTP is normally opportunistic DANE. A DANE-aware sender uses authenticated TLS when it finds a usable, DNSSEC-secure TLSA record. A sender that can securely determine that no TLSA record exists can continue with ordinary opportunistic TLS. A sender that does not implement DANE may behave as it did before.

The important exception is a DNS failure. If a DANE-aware sender cannot validate the DNS data needed to find the TLSA record, it must not pretend that the record is absent and continue to the same host without the DANE check. It skips that host or delays delivery. That protects against downgrade, but it also means a broken DNSSEC deployment can affect mail delivery.

DANE does not authenticate the From: header and does not replace SPF, DKIM, or DMARC. It protects one SMTP hop between a sending MTA and a receiving MTA. For a comparison with another inbound transport policy, see DANE vs MTA-STS for SMTP.

Define the deployment boundary

Before changing DNS, decide what the deployment is intended to protect:

  • Opportunistic DANE for Internet mail: publish TLSA records and allow compatible senders to use them. This is the normal incremental deployment model.
  • Mandatory DANE for selected partners: configure a sending MTA to require DANE-verified delivery to specific domains or next hops. This is a bilateral operational policy, not something the receiving domain can impose on every Internet sender by publishing TLSA.
  • Mandatory TLS for all inbound mail: this is a much stronger operational requirement. It needs an explicit plan for senders that do not support DANE, and it must include queue, retry, and exception handling.

Do not confuse "publish TLSA" with "all mail must use authenticated TLS." Publishing a correct TLSA record improves security for DANE-capable senders, but it cannot change a legacy sender's behavior.

Step 1: Inventory every MX path

Start with the receiving path, not the certificate. List every MX record and record the following for each target:

  • MX preference and hostname
  • IPv4 and IPv6 addresses
  • listening port and any custom transport route
  • certificate and private-key owner
  • certificate renewal process
  • load balancer, proxy, or regional backend involved
  • backup and disaster-recovery path
  • whether the service supports SMTP STARTTLS reliably on every address

For a domain such as example.com, the initial inventory might look like this:

example.com. IN MX 10 mx1.example.com.
example.com. IN MX 20 mx2.example.com.

The TLSA records belong below the MX hostnames, not automatically below the recipient domain:

_25._tcp.mx1.example.com. IN TLSA ...
_25._tcp.mx2.example.com. IN TLSA ...

Every MX that can receive mail must be included in the test plan. A backup MX with an old certificate or no TLSA record is not a harmless detail. A DANE-aware sender can reach it when the preferred MX is unavailable, so it is part of the security and delivery boundary.

Check IPv6 separately. It is common for an IPv4 listener to have the right certificate while the IPv6 listener reaches an older host or a different TLS terminator.

Step 2: Establish a healthy DNSSEC chain

DNSSEC authenticates the DNS data that carries the TLSA association. It does not encrypt DNS traffic, and it does not make a broken zone available. RFC 4033 describes the DNSSEC services and limitations, including authenticated denial of existence.

Before publishing TLSA:

  1. Enable signing for the authoritative zone that contains the MX and TLSA names.
  2. Publish the correct DS record at the parent delegation.
  3. Confirm that authoritative servers serve current DNSKEY, RRSIG, and denial-of-existence records.
  4. Check that all authoritative nameservers serve the same signed data.
  5. Test both positive answers and negative answers through independent validating resolvers.
  6. Monitor signature expiry, key rollovers, DS changes, authoritative reachability, and clock synchronization.

Use DNSSEC-aware queries during the check:

dig +dnssec example.com MX
dig +dnssec mx1.example.com A
dig +dnssec _25._tcp.mx1.example.com TLSA
delv example.com MX
delv _25._tcp.mx1.example.com TLSA

The exact command output depends on the resolver. What matters is that the resolver can distinguish a validated answer from an unsigned answer and from a validation failure. Do not treat the presence of an RRSIG in a response as proof that the answer validated. The resolver must validate the complete chain back to a trust anchor.

Pay attention to SERVFAIL and bogus responses. A signed zone with an expired signature, mismatched DS, incomplete key rollover, or inconsistent authoritative servers may work from one cache and fail from another. That is precisely the kind of split behavior that creates a mail incident.

Step 3: Make SMTP TLS correct before DANE

DANE does not repair SMTP TLS. First make the live service behave correctly without relying on TLSA:

  • STARTTLS is advertised only where the service can complete the handshake.
  • The service presents the intended leaf certificate and the required intermediate chain.
  • The same certificate and key are deployed to every MX node and address family, unless different associations are deliberately published.
  • Old protocol versions and weak cipher choices are disabled according to the MTA's support policy.
  • The certificate is renewed before expiry, with enough overlap for DNS and service caches.
  • The SMTP service does not require a browser-style interaction or manual certificate exception.

Inspect the live certificate from outside the network:

openssl s_client \
  -starttls smtp \
  -connect mx1.example.com:25 \
  -servername mx1.example.com \
  -showcerts

Repeat this against every public IP and every MX hostname. If a load balancer selects different certificates by SNI, test with the SNI value that the receiving service expects and verify that DANE-capable SMTP clients use compatible behavior.

Step 4: Choose the TLSA association

A TLSA record has three numeric fields before its association data:

certificate usage selector matching type association data

RFC 6698 defines the original fields:

FieldValues used in practice
Certificate usage0 PKIX-TA, 1 PKIX-EE, 2 DANE-TA, 3 DANE-EE
Selector0 full certificate, 1 SubjectPublicKeyInfo
Matching type0 exact data, 1 SHA-256, 2 SHA-512

For a service that wants to bind directly to its presented public key, 3 1 1 is a common choice. It means DANE end-entity association, match the DER-encoded SubjectPublicKeyInfo, and compare its SHA-256 digest. Usage 3 does not require the certificate to chain to a public CA. The TLSA association itself is trusted through DNSSEC.

Other usages are valid when their trust model is intentional. Usage 1 still requires PKIX validation in addition to the TLSA match. Usage 2 uses a TLSA certificate or key as a trust anchor. Usage 0 constrains the certification authority path. Do not select usage 0, 1, or 2 simply because the certificate is publicly trusted. The sender's DANE implementation and the certificate lifecycle must be able to satisfy the additional rules.

The selector is not a DKIM selector. With selector 0, the association is made from the complete DER certificate. With selector 1, it is made from the certificate's DER SubjectPublicKeyInfo. A certificate renewal that keeps the same public key can preserve a 3 1 1 association. A renewal that generates a new key cannot.

For DANE-EE(3), the TLSA match authenticates the end-entity certificate association. The usual public-CA hostname check is not the source of trust for that usage under RFC 7672, although keeping the MX hostname in the certificate SAN is still a sensible interoperability and operations practice. For usages where PKIX and name checks apply, verify the MX hostname and the relevant SMTP reference identifiers as well.

Step 5: Generate the association from the live key

Never type the digest by hand and never calculate it from a PEM file's text. TLSA matching uses the selected certificate material in DER form.

For a full-certificate association, obtain the leaf certificate and calculate its digest from DER:

openssl s_client -starttls smtp \
  -connect mx1.example.com:25 \
  -servername mx1.example.com 2>/dev/null \
  | openssl x509 -outform DER > /tmp/mx1-cert.der

openssl dgst -sha256 /tmp/mx1-cert.der

For a SubjectPublicKeyInfo association, extract the public key and encode that structure as DER before hashing it:

openssl s_client -starttls smtp \
  -connect mx1.example.com:25 \
  -servername mx1.example.com 2>/dev/null \
  | openssl x509 -pubkey -noout > /tmp/mx1-pubkey.pem

openssl pkey -pubin -in /tmp/mx1-pubkey.pem -outform DER \
  | openssl dgst -sha256

Use the hexadecimal digest from a tested TLSA generator or your MTA's documented tooling when possible. Compare the result with the certificate or key installed on every node. The digest in a documentation example is not a usable production value.

Publish the record at the port and transport actually used:

_25._tcp.mx1.example.com. IN TLSA 3 1 1 <sha256-of-subjectpublickeyinfo>

If the MTA uses a nonstandard next hop or port, the port label must match that connection. _25._tcp is not a generic label for every SMTP route.

Step 6: Publish complete TLSA coverage

Publish at least one usable TLSA record for each MX host that is meant to offer DANE. If two MX hosts use different keys, each gets its own association. If one service deliberately supports two certificates during a transition, the RRset may contain both associations:

_25._tcp.mx1.example.com. IN TLSA 3 1 1 <old-key-digest>
_25._tcp.mx1.example.com. IN TLSA 3 1 1 <new-key-digest>

Multiple TLSA records are alternatives. The sender succeeds if the certificate matches one usable association. They are not combined into a multi-part credential.

Do not publish a TLSA record before the corresponding service supports TLS. A DNSSEC-secure TLSA RRset signals that the host is committed to TLS. A sender that sees such a record must not silently deliver to that host without TLS just because the handshake failed.

Do not publish an empty or malformed TLSA record as a testing signal. A secure TLSA RRset containing only unusable records still signals TLS support to conforming SMTP clients, and it can cause delivery delays without giving them a usable certificate association.

Step 7: Roll out in stages

DANE has no equivalent to MTA-STS testing mode. The practical staging mechanism is to make the service and DNS correct before advertising the association, then publish it for one complete and observable change window.

Stage A: Preflight

  • Sign and validate DNSSEC.
  • Confirm every MX target, A and AAAA address, certificate, and TLSA digest.
  • Test SMTP TLS from independent networks.
  • Verify backup-MX behavior, not just the primary path.
  • Confirm that the MTA queues and retries when a remote host is temporarily unavailable.
  • Set up TLS-RPT and local alerts before publishing TLSA.

Stage B: Prepare the new service

Install the certificate and key on every relevant node. If this is a certificate renewal with a new key, serve the new certificate everywhere before adding its TLSA association. Keep the old certificate available until the new path has been externally tested.

If the platform cannot deploy the same key everywhere, use separate TLSA associations and test each node. A DNS record covering only the node that was tested is not complete coverage.

Stage C: Publish TLSA

Publish the TLSA RRsets with a deliberate TTL and allow the authoritative servers to serve the new signed data. Check the records through more than one validating resolver, from more than one network, and verify the AD or equivalent validation result where the resolver exposes it.

Then test the live SMTP handshake and compare the presented key with the published association. Test both the normal MX and the backup MX. Check the mail queue and TLS-RPT reports during the first delivery window.

Stage D: Observe before changing anything else

Watch for:

  • DANE validation failures
  • TLS handshake and certificate mismatch errors
  • DNSSEC SERVFAIL or bogus results
  • increased remote-delivery queue age
  • retries that previously succeeded
  • failures isolated to IPv4, IPv6, a region, or one MX host
  • non-delivery reports from senders whose retry windows expired

Do not combine the first TLSA publication with an MX migration, DNS provider migration, certificate authority change, and MTA upgrade. If something fails, a single controlled change is much easier to diagnose.

Step 8: Monitor the policy and the mail path

Publish SMTP TLS Reporting at the recipient domain if you can receive and review aggregate reports:

_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@example.com"

TLS-RPT is useful for compatible senders, but it is not universal telemetry. Keep local monitoring as well:

  • DNSSEC validation and signature-expiry alerts
  • authoritative DNS availability from independent probes
  • MX, A, AAAA, and TLSA consistency checks
  • certificate expiry and key fingerprint checks on every node
  • SMTP STARTTLS availability checks
  • TLS handshake and DANE result counters in the MTA
  • queue size, oldest-message age, retry reason, and final bounce counts

Group reports by sending organization, MX hostname, address family, and failure type. A problem affecting only mx2.example.com or only IPv6 is much more actionable than one combined "DANE failed" number.

Keep the original MX hostname in delivery logs. If an MTA follows an alias or a CNAME, the expanded name is not a substitute for the logical MX target when investigating whether the intended destination was reached.

Certificate rollover without breaking delivery

Certificate renewal is where many DANE deployments fail. The safe sequence for a new key is:

  1. Generate the new certificate and key.
  2. Calculate the new TLSA association from the exact certificate or SPKI that will be served.
  3. Add the new TLSA record while retaining the old record.
  4. Wait for the new signed RRset to be available through authoritative servers and validating resolvers.
  5. Deploy the new certificate and key to every MX node.
  6. Verify that each node presents a certificate matching either the old or new association.
  7. Remove the old TLSA record only after the old certificate is no longer served and the relevant DNS and signature cache windows have passed.

The overlap is important. If the server changes first, a sender with the old TLSA data rejects the new certificate. If DNS changes first and the server changes inconsistently, a sender with the new TLSA data rejects nodes still serving the old certificate. Overlapping associations allow either key during the transition.

The same principle applies to a DNSSEC key rollover. Follow the DNS provider or signer implementation's rollover procedure, keep the DS and DNSKEY chain valid throughout, and test validation from outside the organization. Do not assume that a successful query from a local resolver proves that every cache sees a valid chain.

What causes a mail black hole

A mail black hole is not only an immediate SMTP rejection. It can be a delayed failure that looks healthy until sender retry limits are exhausted. Common causes include:

  • publishing TLSA for mx1.example.com while the MX points to mx1.example.net
  • hashing the PEM text instead of the DER certificate or SubjectPublicKeyInfo
  • publishing a digest for a certificate that is not served on IPv6
  • replacing the private key before adding the new TLSA association
  • removing the old TLSA record before old caches expire
  • letting an RRSIG expire or publishing a DS record that does not match the zone DNSKEY
  • omitting a backup MX from the TLSA inventory
  • using the wrong port or _udp instead of _tcp
  • changing the MX set without keeping old paths available for cache and retry windows
  • assuming a non-DANE sender will enforce the published record
  • forcing mandatory DANE globally without confirming sender and partner support

The dangerous response is to delete TLSA immediately and declare the issue fixed. Deleting a record can itself be cached, and it may turn an authenticated failure into unauthenticated delivery for senders that later see the absence. First identify whether the issue is DNSSEC validation, TLSA lookup, certificate presentation, MX selection, or sender implementation.

Fallback and operational safeguards

Use these rules to preserve delivery while keeping the security meaning of DANE intact:

  1. Treat a validated absence differently from a DNS error. A DNSSEC-secure denial that no TLSA record exists means DANE is not published for that host. A timeout, SERVFAIL, or DNSSEC bogus result means the sender cannot safely determine the TLSA state. The former can fall back to ordinary opportunistic TLS; the latter must skip the host or defer delivery.
  2. Keep MX preferences meaningful. A lower-priority MX with TLSA must not preempt a higher-priority MX merely because it offers stronger security. If all MX hosts are intended to be secure, publish and test all of them instead of relying on a supposedly safer backup.
  3. Queue rather than reroute around an authentication failure. A DANE-aware sender should retry a temporarily unreachable host or use another valid MX. It should not send the same message to an unrelated fallback that has not been authorized by the domain's MX design.
  4. Keep retry windows longer than maintenance windows. Confirm that the sender's queue and retry policy can survive DNS propagation, certificate deployment, and an incident response. A short retry window can turn a repairable outage into a bounce.
  5. Use emergency changes deliberately. If every MX is failing, restore the certificate, key, DNSSEC chain, or TLSA data that was intended. If an emergency removal is considered, record the security tradeoff, account for cached records and negative caching, and restore the association as soon as the service is healthy.
  6. Limit mandatory DANE to known requirements. Configure mandatory DANE for a partner only after both sides have tested DNSSEC validation, TLSA matching, certificate rollover, and queue behavior. Do not use a global mandatory setting as a substitute for staged deployment.

These safeguards do not make a broken strict policy silently deliver insecurely. They make the failure visible and recoverable through retry, alternate MX paths that were actually tested, and a documented repair process.

Validate before calling it complete

Run a final test matrix for every MX host:

TestExpected result
MX lookup through a validating resolverCorrect, DNSSEC-valid MX RRset
TLSA lookup through a validating resolverCorrect, DNSSEC-valid association
TLS handshake on every public addressCertificate and key match a TLSA record
SMTP without STARTTLS on a TLSA hostDANE-aware sender does not deliver in cleartext
Temporary DNS failureHost is skipped or delivery is deferred
Preferred MX unavailableA tested backup MX handles delivery
Old certificate during rolloverOld TLSA association still permits the overlap
New certificate during rolloverNew TLSA association permits the new key
Legacy senderBehavior is documented and does not depend on DANE enforcement
Queue recovery after repairDeferred mail is retried and delivered

Test a deliberately invalid TLSA digest in a non-production environment if possible. The sender should fail the DANE authentication rather than accept a different certificate. Also test a deliberately broken DNSSEC signature or expired signature in a lab, so the operations team recognizes the resulting resolver and MTA errors before production produces them.

Key takeaway

DANE for SMTP is a DNS, certificate, and mail-queue deployment. DNSSEC provides the authenticated source of the TLSA association. TLSA binds each actual SMTP service to a certificate or public key. The sending MTA decides whether the result is usable, whether TLS is mandatory for that host, and whether a failure means retrying another MX or deferring delivery.

The safest rollout is deliberately unexciting: inventory every MX and IP family, validate DNSSEC, make STARTTLS consistent, calculate TLSA data from the live key, publish complete associations, overlap records during rollover, monitor TLS-RPT and queue behavior, and never treat a DNS validation failure as proof that TLSA is absent. That process gives DANE its security benefit without turning certificate maintenance into a mail black hole.

Previous Post