Run a Self-Hosted Mail Server That Lands in the Inbox

Sending email is trivial. Postfix will hand a message to Gmail in seconds. Landing in the inbox instead of the spam folder, or worse, getting silently dropped before it is ever seen, is the part that separates a mail server you own from the managed platforms people pay a premium to avoid this exact problem. The good news is that the gap is almost entirely configuration. A self-hosted Postfix and Dovecot stack, set up correctly, competes with the big providers on deliverability, and even stays out of the Gmail spam folder once the fundamentals are in place. Set up carelessly, it lands every message in spam and you blame the software.

We run mail infrastructure in production, and the pattern is consistent: the server software is the easy 20 percent, and DNS, authentication alignment, TLS, and reputation are the 80 percent that decides whether your password resets and invoices reach a human. Here is how to build the stack so a server you control earns inbox placement.

Start with the boring prerequisites, because they gate everything

Before a single config file, you need three things from your host and your DNS, and if any one is missing nothing downstream matters.

A clean static IP with port 25 open

Many cloud providers block outbound port 25 by default to fight spam, and many IP ranges carry inherited reputation damage from whoever used them before you. Pick a host that allows port 25 and gives you a static IP you can check against the major blocklists before you commit. An IP already on Spamhaus is a dead start, and no amount of perfect configuration rescues it, though if you do inherit a listing there are ways to get your IP off Spamhaus and other blocklists fast. The cleaner choice is to decide between a dedicated and shared IP before you send a single email.

A matching PTR record (reverse DNS)

This is the single most overlooked requirement and one of the most important. The IP your server sends from must resolve back, via a PTR record, to your mail hostname, and that hostname must resolve forward to the same IP. Receiving servers check this on every connection. A missing or mismatched PTR is read as "this is a random box, probably a spam bot," and your mail gets throttled or rejected before content is even evaluated. You set the PTR with your hosting provider, not in your own zone file, which is why people forget it exists.

The DNS records that authenticate you

Publish the full set, all aligned to your sending domain:

• A and MX pointing at your mail host.
• SPF, a TXT record listing the IPs allowed to send for your domain, ending in -all to hard-fail anything else.
• DKIM, a public key in DNS that pairs with the private key Postfix uses to sign outbound mail, so receivers can verify the message was not tampered with.
• DMARC, a policy that tells receivers what to do when SPF or DKIM fails, and where to send aggregate reports.

Getting these three published and passing is its own discipline, covered in detail in setting up SPF, DKIM, and DMARC so your mail stops hitting spam. The records are necessary but not sufficient. The thing that actually earns trust is alignment, and that is where most self-hosted setups quietly fail.

Alignment is the part everyone gets wrong

SPF and DKIM can both pass while DMARC still fails, because DMARC checks that the domain SPF and DKIM authenticated matches the domain in the visible From address. If your From says [email protected] but DKIM signs with a different domain, DMARC sees a mismatch and the message looks spoofed even though every individual check is green.

The fix is to make sure the DKIM signing domain (the d= tag) and the SPF-authenticated domain both line up with the From domain. Sign with your own domain's key, send from your own domain, and the chain aligns. When we run a deliverability engagement, alignment is the first thing we audit, because a server that passes SPF and DKIM in isolation but fails DMARC alignment is the most common reason "correctly configured" mail still lands in spam.

Wire up Postfix and Dovecot with DKIM signing

The division of labor is clean. Postfix is the SMTP server that accepts and relays mail. Dovecot is the IMAP and POP server that lets users read it, and it also handles authentication for sending. Sign outbound mail with an OpenDKIM or Rspamd milter hooked into Postfix.

A minimal signing hook in Postfix's main.cf looks like this:

milter_default_action = accept
smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = inet:127.0.0.1:8891

That routes every message through the milter, which adds the DKIM signature using your private key. The matching public key lives in your DNS DKIM record. Verify the signature actually applies by checking the headers of a test message: you want to see a DKIM-Signature header with your domain in the d= tag, and the receiver reporting dkim=pass.

TLS, both for privacy and for reputation

STARTTLS with a valid certificate is table stakes now. Use Let's Encrypt for the cert and configure both Postfix and Dovecot to offer TLS. Receiving servers increasingly treat unencrypted SMTP as a negative signal, and some refuse it outright.

Two records take this further and signal that you take transport security seriously:

• MTA-STS publishes a policy, served over HTTPS plus a TXT record, asserting that mail to your domain requires TLS. It stops an attacker from downgrading a connection to plaintext.
• TLS-RPT asks receivers to send you reports when a TLS connection to your server fails, so you learn about misconfigurations before they cost you delivery.

Neither is strictly required for inbox placement, but both are markers of a well-run server, and the major receivers notice.

Layer spam filtering on inbound

Deliverability is a two-way street. A server that accepts every inbound message, including the obvious spam, builds a bad reputation and can get its own IP flagged. Run Rspamd in front of your inbound mail to score and filter, and optionally ClamAV for attachments. Rspamd also handles the outbound DKIM signing in many modern setups, so it does double duty.

The goal is a server that is a good citizen in both directions: it authenticates what it sends and it rejects the junk it receives, which keeps your IP and domain reputation clean over time. That same reputation depends on what leaves the server too, so stopping bounces from wrecking your sender reputation is the outbound half of the same hygiene.

Warm up the IP, then watch the reports

A brand-new IP with zero sending history is an unknown quantity to Gmail and Outlook, and unknown quantities get throttled. Do not blast a thousand messages on day one. Ramp volume gradually over days and weeks so the receivers build a track record of your IP sending wanted mail to engaged recipients. There is a disciplined schedule for this, laid out in warming a brand new sending domain to full volume in six weeks. This is the single biggest lever on a new server, and it is pure patience.

Once you are live, the DMARC and TLS-RPT reports are your feedback loop. DMARC aggregate reports show you who is sending as your domain and whether their mail is aligning, which catches both your own misconfigurations and anyone spoofing you. Read them, the way reading your DMARC aggregate reports finds who is spoofing your domain turns raw XML into action. They turn deliverability from guesswork into a measurable signal you can act on, and once your p=none policy is clean you can move DMARC from none to reject without killing legitimate email.

Verify before you trust it

Never assume the stack works because mail sends. Test it against real receivers and real tools before you route production traffic through it:

• Check every DNS record with a tool like MXToolbox.
• Send to a Gmail, an Outlook, a Yahoo, and a ProtonMail address, and read the raw headers. You want spf=pass, dkim=pass, and dmarc=pass on all of them.
• Run a message through Mail Tester for a spam score and a checklist of what is missing.
• Confirm the PTR resolves and matches.

If all of those are green and your IP is clean, a server you own lands in the inbox as reliably as a managed platform, at a fraction of the cost and with full control over your data. If you send at volume, the same stack still has to meet Google and Yahoo's bulk sender rules, and once you mix transactional and marketing mail you will want to split the two across separate subdomains so a campaign can never take your password resets down with it. We run exactly this kind of stack for our own infrastructure and for clients who need email accounts on a domain they control rather than a third party's.

The short version

The mail software is the easy part. Inbox placement is decided by DNS, alignment, TLS, and reputation, in that order of how often they go wrong. Get a clean IP with a matching PTR, publish SPF, DKIM, and DMARC and make sure they align to your From domain, sign every outbound message, enforce TLS, filter inbound spam, warm the IP slowly, and read your DMARC reports. Do that, and the server you run competes with the platforms that charge you to avoid running one.