mailserver/docs/installation.md

Installation

From a fresh Debian 13 to a mail server that accepts and sends mail.

Example values throughout: server mail.example.com, domain example.com, address 203.0.113.10. Substitute your own everywhere.

---

1. What has to be in place first

Without these four things there is no point going on — and they have to be there before the script runs, not after.

A server running Debian 13	Freshly installed. 2 GB of RAM is the minimum, 4 GB is comfortable. Rspamd and Redis need the room.
A static IPv4 address	Dynamic addresses are useless for mail; they sit on the blocklists.
Outbound port 25 open	Many providers block it to stop spam. Have the hosting provider unblock it — depending on the provider that takes hours or days, so apply first.
A PTR record	Reverse DNS for 203.0.113.10 must point at mail.example.com. It is set at the hosting provider, not in the domain's DNS. Without it Google classes every message as spam.

Plus, in the domain's DNS, already before the installation:

mail.example.com.    A    203.0.113.10

The A record has to be there, or Let's Encrypt cannot issue a certificate. All the other records (MX, SPF, DKIM, DMARC) are produced by the installer at the end, ready to copy — it could not name DKIM any earlier anyway, since the key only comes into existence during the run.

A quick check

dig +short mail.example.com          # must show your own address
dig +short -x 203.0.113.10           # must show mail.example.com.
nc -zv gmail-smtp-in.l.google.com 25 # must say "succeeded"

Architecture

Built and run on amd64. arm64 works too, with no change to anything in this repository:

• Every package exists for arm64 in Debian 13: rspamd, dovecot-core, postfix, redis-server, unbound, nftables, golang-go (1.24 — exactly what go.mod asks for). fail2ban and certbot are Architecture: all.
• mailctl compiles for linux/arm64 unchanged. The SQLite driver (modernc.org/sqlite) is pure Go and ships code for linux_arm64, linux_arm, linux_riscv64, linux_ppc64le, linux_s390x and others; with CGO_ENABLED=0 there is nothing to cross-compile.
• Regular expression acceleration is preserved: on amd64 Rspamd uses Hyperscan, on arm64 Debian pulls in libvectorscan5, the ARM port of the same library.

Nothing in this repository is architecture-specific. The installer compiles mailctl natively on the target host, so there is nothing to configure.

32-bit ARM (armhf) is possible but not advisable. The packages exist and mailctl builds, but the Debian rspamd package for armhf depends on neither Hyperscan nor Vectorscan — every regular expression falls back to PCRE, and this setup leans on regexes for the block, allow and language filters.

What actually decides it is RAM, not the instruction set. Measured in normal operation here:

Process	Resident
Rspamd	~510 MB
unbound	~35 MB
Redis	~37 MB
Postfix + Dovecot	~10 MB

Rspamd dominates, and it does so regardless of architecture. A single-board computer with 1 GB is not enough; 2 GB is the floor and 4 GB is comfortable.

Storage matters more on small boards. Maildir means one file per message — this server holds over 20,000 of them. SD cards are a poor fit for that, both for random I/O and for write endurance, and /var/vmail plus the Redis snapshots write constantly. Put the mail on an SSD or NVMe.

---

2. Ports, when the host sits behind a firewall

The server brings its own packet filter (nftables, default policy drop) and opens only what is listed below. If there is another firewall in front of it — a perimeter firewall, a security group, a NAT router — the same ports have to be permitted there as well, or the host firewall's rules are moot.

Everything here is IPv4. This setup does not configure IPv6 (see the note at the end of this section).

Inbound: internet → server

Port	Proto	Service	Needed
25	TCP	SMTP, mail from other servers	always — without it no mail arrives at all
465	TCP	Submission, implicit TLS	for your own users to send from outside
587	TCP	Submission, STARTTLS	alternative to 465; one of the two is enough
993	TCP	IMAPS	for mail clients to read mail
143	TCP	IMAP with STARTTLS	optional, 993 is the better default
80	TCP	ACME challenge for Let's Encrypt	at issue and at every renewal, roughly every 60 days
4190	TCP	ManageSieve	only if clients edit their own rules
your SSH port	TCP	administration	yes, unless there is a console

Port 80 is the one people forget: it is not needed for mail, so it gets closed again after the installation — and about two months later the certificate quietly fails to renew. If it cannot stay open permanently, switch certbot to the DNS-01 challenge instead (certbot --preferred-challenges dns), which needs no inbound port at all.

Nothing else has to be reachable from outside. Rspamd (11332–11334), Redis (6379) and unbound (53) deliberately listen on 127.0.0.1 only.

Outbound: server → internet

This is the half that gets forgotten, and it is where a corporate firewall usually hurts.

Port	Proto	Destination	What for
25	TCP	any mail server on the internet	sending. The single most commonly blocked port.
53	UDP and TCP	arbitrary internet name servers	see the warning below
443	TCP	maps.rspamd.com, updates.rspamd.com, Let's Encrypt, deb.debian.org, proxy.golang.org	spam rule updates, certificates, packages, building mailctl
80	TCP	deb.debian.org, sa-update.surbl.org	packages, one Rspamd map served over HTTP
11335	UDP	fuzzy1.rspamd.com, fuzzy2.rspamd.com	fuzzy hashes — recognises mail already seen elsewhere
123	UDP	NTP servers	the clock. A wrong clock breaks DKIM validation and TLS.

TCP 53 matters as much as UDP 53: DNSSEC answers are large and are frequently truncated, at which point the resolver retries over TCP. A firewall that only lets UDP 53 through causes intermittent, hard-to-place resolution failures.

The trap: DNS must go out to the whole internet.

unbound is set up here as a full resolver. It asks the root servers and then the authoritative name servers of each domain directly — it does not forward to anything. That is deliberate, for two reasons:

• The DNSBL operators (Spamhaus and the rest) block queries arriving through shared resolvers. Asking through the company resolver gets you error codes instead of answers, and the blocklists silently stop working.
• DNSSEC validation is what makes DANE possible, and DANE is what secures outbound TLS.

Many corporate firewalls permit DNS only to the internal resolver. If outbound 53 to the internet genuinely cannot be opened, the setup still runs, but you have to add a forwarder to /etc/unbound/unbound.conf.d/mailserver.conf:

forward-zone:
    name: "."
    forward-addr: 10.0.0.53      # the resolver you are allowed to use

Be clear about the price: the DNSBL checks become unreliable to useless, and DANE only keeps working if that resolver validates DNSSEC and you trust the path to it. Spam detection then rests on Rspamd and Bayes alone.

Behind NAT

Two things have to line up, and they are easy to get wrong separately:

• The A record must name the public address at which the server is reachable from outside (the DNAT address).
• The PTR record must exist for the address the server sends from (the SNAT address) and resolve to the same name.

If inbound and outbound use different public addresses, receiving works while sending fails on PTR and SPF checks — and the diagnosis is miserable, because mail arrives perfectly well. Pin the outbound address to the same one, or leave NAT out of the path for this host.

mailctl cannot see any of this: it reads its own address from the route to the outside, which behind NAT is the private one. That is what /etc/mailserver/server.conf is for — enter the public address there.

Verifying it

From outside, against the server:

for p in 25 80 143 465 587 993 4190; do nc -zv -w5 mail.example.com $p; done

From the server, outwards:

nc -zv -w5 gmail-smtp-in.l.google.com 25     # sending
dig +short @198.41.0.4 . NS                  # UDP 53 straight to a root server
dig +short +tcp @198.41.0.4 . NS             # TCP 53 as well
dig +short debian.org A                      # full recursion through unbound
curl -sSI https://maps.rspamd.com/ | head -1 # 443 outbound
timedatectl show -p NTPSynchronized          # must say yes

If the DNSBL lookups are the question, this is the direct test — it must return addresses in 127.0.0.x, not an error:

dig +short 2.0.0.127.zen.spamhaus.org A

IPv6

Not configured. inet_protocols = ipv4 in Postfix, do-ip6: no in unbound. That is harmless as long as no AAAA record exists for the server. Publish an AAAA record without adapting the configuration and remote servers will try IPv6, get nowhere, and delivery will stall until they fall back. Either leave IPv6 out entirely, or do it properly: inet_protocols = all, unbound on IPv6, nftables rules for it — plus a second PTR record for the IPv6 address.

---

3. Installing

git clone https://git.fhi.mpg.de/mike/mailserver.git
cd mailserver
cp install/mailserver.conf.example install/mailserver.conf
$EDITOR install/mailserver.conf
sudo install/mailserver-install

Without install/mailserver.conf the script asks for the values in the terminal. The file is nevertheless the better choice: it documents, as a side effect, what the server was set up with.

The run takes five to fifteen minutes depending on the line, most of it apt and building mailctl.

The one value where a mistake hurts

SSH_PORT. The firewall lets exactly that port in and drops everything else. Put the wrong number there and your session is dead after stage 6 — with the server reachable only through the hosting provider's rescue console.

The script therefore checks beforehand whether an sshd really is listening on the port given, and asks if not. Even so: look at the value twice. To find it:

sudo sshd -T | grep '^port '

To be completely safe, keep a second SSH session open during the installation. As long as it exists it stays usable even after a wrong firewall rule (ct state established covers it), and the mistake can be undone at leisure.

---

4. What the script does

Ten stages. Each can be repeated on its own: sudo install/mailserver-install --only <stage>.

Stage	What happens
checks	Debian version, resolution of the server name, outbound port 25, port 80 free, memory
packages	apt install of Postfix, Dovecot, Rspamd, Redis, unbound, nftables, fail2ban, certbot, Go
users	user vmail (uid/gid 5000), group mailauth, directories
config	every file from config/ with the placeholders filled in; the database from schema.sql; Sieve scripts compiled
resolver	unbound set up, /etc/resolv.conf pointed at 127.0.0.1
firewall	nftables and fail2ban — this is where the SSH check happens
certificate	certbot fetches the certificate for MAILHOST
mailctl	sources to /usr/local/src/mailctl, compiled, installed
services	in order: Redis, Rspamd, Postfix, Dovecot; then a status check
firstdomain	mailctl domain add, mailbox postmaster@, language filter

When an existing file is replaced, the old version is put in /var/backups/mailserver-install_<timestamp>/ beforehand.

An order that is not arbitrary

• Rspamd before Postfix. Postfix has Rspamd attached as a milter and is set to milter_default_action = tempfail. If Rspamd is not running, Postfix defers every message. That is deliberate — better to delay than to wave mail through unfiltered — but during setup it looks like a fault.
• Certificate before Dovecot. Without the certificate files Dovecot does not start.
• Firewall before certbot. The ACME challenge needs port 80 from outside.

---

5. When certbot fails

The most common stumbling block, and almost always one of two reasons:

1. The A record for mail.example.com does not point here yet. Check with dig +short mail.example.com, then wait.
2. Port 80 is not reachable from outside — the provider's firewall, or a web server already occupying it.

To catch up:

sudo install/mailserver-install --only certificate

For a lab setup without public DNS there is an escape hatch:

sudo install/mailserver-install --only certificate --self-signed

The services then start, but no mail client and no foreign mail server will accept the certificate. For experiments only.

---

6. Afterwards

At the end the script prints every DNS record still missing. The same list any time:

mailctl dns example.com

Publish, wait, verify:

mailctl check example.com

Only once all six ticks are there is the server finished. Then create the actual mailboxes:

mailctl user add martin@example.com -g -q 5G

Checking from outside

Send a message from the new mailbox to check-auth@verifier.port25.com. The reply states, line by line, whether SPF, DKIM and iprev pass. Everything else is guesswork.

swaks --to check-auth@verifier.port25.com \
      --from martin@example.com \
      --server mail.example.com:587 --tls \
      --auth-user martin@example.com

Expected are SPF check: pass, DKIM check: pass and iprev check: pass. If DKIM says anything else, the TXT record is wrong or has not spread yet.

---

7. Checking later that everything still matches

install/verify-templates

Compares every file under config/ with the one actually running on the server, placeholders filled in. That is how you spot what has been changed by hand since the installation.

Differences that concern comment lines only are counted separately: they cannot change behaviour.

---

8. By hand instead of by script

If you would rather do every step yourself: install/mailserver-install is deliberately written as a readable sequence, one function per stage. Working through it from top to bottom is enough.

The placeholders in config/ are:

Placeholder	Meaning
@@MAILHOST@@	fully qualified name of the server
@@MAILDOMAIN@@	first mail domain
@@SERVER_IP@@	public IPv4 address
@@SSH_PORT@@	port of the SSH daemon

---

9. Migrating an existing server

The route to take when the point is not to build anew but to move:

1. Install the new server following this guide — but do not switch the domain over in DNS yet.
2. Fetch the backup from the old server (/var/backups/mailserver/) and restore from it:
   • mail.db to /etc/mailserver/mail.db — it holds domains, mailboxes, password hashes and aliases.
   • The DKIM keys to /var/lib/rspamd/dkim/. This is the part you do not want to regenerate: a new key means changing DNS and waiting until the old signature has expired everywhere.
3. Transfer the mail data. It is not in the backup, because it grows arbitrarily large:

   rsync -aH --numeric-ids old:/var/vmail/ /var/vmail/
   chown -R vmail:vmail /var/vmail
   

   Maildir is built for exactly this — one file per message, no locking trouble.
4. Fix the permissions and reload the services:

   chown root:mailauth /etc/mailserver/mail.db && chmod 640 /etc/mailserver/mail.db
   chown -R _rspamd:_rspamd /var/lib/rspamd/dkim
   systemctl reload postfix dovecot rspamd
   

5. mailctl check example.com, then point the A record and the PTR at the new server.
6. Leave the old server running for a few more days. Until the change has filtered through everywhere, mail keeps arriving there.

The Bayes training state cannot simply be taken along — it lives in Redis. Either fetch it from the old server with redis-cli --rdb and load it, or train anew. Training anew is the more honest option if usage has changed anyway; how to do it is in the operating manual under Targeted retraining.