A spanner in the works.
Last night a customer of mine complained that some of his contacts couldn’t send him mail. Apparently, his contact received the following a few days after sending my customer something:
Hi. This is the qmail-send program at mail.example.com.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.
Connected to 255.12.255.14 but connection died. (#4.4.2) I'm not going to try again; this message has been in the queue too long.
Whoa. Look at that. Another qmail installation just like my customer’s! The sending MTA’s IP address were nowhere to be found in the usual smtp or rblsmtpd logs, nor in the firewall logs. The DNSBL clearing house openrbl.org had nothing on them either. In other words, the traffic from the sending MTA didn’t even reach the various daemons handling high-level communications (qmail-smtpd, rblsmtpd etc.), but were turned back at the door so to speak.
The next thing I did was setting up a tcpdump and have them send another mail to my customer’s mail address:
tcpdump -f src net 255.12.255.0/24
Sure enough, after a three-way handshake the communication from the sending MTA stopped:
10:47:24.118697 IP mail.example.com.50065 > 10.10.0.2.smtp: S 1000285782:1000285782(0) win 5840 <mss 1380,sackOK,timestamp 1764882384 0,nop,wscale 9>
10:47:24.166551 IP mail.example.com.50065 > 10.10.0.2.smtp: . ack 1709646105 win 12 <nop,nop,timestamp 1764882431 1755154237>
10:47:24.172664 IP mail.example.com.50065 > 10.10.0.2.smtp: R 0:12(12) ack 1 win 32832 <nop,nop,timestamp 1755154289 1764882431>
Now, between the FreeBSD server and the internet proper, there is a Cisco PIX firewall with IOS 7. This was the only place left to check since the customer’s mail server did receive mail from other places without any (reported) problems. The PIX have no ACLs whatsoever and simply works as a NAT with port-holes.
After we turned off “Inspect ESMTP” in the PIX, the mails from the offending (for lack of a better word) MTA were received without a hitch.
Now, I don’t really know why Cisco puts this functionality into a firewall, but it does not belong there in my (not so) humble opinion. Why would a firewall need to inspect SMTP traffic? How do we know Cisco have understood all of the relevant RFCs and implemented the checks correctly? How do we even know why the PIX decided to block the entire connection after just three packets? Is it a bug in qmail (or qmqp, as the sending MTA seems to use) or IOS? Why do I even need to poke around in the firewall after port 25 has been opened?
Oh, well. I realise, the usual allegations of qmail “not being a real mail server” (hello, Vernon) and the PIX “not being a real firewall” aside, that a PIX probably isn’t a good choice for a professional-use mail server, but that’s what they have.