Exim Troubleshooting - errorCodex

Miraenda · #1 21 November 2009, 18:55

Since I am not sure where else to start on this forum, I am going to start with some basic tips for exim troubleshooting. I am relatively familiar with exim, and I actually happen to enjoy using and fixing it.

Exim Troubleshooting

Logs

Main logging - /var/log/exim_mainlog or /var/log/exim/mainlog
Main panic log - /var/log/exim_paniclog or /var/log/exim/paniclog
Main reject log - /var/log/exim_rejectlog or /var/log/exim/rejectlog

Also, logging for services like IMAP, POP3 and Spam Assassin writes to /var/log/maillog (not really exim here, but I might as well mention this log file)
___________________________________

Port Checking

Exim usually runs on port 25 and possibly 26 or 587 for an alternate port. To check if it is working on a server, telnet would be a good tool to use:

Code:

telnet servername 25

or

Code:

telnet IP# 25

This will return a transcript similar to the following:

Code:

Trying IP#...
Connected to domain.com.
Escape character is '^]'.
220-server.domain.com ESMTP Exim 4.69 #1 Sat, 21 Nov 2009 17:18:29 -0800 
220-We do not authorize the use of this system to transport unsolicited, 
220 and/or bulk e-mail.

(If the server returns no response on port 25, then something is wrong with the server for exim and email services probably are not functioning properly on that server.)

Notably, the response in the 220-server.domain.com ESMTP area indicates the server's port 25 response. The PTR or rDNS (reverse DNS) record should be the same in the DNS zone as this response, so basically server.domain.com. If the rDNS does not match the server's response, many email servers may reject or spam flag the email sent by the server.

To find out the PTR record for a domain's IP, use the following command:

Code:

dig -x IP# +short

The return will be something like the following:

Code:

server.domain.com

The reason why most receiving servers would spam flag emails where the PTR does not match the port 25 response occurs due to spammers either using a phony IP address, using dynamically bound IPs, or spoofing an email address to send emails. Since these are common abusive techniques used, most email servers use a technique called sender verify. Sender verify will send a verification check back to the sending server on port 25 to check the sending server's response. That response will be the server's hostname (found in /etc/hosts and /etc/sysconfig/network files typically). If the PTR record does not match the sending server's port 25 response, then the receiving server will typically either reject the email or spam flag it.

To see the hostname for a server on the server itself, issue the following command:

Code:

hostname

This will return the current hostname for the machine. If the hostname needs to be changed, please review the instructions at this location.

The hostname must be a fully qualified domain name (FQDN) with a DNS zone A record on the server. A subdomain for a domain on the server works best, since many servers will have a conflict in functionality if a domain name with a site on the server is used for the server's hostname. As such, avoid using mydomain.com and pick something like server.mydomain.com for the hostname instead.
___________________________________

Checking exim's mainlog

To check how an email sent or received on the server routed, or for error messages received for that email address, exigrep command should be issued:

Code:

exigrep admin@mydomain.com /var/log/exim_mainlog

Here admin@mydomain.com would be the email address being checked, while /var/log/exim_mainlog is the path to the exim mainlog on the server. Replace the exim mainlog path with the correct path for your server.

The output may look similar to the following:

Code:

2009-11-21 04:02:34 1NBofi-0003mK-Sq <= root@server.domain.com U=root P=local 
S=53427
2009-11-21 04:02:38 1NBofi-0003mK-Sq => 
admin@mydomain.com R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [IP#]
2009-11-21 04:02:38 1NBofi-0003mK-Sq Completed

Since many different extended logging options exist along with possible error messages and routing possibilities, the log output will depend greatly on those mechanisms.

I did want to mention why exigrep is being used rather than a simple egrep or grep for the log. I can illustrate this with running an egrep on the log and then comparing it to the exigrep above. First, let's say we egrep for the email account admin@mydomain.com:

Code:

egrep admin@mydomain.com /var/log/exim_mainlog

The output would be this in my example:

Code:

2009-11-21 04:02:38 1NBofi-0003mK-Sq => admin@mydomain.com 
R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [IP#]

Compare that to the prior exigrep command output for the same email address:

Code:

2009-11-21 04:02:34 1NBofi-0003mK-Sq <= root@server.domain.com U=root P=local 
S=53427
2009-11-21 04:02:38 1NBofi-0003mK-Sq => 
admin@mydomain.com R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [IP#]
2009-11-21 04:02:38 1NBofi-0003mK-Sq Completed

Basically, egrep and grep will only show the part of the route for the email address or domain being checked, while exigrep will output the entire route path for that email address message.
___________________________________

Checking MX record entries

To see the MX record entries for a domain, use the following command:

Code:

dig mx domain.com +short

This will output something similar to the following:

Code:

0 domain.com.

If the MX records have multiple entries, the return instead might look like the following:

Code:

5 alt1.server.domain.com.
5 alt2.server.domain.com.
10 alt3.server.domain.com.
0 server.domain.com.

For the full output for the dig, remove the +short portion.

Of note, the MX records received from running dig are dependent on the server's resolver. This means that one machine might return different results than another machine if their resolvers have different records returning. Sometimes, MX records are changed and a resolver caches the old records for a period of time, so always check a couple of different sources for the MX record information.
___________________________________

Resolver routing records

To see how a specific server will route a given email address, check the routing for that email using the following command:

Code:

exim -bt admin@mydomain.com

Replace admin@mydomain.com with the email address being checked. A return such as the following might occur:

Code:

admin@mydomain.com
  router = lookuphost, transport = remote_smtp
  host mydomain.com [IP#] MX=0

If the domain has multiple MX records, the return might instead be something like the following:

Code:

admin2@mydomain.com
  router = lookuphost, transport = remote_smtp
  host alt1.server.domain.com [IP1] MX=5
  host alt2.server.domain.com [IP2] MX=10
  host alt3.server.domain.com [IP3] MX=30

If the IP listed does not match the MX record expected for routing, the server in question might have cached records for the routing process, or the server might be using some alternate smarthost or relay to route from another server to that server. In that instance, access would be needed to that smarthost or relaying server to see why messages aren't routing properly. Without access to check that relaying server, there would be no method to troubleshoot why the secondary routing server did not route the message from one server to the other (since that relaying server is acting as an intermediary for the email routing process).
___________________________________

Sending Test Email from Command Line

To send a test email from one server to another, the following command would be used:

Code:

exim -odf -v admin@mydomain.com
Example message.
.

This command tells exim to start an email session from the server to the email address admin@mydomain.com and the message body will be Example message. for that message. The session must end with a final period . as indicated. Without the terminating period, the exim session does not know to end the message and send it.

An example output that might occur for such a session would be the following:

Code:

LOG: MAIN
  <= root@server.domain.com U=root P=local S=336
delivering 1NC1Y4-0007T7-VS
Connecting to server.mydomain.com [receiving-IP#]:25 ... connected
  SMTP<< 220-server.mydomain.com ESMTP Exim 4.69 #1 
  Sat, 21 Nov 2009 19:47:27 -0600
         220-We do not authorize the use of this system to transport
  unsolicited,
         220 and/or bulk e-mail.
  SMTP>> EHLO server.domain.com
  SMTP<< 250-server.mydomain.com Hello domain.com [sending-IP#]
         250-SIZE 52428800
         250-PIPELINING
         250-AUTH PLAIN LOGIN
         250-STARTTLS
         250 HELP
  SMTP>> STARTTLS
  SMTP<< 220 TLS go ahead
  SMTP>> EHLO server.domain.com
  SMTP<< 250-server.mydomain.com Hello domain.com [sending-IP#]
         250-SIZE 52428800
         250-PIPELINING
         250-AUTH PLAIN LOGIN
         250 HELP
  SMTP>> MAIL FROM:<root@server.mydomain.com> SIZE=1368
  SMTP>> RCPT TO:<admin@mydomain.com>
  SMTP>> DATA
  SMTP<< 250 OK
  SMTP<< 250 Accepted
  SMTP<< 354 Enter message, ending with "." on a line by itself
  SMTP>> writing message and terminating "."
  SMTP<< 250 OK id=1NC1Y1-00052F-8p
  SMTP>> QUIT
LOG: MAIN
  => admin@mydomain.com R=lookuphost 
  T=remote_smtp H=server.domain.com [receiving-IP#] 
  X=TLSv1:AES256-SHA:256
LOG: MAIN
  Completed

If the message failed at some point in the route, it would show in the log. This actually sends a real message via the server to the other server, so it is a powerful tool for troubleshooting email issues.

To increase the logging debug output for the session, replace the -v with a -d instead, resulting in this command:

Code:

exim -odf -d admin@mydomain.com

These are the main tips I have for this lesson. For any questions related to this guide, please contact me via the forum.

litemotiv · #2 20 April 2010, 04:18

Hi Miraenda, thanks for your tutorial

I have a small issue with Exim that i hope you might be able to help me with.

Here is the scenario, i have installed and configured everything and can receive mail for admin@mydomain.com (it gets stored in the local Maildir). Then i set up a forward address for admin:

Code:

admin: myadress@gmail.com

Then i send a local test mail to admin

Code:

exim -odf -v admin@mydomain.com
Example message.
.

And the mail is delivered to myadress@gmail.com, so far everything works.

Then i send the same email to admin@mydomain from an external address (regular email). I then check the mainlog on the server with exigrep:

Code:

2010-04-20 04:24:56 1O48lM-00059d-46 SA: Debug: SAEximRunCond expand returned: '1'
2010-04-20 04:24:56 1O48lM-00059d-46 SA: Debug: check succeeded, running spamc
2010-04-20 04:24:58 1O48lM-00059d-46 SA: Action: flagged as Spam but accepted: score=5.4 required=5.0 (scanned in 2/2 secs | Message-Id: l2j36ba9b061004200124u95ab8ae1mbc12490e83da2d14@mail.gmail.com). From <myotheraddress@gmail.com> (host=mail-bw0-f219.google.com [209.85.218.219]) for admin@mydomain.com
2010-04-20 04:24:58 1O48lM-00059d-46 <= myotheraddress@gmail.com H=mail-bw0-f219.google.com [209.85.218.219] P=esmtp S=2480 id=l2j36ba9b061004200124u95ab8ae1mbc12490e83da2d14@mail.gmail.com
2010-04-20 04:25:03 1O48lM-00059d-46 => myaddress@gmail.com <admin@mydomain.com> R=dnslookup T=remote_smtp H=cluster.onlinemailscanner.com [81.4.88.154]
2010-04-20 04:25:03 1O48lM-00059d-46 Completed

As you can see the log says completed, no errors, but for some reason the email never arrives. I've tried sending from different addresses and checked the spamfolder in gmail but it's not there either, it seems to vanish into thin air..

Do you have any idea what could be causing this?

Thanks in advance for any insight.

-lite

Miraenda · #3 20 April 2010, 06:42

Hi and welcome lite

For this routing:

Code:

2010-04-20 04:24:56 1O48lM-00059d-46 SA: Debug: SAEximRunCond expand returned: '1'
2010-04-20 04:24:56 1O48lM-00059d-46 SA: Debug: check succeeded, running spamc
2010-04-20 04:24:58 1O48lM-00059d-46 SA: Action: flagged as Spam but accepted: score=5.4 required=5.0 (scanned in 2/2 secs | Message-Id: l2j36ba9b061004200124u95ab8ae1mbc124...mail.gmail.com). From <myotheraddress@gmail.com> (host=mail-bw0-f219.google.com [209.85.218.219]) for admin@mydomain.com
2010-04-20 04:24:58 1O48lM-00059d-46 <= myotheraddress@gmail.com H=mail-bw0-f219.google.com [209.85.218.219] P=esmtp S=2480 id=l2j36ba9b061004200124u95ab8ae1mbc124...mail.gmail.com
2010-04-20 04:25:03 1O48lM-00059d-46 => myaddress@gmail.com <admin@mydomain.com> R=dnslookup T=remote_smtp H=cluster.onlinemailscanner.com [81.4.88.154]
2010-04-20 04:25:03 1O48lM-00059d-46 Completed

It is showing that the email was flagged as spam "SA: Action: flagged as Spam but accepted" so is the email you've sent even arriving to admin@mydomain.com box itself?

Also, is the sender address to admin@mydomain.com actually myotheraddress@gmail.com as it's hard to believe that a gmail address would get flagged as spam by your machine. If it is, you might need to update the SpamAssassin rulesets to ensure you have the latest ones:

sa-update -D

Or, on cPanel systems:

/scripts/update_sa_rules

Thanks.

litemotiv · #4 20 April 2010, 07:13

Thanks for your reply Miraenda

I did another test to avoid that spam score:

Code:

2010-04-20 08:58:55 1O4D2V-0005MU-7M SA: Debug: SAEximRunCond expand returned: '1'
2010-04-20 08:58:55 1O4D2V-0005MU-7M SA: Debug: check succeeded, running spamc
2010-04-20 08:58:56 1O4D2V-0005MU-7M SA: Action: scanned but message isn't spam: score=2.7 required=5.0 (scanned in 1/1 secs | Message-Id: r2u36ba9b061004200558pb18b963ajdfefce56520cff34@mail.gmail.com). From <*****.*****@gmail.com> (host=mail-bw0-f219.google.com [209.85.218.219]) for admin@mydomain.com
2010-04-20 08:58:56 1O4D2V-0005MU-7M <= *****.*****@gmail.com H=mail-bw0-f219.google.com [209.85.218.219] P=esmtp S=2512 id=r2u36ba9b061004200558pb18b963ajdfefce56520cff34@mail.gmail.com
2010-04-20 08:59:02 1O4D2V-0005MU-7M => *****.*****@gmail.com <admin@mydomain.com> R=dnslookup T=remote_smtp H=gmail-smtp-in.l.google.com [74.125.67.27]
2010-04-20 08:59:02 1O4D2V-0005MU-7M Complete

Sent from gmail to mydomain, which would then forward it back to gmail. Unfortunately again the mail never arrives..

Here too when i remove the forward the mail gets delivered to the local Maildir without problems, and sending it from the server itself (with exim -odf) does get it delivered to gmail almost instantly. Strange huh?

Miraenda · #5 20 April 2010, 07:27

If the message were actually stuck, you could view the headers:

exim -Mvh <message-id>

So, like the following:

exim -Mvh 104D2V-0005MU-7M

But it doesn't really seem the message is stuck. Do you have any other filters on that account? It should show the filter processing it, though, and I only see the spam filtering. If you do have any other filters, I would take them offline. If you have anything other than spam assassin handling scanning main also, I would take it offline.

Again, if you send the email from outside to the machine to that admin@mydomain.com address, is it reaching admin@mydomain.com itself even? I realize it isn't going to gmail, but is it at least getting into the admin@mydomain.com box?

Thanks.

litemotiv · #6 20 April 2010, 08:34

You mean stuck as in the queue? The queue shows empty with exim -bp | exiqsumm.

Quote:

Again, if you send the email from outside to the machine to that admin@mydomain.com address, is it reaching admin@mydomain.com itself even? I realize it isn't going to gmail, but is it at least getting into the admin@mydomain.com box?

Yes, when i remove the forward for admin@mydomain the mail gets delivered to /home/admin/Maildir/new/ without problems (i already mentioned that in my last post

). If i keep the forward and send a mail locally to admin, it gets delivered to remote (gmail) fine. So the only failing scenario is mail coming from outside, going outside...

I will check for other filters running. For reference, this was the how-to i followed to install everything: http://koivi.com/exim4-config/

Thanks again

Miraenda · #7 20 April 2010, 11:15

I wasn't actually wondering about when the forward was removed for it as I did see you state that. I wanted to know with the forward existing if you get the message in your admin@mydomain.com account as, if the box exists, you should both get a copy in that box and then it should also forward. So, I wanted to know if you got the email in that admin@mydomain.com box *with* the forwarder existing.

If the message is there in the box, it would be helpful to see the header (for the message that did arrive at admin@mydomain.com but did not forward to gmail).

It's a very strange scenario that I've never seen happening when a filter wasn't involved somehow impacting routing.

litemotiv · #8 20 April 2010, 13:21

Ah right i understand

The message does get saved, here it is integrally: http://pastebin.org/162678

All the filters/routers are 'stock' Debian afaik, i haven't made any changes unless they were specifically mentioned in the how-to.

Miraenda · #9 20 April 2010, 14:12

You aren't by chance trying to email from your gmail account to that admin@mydomain.com account and then have it re-forward to your same gmail account?

I tested that myself by sending from my gmail account to an email account on my machine (miraenda@lunaradmin.com), which I put a forward on back to my same gmail account. Gmail wouldn't route it even though the email shows the same as your example above on it completing:

[root@synthesis ~]# exigrep miraenda@lunaradmin.com /var/log/exim_mainlog
2010-04-20 13:09:20 1O4Jl2-0005RE-J0 <= me@gmail.com H=mail-vw0-f48.google.com [209.85.212.48] P=esmtp S=1857 id=k2r4f1bee21004201309v5b678b1aw60e9af7040b194c3@ mail.gmail.com
2010-04-20 13:09:23 1O4Jl2-0005RE-J0 => me@gmail.com <miraenda@lunaradmin.com> R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [209.85.222.25]
2010-04-20 13:09:23 1O4Jl2-0005RE-J0 Completed

Google is probably seeing it as a loop, so it's not going to deliver it. You cannot send an email from a gmail account to forward back to that same gmail account by transmitting it through your server. If you are using a different gmail account than is being forwarded, I'm not sure why it isn't working. If you are using the same account for gmail to re-forward it back to, I'd test with a new gmail account that's different because I think that's going to work. It's just when using the same gmail account that is being forwarded that you won't get the email (and this is why the local deliveries worked as they aren't coming from the exact same gmail account).

Thanks.

litemotiv · #10 21 April 2010, 02:15

Uh oh, i believe you are right Miraenda... I was using different email addresses to test, but i was sending them through gmail. Apparently it doesn't matter whether it's a gmail address, as long as you send through gmail it will be seen as a loop...

I just sent a few messages through php and they all are delivered.

Wow, silly, thanks a lot for that suggestion though!
-lite

Powered by vBulletin® Version 3.8.1 Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.