Searching the log Files

SurgeMail generates two log files of general interest:

• msg.log (surgemail\recYYMM\msgYYMMDD.rec) = permanent record of mail received, bounced, delivered, etc
• mail.log (surgemail\mail*.log) = temporary record of almost everything that happens inside SurgeMail . These are rotated on restart or when they get too big (typically several hours, but if server load is high this may be as small as several minutes)

Log file searching

With this form you can do some quite advanced log file searching and troubleshooting. The search page has several ways it can be used:

1. The default behaviour is that the complete last part of each log is displayed to the size of the "limit results to" field.
2. You can manually search for specific things
3. You can drill down from the msg.log entries to mail.log entries
4. You can use the sample searches as supplied at the bottom of the page
5. You can click the suggested highlighting possibilities at top of displayed log items
6. Complete custom searches using the advanced page

These are explained in more detail below for now let us look at some examples.

The msg.log entries are formatted as: day, time, [queuenumber], action, and a variety of more detailed information

eg.
19 14:26:27 [2] Received 127.0.0.1 <test1@orion> <test3@lap> 588 <1@test> "In Queue"
19 14:26:27 [2] Stored 127.0.0.1 <test1@orion> <test3@lap> 588 <1@test> "Stored locally"

The mail.log entries are formatted as: day, time, log entry type, thread number, variety of more detailed information

eg.
08 12:25:29.06:Info:2120: SurgeSMTP 1.5d, User connected (210.86.15.138) (10.0.0.5)
08 12:25:29.07:Info:2120: smtp:[210.86.15.138] In: HELO mta5-rme.xtra.co.nz

By clicking on the thread id or date fields you can drill down to get more specific information related to a particular message or thread. By clicking on the day/time fields you are zooming out to get a display of all things happening at this time.

Using the search page

Searches consist of text to search "for", particular text to be "highlighted" and text to be "excluded". These are all comma separated lists of wildcard search terms that get combined.

eg to search for anything related to "fred@mydomain.com" OR "joe@mydomain.com" but NOT"alfred@mydomain.com" with joe's stuff highlighted you could use:

  Search for: "userfred@mydomain.com,userjoe@mydomain.com"
  highlight: "userfred@*"
  exclude: "alfred@mydomain.com"

So for multiple entries using comma separated search terms::

- "Search for:A,B" means: A or B
- "Limit to:A,B" means: A and B
- "Search for:A,B" + "Limit to:C,D" means: (A or B) and (C and D)
- "Search for:A,B" + "Limit to:C,D" + "Exclude:E,F" means: (A or B) and (C and D) and NOT (E or F)

In addition the search can be limited to a certain amount of output and log text in certain files. (See right hand side of the search page) You would typically just limit to a certain amount of output from the last one log file. But it is also useful to select a small amount of output from multiple log files and to actually limit the output by searched for text.

eg: to limit search for anything related to thread with id "12345" AND "over a timerange between 12:45 and 12:49 on the 15th of the month" in any of the mail.log files the following limits could be used:

  Limit results to : 1000K of each All files
  and limit to: ":12345:,15 12:4[5-9]"

Wildcards that may be used are *, ? and [].
( eg: the following would all match "text" or "test" : "te?t" , "t*t" , "te[sx]t" , "te[a-z]t" ).

To search for wildcarded characters these need to be escaped with a forward slash.
eg: to search for "] imap" you would use "\] imap"

Also the highlight field has special formatting that allows you to specify any arbitrary valid HTML colour in curly brackets. If none is specified each highlighted term will be given a separate colour.

Further searching tips

The links at the bottom of the page are fully specified searches. Just click the link (after filling out any optional required information fields). After having the relevant log file lines you can get further information by clicking the message ID or thread ID several times. In the msg.log the first click will highlight entries, the second click will limit to these entries, the third click will attempt to display mail.log related to this message. In mail.log the first click will highlight entries related to this thread, the second click will limit to these entries and the third click will just center the search window on the item in question.

The highlight fields at the to of the the log file output are defendant on what you are currently doing and will display some information that would be sensible to highlight based upon the log file you are currently searching.

Tracking a message

To track a particular message start with the link at the bottom of the page and drill down the queried links as much as you can.

eg Delivered and redirected to two local addresses

26 12:20:35 [1] Received 203.167.148.167 <fred@xxx> <user1@yyy> 366 <102@marijn> "In Queue"
26 12:20:35 [1] Stored 203.167.148.167 <fred@xxx> <redirect2@yyy> 366 <102@marijn> "Stored locally"
26 12:20:35 [1] Stored 203.167.148.167 <fred@xxx> <redirect1@yyy> 366 <102@marijn> "Stored locally"

eg. Delivered and retrieved via pop

26 12:46:38 [2] Received 203.167.148.167 <fred@xxx> <user1@yyy> 366 <103@marijn> "In Queue"
26 12:46:38 [2] Stored 203.167.148.167 <fred@xxx> <user1@yyy> 366 <103@marijn> "Stored locally"
26 12:47:29 [0] pop 127.0.0.1 "Fred" <fred@xxx> user1 394 <103@marijn> "103.marijn"

Format of msg.log

The msg.log files can be used to determine whether a particular message has been received and if it has been received what has happened to it. A new file is created on a daily basis named "msg<year><month><day>.rec". This log is a permanent record and is not rotated.

There are the following types of entries:

• Received - Message has been received via SMTP and queued for local or remote delivery
• Stored - Message has been delivered to local account
• Sent - Message has been delivered to remote account
• Later - Message delivery failed, will try again later
• pop - Message has been retrieved using POP3
• del - Message has been deleted using POP3 or IMAP

There are also the following entries

• New - Single line logged per recipient (requires g_log_rcpt)
• Fwd - Redirection has been applied(requires g_log_fwd)
• Rejected - Message has been received via SMTP and rejected (with reason)
• Tarpitted - Message has been tarpitted and rejected
• NoSubmit - Message was never submitted (with reason)

Log entry format:

Entries for Received, Stored, Sent, Later have the same syntax:

"date & time" "queue id" Received "from ip" "from address" "dest address" "size" "id" "status"

"date & time" "queue id" tarpited "from ip" "from address" "dest address" 0 "status message"
"date & time" 0 pop "client ip" "from address" "pop login" "size" "xmailer id" "filename"
"date & time" 0 del "." "." "login" 0 "." "filename"

Format of mail.log

The mail.log - mail6.log contain information logged from within SurgeMail to record what is happening. There are three levels that are recorded error / information / debug. The information logging level can be set using the g_log_level setting.

This is the logfile NetWin support staff will probably request if you have any problems that need investigation.

 

Additional log files

• surgemail\startstop.log = Record of SurgeMail startup and shutdown
• webmail*.log (surgemail\scripts\webmail*.log) = Similar to mail*.log but for WebMail and must be explicitly enabled in wemail.ini
• surgemail\install.log = Logfile of what is done during installation
• surgemail\mon.log = Logfile of surgemail monitor (swatch.exe) activity
• surgemail\wweb.log = SurgeMail web serving activity
• surgemail\trace.log, mutex.log = Debug logs useful if SurgeMail dies
  unexpectedly 

Particular questions

What does NoSubmit mean?

The "NoSubmit" log entry means that the message was never submitted. This could be for a variety of reasons such as an network problems, broken sending mailer, or surgemail filtering options.

24 01:07:39 [132061] NoSubmit 24.61.98.153 <support@domain.com> <evan@mydomain.net> 219 "tcp_read_dot"

It means the sending system sent
mail from:<yyy>
rcpt to:<xxx>
data
(closed connection)

In this case no 'message' was sent after the data command, this can occur if you have enabled g_badfrom_* checking because some systems take 20 seconds to respond to the from check and then their own sending stage timesout. You can add known domains that have this problem to the g_badfrom_whitelist.
eg. g_badfrom_whitelist "*slow.domain,*other.domain"

24 01:07:39 [132061] NoSubmit 24.61.98.153 <support@domain.com> <evan@mydomain.net> 219 "No DATA command sent"

It means the sending system sent
mail from:<yyy>
rcpt to:<xxx>
(closed connection)

In this case no the connection was closed before the data command. This is likely to be the result of a g_badfrom check on on an outgoing email by a system running surgemail with the g_bad_from check enabled. This is not a problem (NoSubmits was just not being logged previously).