Decoding FTP’s xferlog

I was messing around with FTP and needed to decode the log file. It contains lines such as

Thu Nov  9 03:11:13 2006 1 35 /ftp/pub/anonymous/incoming/test.txt b _ i a IEUser@ ftp 0 * c

but what does this mean? Time to find out… Off to the WU-FTP website







the current local time in the form “DDD MMM dd hh:mm:ss YYYY”. Where DDD is the day of the week, MMM is the month, dd is the day of the month, hh is the hour, mm is the minutes, ss is the seconds,
and YYYY is the year.

Thu Nov  9 03:11:13 2006



the total time in seconds for the transfer.



Variable the remote host name
file-size Variable the size of the transferred file in bytes 35
file-name Variable the name of the transferred file /ftp/pub/anonymous/incoming/test.txt
transfer-type 1 a single character indicating the type of transfer. Can be one of:
 a for an ascii transfer
 b for a binary transfer
special-action-flag Variable one or more single character flags  indicating any special action taken..   Can be one or more of:
 C file was compressed
 U file was uncompressed
 T file was tar’ed
 _ no action was taken
direction 1 the direction of the transfer. Can be one of:
 o outgoing
 i incoming
access-mode 1

the method by which the user is logged in. Can be one of:
 a (anonymous) is for an anonymous guest user.
 g (guest) is for a passworded guest user. (see the guestgroup command in ftpaccess(5) )
 r (real) is for a local authenticated user.

username Variable the local username, or if guest, the ID string given IEUser@
service-name Variable (Typically 3) the name of the service being invoked, usually FTP ftp
authentication-mode 1

the method of authentication used.
 Can be one of:
 0 none
 1 RFC931 Authentication

authenticated-user-id Variable the user id returned by the authentication method. A * is used if an authenticated user id is not available. *
completion-status 1 a single character indicating the status of the transfer. Can be one of:
 c complete transfer
 i incomplete transfer

Procmail recipes

If you want to modify your procmail.rc rules file, then I suggest that you try your rule in isolation first before you break procmail and cause problems.
The first thing to do is to create the file with your test rule in it.

  • pico proctest.rc
    1. SHELL=/bin/sh
    2. TESTDIR=/home/rollingr/test
    4. LOGFILE=${TESTDIR}/results.log
    5. LOG="--- Logging for ${LOGNAME}, "
    7. #Troubleshooting:
    8. VERBOSE=yes
    9. LOGABSTRACT=all
    12. ##Insert your rule here instead of sample ##
    13. # Start of sample recipe to bounce blacklisted mail
    14. :0
    15. # Test whether required user is in any of these fields
    16. #Original-, Resent-, To, Cc, Bcc,.X-Envelope-To, Apparently-To or -Resent-To
    17. * ^
    18. # If match found then
    19. {
    20.   # Check if the sender is whitelisted
    21.   # Lock destination file for writing if match
    22.   :0:
    23.   # Extract headers From, Sender, Reply-To, Return-Path, To
    24.   # and compare to entries in white.list
    25.   # Entries should take form [email]me@my\.domain[/email]
    26.   # otherwise dot will match any character
    27.   # grep case insensitive, supress errors
    28.   # Click for a full explanation of [url=]egrep[/url]
    29.   * ? formail -x"From" -x"From:" -x"Sender:" \
    30.       -x"Reply-To:" -x"Return-Path:" -x"To:" \
    31.       | egrep -is -f white.lst
    32.   # If match found, deliver to /path/to/mailbox and stop
    33.   /path/to/mailbox
    35.   # Bounce unwanted mail
    36.   # Start test - strip body from message
    37.      :0 h
    38.     # If not from mailing daemon
    39.     * ! ^FROM_DAEMON
    40.     # and not bounce of a bounce
    41.     * ! ^X-Loop:
    42.     # then pipe header with extra bounced header and error message in body to sendmail and stop
    43.     | ( formail -rt -A "X-Loop:"; \
    44.         echo "Your email address is not on our whitelist.  Please go to [url][/url] and subscribe"; \
    45.         ) | $SENDMAIL -t
    47.   # If we got this far, then message is on blacklist and was previously bounced or is from the mailing daemon, so throw it away and stop
    48.   # Bin everything else sent to this user (such as bounces)
    49.   :0
    50.   /dev/null
    51. }
    52. # if not to required user then continue processing
    54. ## End of sample Recipe ##
    57. # Catch anything that failed test Recipe here
    58. :0
    59. ${TESTDIR}failed.mail

    Now we need to create the supporting files

  • pico white.lst
  • Generate a valid message file which will trigger the rule
    1. From  Wed Nov 15 08:48:26 2006
    2. Return-Path: <>
    3. Received: from ( [])
    4.         by (8.11.6/8.11.6) with ESMTP id kAFFmPd29195
    5.         for <>; Wed, 15 Nov 2006 08:48:25 -0700
    6. Received: by with HTTP; Wed, 15 Nov 2006 07:48:23 -0800 (PST)
    7. Message-ID: <>
    8. Date: Wed, 15 Nov 2006 15:48:23 +0000
    9. From: "Sender" <>
    10. To: "Recipient" <>
    11. Subject: Testing Mail Header
    12. MIME-Version: 1.0
    13. Content-Type: text/plain; charset=ISO-8859-1; format=flowed
    14. Content-Transfer-Encoding: 7bit
    15. Content-Disposition: inline
    17. This is a test for procmail
    19. --
    20. Richard

    Next create a batch file to run the rule. Point TESTDIR to your directory.

  • pico procmail.bat
    1. #!/bin/sh
    2. #The executable file named "proctest"
    3. #
    4. # You need a test directory.
    5. TESTDIR=/home/yourlogin/test
    6. if [ ! -d ${TESTDIR} ] ; then
    7.   echo "Directory ${TESTDIR} does not exist; First create it"
    8.   exit 0
    9. fi
    10. #
    11. #Feed an email message to procmail. Apply proctest.rc recipes file.
    12. #First prepare a mail.msg email file which you wish to use for the
    13. #testing.
    14. procmail ${TESTDIR}/recipe.rc < mail.msg
    15. #
    16. #Show the results.
    17. less ${TESTDIR}/results.log
    18. #
    19. #Clean up.
    20. rm -i ${TESTDIR}/results.log
    21. rm -i ${TESTDIR}/failed.mail
  • And make it an executable
    chmod u+x procmail.bat
  • Run the test

Backing up AWStats

There are two sets of data which you need to backup; the configuration data and the historic data. The configuration file awstats.domainname.conf is stored in either the same directory as, /etc/awstats, /usr/local/etc/awstats or /etc while the historic data is stored in a database file in DirData (Default value “/var/lib/awstats”). There is one database file (e.g. awstatsMMYYYY.domainname.txt ) for each month. The historic data is summarised by analysing the data in the file pointed to by LogFile (Default value “/var/log/httpd/access_log”).

Simply copy the configuration and database files somewhere else using “cp -p”

Updating AWStats database

To update your statistics, run the command /usr/local/awstats/wwwroot/cgi-bin/ -update -config=domainname [-LogFile=xxx]

NOTE: When you access AWStats from your browser, you simply point to This is because /awstats/ is defined as a Script Alias in /etc/httpd/conf/httpd.conf. You cannot access this alias from the command line, hence the long path.

Also, AWStats needs to analyse the data in chronological order. Thus, if you want to update the log to include data from previous dates, you must first move the later database files out of DirData


How can I update my statistics for several log file, in one run ?
LogFile=”/usr/local/awstats/tools/ access_log.* |”

Logresolvemerge is a tool provided with AWStats (in the tools directory) which merges several log files on the fly. It opens a pointer for each file and sends each record for processing a line at a time, starting with the oldest. Using such a tool as a pipe source for AWStats LogFile parameter is a very good solution because, it allows you to merge log files whatever their size with no memory use, minimal hard disk use (no temporary files built), and is fast. It prevents you from getting a “bad order” error if your log files are not correctly ordered, etc… This tool can also be used to process log files from load balanced systems (see FAQ-COM400)