Decoding FTP’s xferlog

I was messing around with FTP and needed to decode the log file. It contains lines such as

Thu Nov  9 03:11:13 2006 1 xxx.xxx.xxx.xxx 35 /ftp/pub/anonymous/incoming/test.txt b _ i a IEUser@ ftp 0 * c

but what does this mean? Time to find out… Off to the WU-FTP website

Field

Width

Meaning

Sample

current-time

23

the current local time in the form “DDD MMM dd hh:mm:ss YYYY”. Where DDD is the day of the week, MMM is the month, dd is the day of the month, hh is the hour, mm is the minutes, ss is the seconds,
and YYYY is the year.

Thu Nov  9 03:11:13 2006

transfer-time

Variable

the total time in seconds for the transfer.

1

remote-host

Variable the remote host name xxx.xxx.xxx.xxx
file-size Variable the size of the transferred file in bytes 35
file-name Variable the name of the transferred file /ftp/pub/anonymous/incoming/test.txt
transfer-type 1 a single character indicating the type of transfer. Can be one of:
 a for an ascii transfer
 b for a binary transfer
b
special-action-flag Variable one or more single character flags  indicating any special action taken..   Can be one or more of:
 C file was compressed
 U file was uncompressed
 T file was tar’ed
 _ no action was taken
_
direction 1 the direction of the transfer. Can be one of:
 o outgoing
 i incoming
i
access-mode 1

the method by which the user is logged in. Can be one of:
 a (anonymous) is for an anonymous guest user.
 g (guest) is for a passworded guest user. (see the guestgroup command in ftpaccess(5) )
 r (real) is for a local authenticated user.

a
username Variable the local username, or if guest, the ID string given IEUser@
service-name Variable (Typically 3) the name of the service being invoked, usually FTP ftp
authentication-mode 1

the method of authentication used.
 Can be one of:
 0 none
 1 RFC931 Authentication

0
authenticated-user-id Variable the user id returned by the authentication method. A * is used if an authenticated user id is not available. *
completion-status 1 a single character indicating the status of the transfer. Can be one of:
 c complete transfer
 i incomplete transfer
c

Procmail recipes

If you want to modify your procmail.rc rules file, then I suggest that you try your rule in isolation first before you break procmail and cause problems.
The first thing to do is to create the file with your test rule in it.

  • pico proctest.rc
    1. SHELL=/bin/sh
    2. TESTDIR=/home/rollingr/test
    3. MAILDIR=${TESTDIR}
    4. LOGFILE=${TESTDIR}/results.log
    5. LOG="--- Logging for ${LOGNAME}, "
    6.  
    7. #Troubleshooting:
    8. VERBOSE=yes
    9. LOGABSTRACT=all
    10.  
    11.  
    12. ##Insert your rule here instead of sample ##
    13. # Start of sample recipe to bounce blacklisted mail
    14. :0
    15. # Test whether required user is in any of these fields
    16. #Original-, Resent-, To, Cc, Bcc,.X-Envelope-To, Apparently-To or -Resent-To
    17. * ^TO_requireduser@mydomain.com
    18. # If match found then
    19. {
    20.   # Check if the sender is whitelisted
    21.   # Lock destination file for writing if match
    22.   :0:
    23.   # Extract headers From, Sender, Reply-To, Return-Path, To
    24.   # and compare to entries in white.list
    25.   # Entries should take form [email]me@my\.domain[/email]
    26.   # otherwise dot will match any character
    27.   # grep case insensitive, supress errors
    28.   # Click for a full explanation of [url=http://unixhelp.ed.ac.uk/CGI/man-cgi?egrep]egrep[/url]
    29.   * ? formail -x"From" -x"From:" -x"Sender:" \
    30.       -x"Reply-To:" -x"Return-Path:" -x"To:" \
    31.       | egrep -is -f white.lst
    32.   # If match found, deliver to /path/to/mailbox and stop
    33.   /path/to/mailbox
    34.  
    35.   # Bounce unwanted mail
    36.   # Start test - strip body from message
    37.      :0 h
    38.     # If not from mailing daemon
    39.     * ! ^FROM_DAEMON
    40.     # and not bounce of a bounce
    41.     * ! ^X-Loop: bounced@mydomain.com
    42.     # then pipe header with extra bounced header and error message in body to sendmail and stop
    43.     | ( formail -rt -A "X-Loop: bounced@mydomain.com"; \
    44.         echo "Your email address is not on our whitelist.  Please go to [url]http://www.mydomain.com/whitelist[/url] and subscribe"; \
    45.         ) | $SENDMAIL -t
    46.  
    47.   # If we got this far, then message is on blacklist and was previously bounced or is from the mailing daemon, so throw it away and stop
    48.   # Bin everything else sent to this user (such as bounces)
    49.   :0
    50.   /dev/null
    51. }
    52. # if not to required user then continue processing
    53.  
    54. ## End of sample Recipe ##
    55.  
    56.  
    57. # Catch anything that failed test Recipe here
    58. :0
    59. ${TESTDIR}failed.mail

    Now we need to create the supporting files

  • pico white.lst
    anyuser\@gmail\.com
  • Generate a valid message file which will trigger the rule
    1. From anyuser@gmail.com  Wed Nov 15 08:48:26 2006
    2. Return-Path: <anyuser@gmail.com>
    3. Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170])
    4.         by mydomain.com (8.11.6/8.11.6) with ESMTP id kAFFmPd29195
    5.         for <me@mydomain.com>; Wed, 15 Nov 2006 08:48:25 -0700
    6. Received: by 10.66.252.9 with HTTP; Wed, 15 Nov 2006 07:48:23 -0800 (PST)
    7. Message-ID: <6757a1050611150748h6b0ada1fkf4dfe9aaefe721db@mail.gmail.com>
    8. Date: Wed, 15 Nov 2006 15:48:23 +0000
    9. From: "Sender" <anyuser@gmail.com>
    10. To: "Recipient" <me@mydomain.com>
    11. Subject: Testing Mail Header
    12. MIME-Version: 1.0
    13. Content-Type: text/plain; charset=ISO-8859-1; format=flowed
    14. Content-Transfer-Encoding: 7bit
    15. Content-Disposition: inline
    16.  
    17. This is a test for procmail
    18.  
    19. --
    20. Richard

    Next create a batch file to run the rule. Point TESTDIR to your directory.

  • pico procmail.bat
    1. #!/bin/sh
    2. #The executable file named "proctest"
    3. #
    4. # You need a test directory.
    5. TESTDIR=/home/yourlogin/test
    6. if [ ! -d ${TESTDIR} ] ; then
    7.   echo "Directory ${TESTDIR} does not exist; First create it"
    8.   exit 0
    9. fi
    10. #
    11. #Feed an email message to procmail. Apply proctest.rc recipes file.
    12. #First prepare a mail.msg email file which you wish to use for the
    13. #testing.
    14. procmail ${TESTDIR}/recipe.rc < mail.msg
    15. #
    16. #Show the results.
    17. less ${TESTDIR}/results.log
    18. #
    19. #Clean up.
    20. rm -i ${TESTDIR}/results.log
    21. rm -i ${TESTDIR}/failed.mail
  • And make it an executable
    chmod u+x procmail.bat
  • Run the test
    procmail.bat

Backing up AWStats

There are two sets of data which you need to backup; the configuration data and the historic data. The configuration file awstats.domainname.conf is stored in either the same directory as awstats.pl, /etc/awstats, /usr/local/etc/awstats or /etc while the historic data is stored in a database file in DirData (Default value “/var/lib/awstats”). There is one database file (e.g. awstatsMMYYYY.domainname.txt ) for each month. The historic data is summarised by analysing the data in the file pointed to by LogFile (Default value “/var/log/httpd/access_log”).

Simply copy the configuration and database files somewhere else using “cp -p”

Updating AWStats database

To update your statistics, run the command /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=domainname [-LogFile=xxx]

NOTE: When you access AWStats from your browser, you simply point to http://www.yourdomain.com/awstats/awstats.pl. This is because /awstats/ is defined as a Script Alias in /etc/httpd/conf/httpd.conf. You cannot access this alias from the command line, hence the long path.

Also, AWStats needs to analyse the data in chronological order. Thus, if you want to update the log to include data from previous dates, you must first move the later database files out of DirData

FAQ-COM360 : HOW CAN I PROCESS SEVERAL LOG FILES IN ONE RUN ?

PROBLEM:
How can I update my statistics for several log file, in one run ?
SOLUTION:
LogFile=”/usr/local/awstats/tools/logresolvemerge.pl access_log.* |”

Logresolvemerge is a tool provided with AWStats (in the tools directory) which merges several log files on the fly. It opens a pointer for each file and sends each record for processing a line at a time, starting with the oldest. Using such a tool as a pipe source for AWStats LogFile parameter is a very good solution because, it allows you to merge log files whatever their size with no memory use, minimal hard disk use (no temporary files built), and is fast. It prevents you from getting a “bad order” error if your log files are not correctly ordered, etc… This tool can also be used to process log files from load balanced systems (see FAQ-COM400)