Rolling Resource

Westhost have recently been “reorganising” their forums with the result that various posts have disappeared. I posted a comment on the forum encouraging them to leave posts from dissatisfied customers, but this has also been deleted. A pity, since I felt it was quite eloquent

Here is their reasoning

Originally Posted by WESTHOST – BChambers Hello Rolling,

As you are aware, we have a number of different threads that provide great information to many people who actively use our forums. Because of the large amount of content within our forums, it is easy for people to get lost in all the posts. We try and keep things organized the best we can so people can find what they are looking for more easily. Recently, we have merged a number of threads that discuss the same topic but yet are spread around in many different locations. We also try to weed out spam and other content that adds little to no value to the forums. You can still find plenty of posts currently within our forums that might feel negatively about a specific issue. It is not our intent or desire to rid our forums of those types of posts.

I agree with you in the sense that the forums provide a place for people to speak their mind. But I also need to balance that with the fact that our forums are frequently used as a reference for many clients looking for information to help them with some question. If they can find what they are looking for more quickly, then we are meeting our objectives.

Obviously they felt that my exhortations added little or no value

In order to modify Sendmail, you need to understand a little bit of m4. You can edit /etc/mail/sendmail.cf if you want to avoid the horrors of m4, but you will lose any changes if your host decides to modify one of the base files and rebuild your sendmail.cf

The components of sendmail.cf are

• /etc/mail/sendmail.mc
• /usr/share/sendmail-cf/m4/cf.m4
• /usr/share/sendmail-cf/m4/cfhead.m4
• /usr/share/sendmail-cf/sh/makeinfo.sh
• /usr/share/sendmail-cf/ostype/$1.m4
• /usr/share/sendmail-cf/mailer/$1.m4
• /usr/share/sendmail-cf/domain/$1.m4
• /usr/share/sendmail-cf/feature/$1.m4
• /usr/share/sendmail-cf/hack/$1.m4
• /usr/share/sendmail-cf/siteconfig/$1.m4
• /usr/share/sendmail-cf/m4/proto.m4

$1 will be replaced by the m4 processor with the parameter passed using the keyword. For example OSTYPE(`linux’) causes the file /usr/share/sendmail-cf/ostype/linux.m4 to be included. Any changes we make will have to be made in /etc/mail/sendmail.mc as we do not have write access to any of the files in /usr/share on the Westhost VPS platform.

Structure of sendmail.cf

It is important that you maintain the order of your configuration file. The general rules are that the order should be:

VERSIONID OSTYPE DOMAIN FEATURE local macro definitions MAILER LOCAL_RULE_* LOCAL_RULESETS

There are a few exceptions to this rule. Local macro definitions which influence a FEATURE() should be done before that feature. For example, a define(`PROCMAIL_MAILER_PATH’, …) should be done before FEATURE(`local_procmail’).

Beware: MAILER declarations should always be at the end of the configuration file, and MAILER(`smtp’) should always precede MAILER(`procmail’), and MAILER(`uucp’).

Adding Comments

You can include comments in your file, but it is imperative that you do not include keywords such as define or FEATURE in a comment as m4 will expand them. (A full list of m4 keywords can be found here). If you want the comment to appear in sendmail.cf, then start it with the # character. If you just want the comment to appear in sendmail.mc, then start it with dnl. For example # This comment will appear in sendmail.cf dnl but this one will not # The first part of this comment will dnl and the rest will not dnl # but none of this comment will appear in sendmail.cf

Diversions

m4 controls the generated output using diversions.What this means is that the order of the input files is not necessarily that of the output file. To quote from the m4 manual

When you start using m4, output is written to diversion zero, which is the standard output. When m4 reaches the end of the input, it writes out the contents of all the diversions in numerical order.

Use PUSHDIVERT ( new diversion ) and POPDIVERT to change the current diversion and restore it. For example: PUSHDIVERT(9) dnl Change logging level undefine(`confLOG_LEVEL')dnl define(`confLOG_LEVEL',`9')dnl POPDIVERT

Making your changes

As I said earlier, the preferred place for changes at Westhost is in the /etc/mail/sendmail.mc file. Lets say that we wanted to add some headers to incoming email. We would add dnl 15Nov06 Add extra mail headers for spam checking dnl The Sender Address Relative to the recipient HX-Envelope-From: $g dnl Recipient's username - needed to check bcc HX-Envelope-To: $u dnl The incoming protocol used HX-Protocol: $r A full list of the available macros ($g, $u etc) are available here or on page 42 of the Sendmail Installation and Configuration guide

Compiling your changes

Once your changes are complete, you need to convert them into something that sendmail can understand. I recommend that you redirect the output to a new file so that you can check for changes before going live.

m4 sendmail.mc >sendmail.new diff sendmail.cf sendmail.new

I left WestHost a number of months back for various reasons, but I keep checking the forums because of the excellent information, much of which comes from your posts (plus those from Shawn, Jalal, and others).

I am sending this PM because I wanted to thank you for the help. You may reply or not (I prefer email to PM’s), as you wish.

Howard Lang (now hosting with Inf-inet)

There are two sets of data which you need to backup; the configuration data and the historic data. The configuration file awstats.domainname.conf is stored in either the same directory as awstats.pl, /etc/awstats, /usr/local/etc/awstats or /etc while the historic data is stored in a database file in DirData (Default value “/var/lib/awstats”). There is one database file (e.g. awstatsMMYYYY.domainname.txt ) for each month. The historic data is summarised by analysing the data in the file pointed to by LogFile (Default value “/var/log/httpd/access_log”).

Simply copy the configuration and database files somewhere else using “cp -p”

Updating AWStats database

To update your statistics, run the command /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=domainname [-LogFile=xxx]

NOTE: When you access AWStats from your browser, you simply point to http://www.yourdomain.com/awstats/awstats.pl. This is because /awstats/ is defined as a Script Alias in /etc/httpd/conf/httpd.conf. You cannot access this alias from the command line, hence the long path.

Also, AWStats needs to analyse the data in chronological order. Thus, if you want to update the log to include data from previous dates, you must first move the later database files out of DirData

FAQ-COM360 : HOW CAN I PROCESS SEVERAL LOG FILES IN ONE RUN ?

PROBLEM: How can I update my statistics for several log file, in one run ? SOLUTION: LogFile=”/usr/local/awstats/tools/logresolvemerge.pl access_log.* |”

Logresolvemerge is a tool provided with AWStats (in the tools directory) which merges several log files on the fly. It opens a pointer for each file and sends each record for processing a line at a time, starting with the oldest. Using such a tool as a pipe source for AWStats LogFile parameter is a very good solution because, it allows you to merge log files whatever their size with no memory use, minimal hard disk use (no temporary files built), and is fast. It prevents you from getting a “bad order” error if your log files are not correctly ordered, etc… This tool can also be used to process log files from load balanced systems (see FAQ-COM400)

These are the commands you will have to issue from the command prompt on your server in order to download the necessary libraries and applications needed to build Python. I have piped all output from each stage into log.xxxx and err.xxxx files. Make sure that you read the err.xxxx files if they are not zero length and action any errors before proceeding to the next step. The information in log.xxxx will help you troubleshoot any errors.

As I do not have root access on Westhost, I have decided to create my own tree under /usr/mylocal for libraries and executables.

Download gdbm

• ftp ftp.gnupg.org
• login: anonymous password:anon@
• cd gnu/gdbm
• passive
• dir gdbm-*
• get gdbm-1.8.3.tar.gz
• quit
• gunzip -c gdbm-1.8.3.tar.gz |tar x
• cd gdbm-1.8.3
• ./configure --prefix=/usr/mylocal >log.config 2>err.config
• make >log.make 2>err.make
• make install >log.install 2>err.install
• ls -l err.*
• edit /etc/ld.so.conf to include /usr/mylocal/lib as first line
• ldconfig
• cd ..

Download ncurses

• ftp ftp.gnupg.org
• login: anonymous
• cd /pub/gnu/ncurses
• passive
• dir ncurses*
• get ncurses-5.5.tar.gz
• get ncurses-5.5.tar.gz.sig
• quit
• gunzip -c ncurses-5.5.tar.gz |tar x
• cd ncurses-5.5
• ./configure --prefix=/usr/mylocal >log.config 2>err.config
• make >log.make 2>err.make
• make install >log.install 2>err.install
• ls -l err.*
• ldconfig
• Run test/newdemo to check your installation. Press Q to stop.
• cd ..

I was unable to link to ncurses/xxxxx.h files when building Python. As a workaround, you need to define these three symbolic links

• ln -s /usr/mylocal/include/ncurses/ncurses.h /usr/mylocal/include/ncurses.h
• ln -s /usr/mylocal/include/ncurses/curses.h /usr/mylocal/include/curses.h
• ln -s /usr/mylocal/include/ncurses/panel.h /usr/mylocal/include/panel.h

Download Python

• wget http://www.python.org/ftp/python/2.5/Python-2.5.tar.bz2
• wget http://www.python.org/download/releases/2.5/Python-2.5.tar.bz2.asc
• gpg --verify Python-2.5.tar.bz2.asc
• bunzip2 -c Python-2.5.tar.bz2 |tar x
• cd Python-2.5
• ./configure --prefix=/usr/mylocal 1>log.config 2>err.config
• make >log.make 2>err.make
• make install >log.install 2>err.install
• ls -l err.*

If /usr/mylocal/bin is not in your path, then you will need to create a symbolic link for it

• ln -s /user/mylocal/bin/python /user/local/bin/python