mod-gzip – How To Download pages faster and Save Server Bandwidth

I recently ran these pages through a Web Optimiser and the speeds were not that impressive, so I have decided to implement mod_gzip. This will compress the textual portion of these pages so that they download quicker.

The procedure will NOT follow the familiar procedure for those who follow this site, so read on…

  • Download the source from Sourceforge.net using the command wget, for example wget http://easynews.dl.sourceforge.net/sourceforge/mod-gzip/mod_gzip-1.3.26.1a.tgz. Choose the mirror nearest to your server. If you are running Apache 2.0, then you will need to donload the module from here instead.
  • Unzip this file and extract the contents so that we can get on with the build gunzip -c mod_gzip-1.3.26.1a.tgz | tar x
  • Change into the source directory cd mod_gzip-1.3.26.1a.
  • The directions for building mod_gzip are included at the end of the ChangeLog file.

    There are two ways to build mod_gzip:

    1. statically compiled into Apache and
    2. a DSO-File for mod_so.

    The DSO-Version is much easier to build. Just type

    make APXS=/path/to/apxs
    make install APXS=/path/to/apxs
    /path/to/apachectl restart

    The apxs script is normaly located inside the bin directory of Apache.

    The path to APXS is /usr/local/apache/bin/apxs or /usr/sbin/apxs, so we will use the command
    make APXS=/usr/sbin/apxs >log.make 2>err.make
    NOTE:You will get an error complaining about ok_to_send being unused. This Warning can be safely ignored.
    mod_gzip.c: In function `mod_gzip_sendfile2':
    mod_gzip.c:5262: warning: unused variable `ok_to_send'

  • The final step of the build process is to install the module. This will copy the library to the relevant place and edit your httpd.conf file.
    make install APXS=/usr/sbin/apxs >log.install 2>err.install
    The installer inserted two lines into the httpd.conf file, which are commented out. However, it did not quite get it right on my system, so it is just as well that they are commented out! The lines read

    <IfDefine HAVE_FRONTPAGE_SPHERA>
    LoadModule frontpage_module modules/mod_frontpage_sphera.so
    #LoadModule gzip_module /usr/lib/apache/mod_gzip.so
    </IfDefine>

    and should have read


    <IfDefine HAVE_FRONTPAGE_SPHERA>
    LoadModule frontpage_module modules/mod_frontpage_sphera.so
    </IfDefine>
    <IfDefine HAVE_GZIP>
    LoadModule gzip_module /usr/lib/apache/mod_gzip.so
    </IfDefine>

    The variable HAVE_GZIP is automatically created by the start up process. The installer also made the same mistake with the AddModule directives which read
    <IfDefine HAVE_FRONTPAGE_SPHERA>
    AddModule mod_frontpage_sphera.c
    #AddModule mod_gzip.c
    </IfDefine>

    instead of

    <IfDefine HAVE_FRONTPAGE_SPHERA>
    AddModule mod_frontpage_sphera.c
    </IfDefine>
    <IfDefine HAVE_GZIP>
    AddModule mod_gzip.c
    </IfDefine>

  • There are still a couple of changes to make to the httpd.conf file according to Apache’s own website. The following needs to be added to the configuration file; I added it just after the AddModule (within the define block)
    mod_gzip_on Yes
    mod_gzip_can_negotiate Yes
    mod_gzip_dechunk Yes
    mod_gzip_minimum_file_size 600
    mod_gzip_maximum_file_size 0
    mod_gzip_maximum_inmem_size 100000
    mod_gzip_keep_workfiles No
    mod_gzip_temp_dir /usr/local/apache/gzip
    mod_gzip_item_include file \.html$
    mod_gzip_item_include file \.txt$
    mod_gzip_item_include file \.jsp$
    mod_gzip_item_include file \.php$
    mod_gzip_item_include file \.pl$
    mod_gzip_item_include mime ^text/.*
    mod_gzip_item_include mime ^application/x-httpd-php
    mod_gzip_item_include mime ^httpd/unix-directory$
    mod_gzip_item_include handler ^perl-script$
    mod_gzip_item_include handler ^server-status$
    mod_gzip_item_include handler ^server-info$
    mod_gzip_item_exclude file \.css$
    mod_gzip_item_exclude file \.js$
    mod_gzip_item_exclude mime ^image/.*

    Additional filetypes can be defined. For example, there is no need to compress PDF files as they are already compressed, so we could also add
    mod_gzip_item_exclude mime ^application/pdf$
  • You will also need to create the mod_gzip_temp_dir specified in httpd.conf. mkdir /usr/local/apache/gzip
  • Next restart Apache and check that everything is still working. If you get the error
    Syntax error on line xxx of /etc/httpd/conf/httpd.conf:
    Cannot add module via name 'mod_gzip.c': not in list of loaded modules
    then you forgot top remove the # from the LoadModule command or mod_glib.so is not in the correct directory.

WH3 Beta testing

Unfortunately I was unable to do much testing during the actual Beta period, but here are my results / comments

The speed is MUCH better than WH2.0, but still frustrating when you have to repeat operations unneccessarily.

Site Manager Performance (50.6kbps dial up)

Time to Display		WH2.0		WH3.0		Improvement
Login			12 sec		 7 sec		5 sec
Home Page		64 sec		20 sec		44 sec
Install Applications	108 sec		58 sec		50 sec
Email Users		31 sec		30 sec		1 sec
EMail Aliases		36 sec		22 sec		14 sec
Restart Account		34+23+33 sec	8+10 sec 	72 sec
Domain Management	16 sec		13 sec		3 sec
Web Aliases		23 sec		18 sec		5 sec
  • DNS Records – unable to add TXT record. unable to add domain specific record (example domain1.com IN NS ns1.twisted4life.com) as opposed to VPS generic (IN NS ns1.twisted4life.com.)
  • Processor loading was very low and so we were unlikely to run into typical issues:
    cat /proc/loadavg
    0.10 0.06 0.01 1/562 3102
    Compared to WH2.0 server:
    4.41 7.17 9.75 1/1299 21365
  • No Feedback SSH Frreze Issue – I managed to increase the loading on the server by running a
    stress test. I was able to get the server to freeze quite easily. To summarize, the performance is much better than WH2.0 as it takes a higher load to cause problems, but is obviously not fully resolved. I am concerned that I was able to consume so much resource as a single user. Was this because I was running on a test server, or is it an inherent flaw in the system? Surely there is some way to limit a single users CPU time slice? I am not willing to run this stress test on a live server to check! I managed to get the availability down to 40% (60% downtime). One point to note is that I was able to disupt the whole server as the number of active processes dropped to 718 at one stage (from ~9300 before punishing the server)
    cat /proc/loadavg
    21.02 29.27 26.22 21/791 718
  • Possible Future Feature User Manager – Remove principal domain name from banner
  • Probable Future Feature Site Manager – The site manager ‘home page’ indicates that I have a dedicated IP
    despiite the fact that the beta accounts have dynamically assigned IP addresses.
  • Probable Future Feature Site Manager – Logging in to both Site Manager and Users Manager is not possible.
    Results in redirection to https://wsl06001.west-datacenter.net/php/login/login_error_screen.php and told to sign on again. No link is provided to the login page -
    this should be possible from the cookie.
  • Probable Future Feature Domain Management – Tried adding a domain. Error message “Operation failed on server due to error: The domain newdomain.com is occupied.” not very clear. Should read something like “This domain is already hosted elsewhere on Westhost. Please delete that definition first”
  • Probable Future Feature Domain Management – Does not create custom directory if it does not already exist.
  • Probable Future Feature Sub-Domain Management – Does not create custom directory if it does not already exist.
  • Accepted File Manager – Returned multiple (incorrect) results when searching for file httpd.conf
  • Accepted Autoresponder – The autoresponder and Auto forwarder did not appear to work as neither service was enabled
    • Created a new email user with the default settings
    • Enabled email, ftp and autoresponder Quota-0 MB
    • Signed in to user manager at http://www.wh3test091.whsites.net/users
    • Selected Autoresponder and Auto Forwarder. Neither service is enabled.
    • Oops! They were working. It was just a very large, conspicuous red X and a very small, grey edit button. Suggested that button was enlarged and that text is changed to “Change settings”

  • No Feedback Application Installation – When you install applications such as PHPMyAdmin from the Site Manager, there are usually links on the final page for Administration. However, these windows are a fixed size and not alterable. When I click on the links in Firefox, the application opens in a new tab of the same window, which makes them illegible. Also, since the address bar is hidden, most users will be unable to find out what URL they should use. This problem occurs on all pop ups from package installation. Suggest URL is shown beside hyperlink
  • REJECTED Application Installation – Some installations request an email address. If this email address is pertinent to the domain and does not exist, then the user should be offerred the chance to create a new user or alias.
  • Custom 404 Returns correct status code of 404 rather than 200

Configuring a Secondary DNS

I have recently experienced some DNS problems and so decided to look into configuring an alternative or Secondary DNS for my domain. A secondary DNS server will copy the DNS settings from your primary DNS server on a regular basis, so there is no need to maintain two sets of data. If you use separate records, then browsers will have problems deciding which is up to date. There are several free services out there including Twisted for Life and Zone Edit. Other fee paying services you might consider are SecondaryDNS or EasyDNS.

Once you have signed up with your secondary DNS provider, you will need to submit a support ticket to your hosting provider (unless you have true root access) in order to modify your current DNS server’s configuration. Let's assume that your new nameserver is called ns1.alternativeDNS.com. On Westhost you will need to ask them to edit your BIND file (/var/named/db.yourdomain.com) and add a line similar to yourdomain.com. IN NS ns1.alternativeDNS.com. The trailing dot is important!. Your hosting provider may also have to add ns1.alternativeDNS.com to their nameserver to permit AXFR transfers of the information . If this does not work, try editiing /etc/named.conf.

Once this has been done, your secondary DNS provider will be able to mirror the details from your current provider. You can check that your secondary DNS provider has the correct information by retrieving the information from their nameserver. You need to type nslookup www.yourdomain.com ns1.alternativeDNS.com in Windows, or dig @ns1.alternativeDNS.com www.yourdomain.com in Linux.

The final step is to modify the nameserver entries with your registrar. Log in to your account with them and add this new nameserver to your existing list. This will take a while to propogate around the internet, so come back later and check that all your nameservers are listed when you use the command nslookup -type=NS yourdomain.com in Windows or dig -t NS yourdomain.com in Linux.

Check that everything is correct by going to DNSReport.com

Sendmail upgrade

I am glad to say that Westhost recently upgraded sendmail from 8.11.6 to 8.12.11. Thank you. However I have two questions:

1) Why did they not upgrade to Sendmail 8.13.x? Plenty of milters require 8.13 ahich was released on 06/20/2004. The latest stable version (8.13.6) was released on 03/22/2006.

2) While they included SASL support, they failed to include STARTTLS. Hopefully they will rebuild sendmail with STARTTLS as well. That way we can encrypt our login and passwords as well

EMail – Setting up Multiple Domains

or How to setup recipient@domain1.com and recipient@domain2.com to be delivered to different mailboxes

Apache provides the ability to host multiple domains on a single VPS account using Virtual Hosts.
However, we run into a problem if we want email to be treated uniquely for each domain.

There are several programs involved in processing email on the server, but we won't have to change most of them. These include

Program Description
* If installed
sendmail used to send and receive email by your server
procmail used to process email after it is received by sendmail
spamassassin* called from procmail to filter out spam messages
clamav* called from procmail to filter out viruses
qpopper used to deliver email to your computer using the pop3 protocol
uw-imap* used to view email on your computer using the imap protocol

Sendmail uses a lookup table (virtusertable) to convert an email address into a Unix user account.
Full details can be found at Sendmail's website

If you have a hosting plan which allows for multiple domains, then Westhost configures virtusertable so that email to
recipient@anydomain.com is delivered to the Unix mailbox for recipient's account.
We need to modify this so that the email for recipient@domain1.com is delivered to a separate mailbox from recipient@domain2.com

You will need SSH access to your server to complete this task. PuTTY is available for free for this purpose from
here; configuration instructions are available
from Westhost.

Here are the steps you will have to take:

Create separate user accounts

  • Login in to your site manager
  • Click on EMail/FTP Management (Second item on menu bar)
  • Click on New E-mail/FTP button (in lower right)
  • A window titled “E-mail/FTP Wizard (1 of 3)” will pop up.
    • Enter the user details for recipient@domain1.com. Use a unique user name – I suggest recipient.domain1.com
  • Click on Next once you have entered user name, description and password
  • A window titled “E-mail/FTP Wizard (2 of 3)” will pop up.
    • Modify the email quota if desired. We can change this later by editing the file /etc/features
    • Enable the Autoresponder – it can be setup later by logging in to the the user account at http://www.domain1.com/users/
  • A window titled “E-mail/FTP Wizard (3 of 3)” will pop up.
    • Disable FTP access unless needed
  • Repeat the above for recipient@domain2. I suggest using a user name of recipient.domain2.com

Edit virtusertable

  • If you are comfortable with SSH then edit /etc/mail/virtusertable on the server or, alternatively, download a copy to your computer
    and edit it using a simple text editor such as Notepad
    (on a Windows PC). DO NOT USE A WORD PROCESSOR. Make sure you backup the file first.
  • virtusertable will contain various lines, but you should find these at the end
    recipient.domain1.com@domain1.com      recipient.domain1.com
    recipient.domain1.com@domain2.com   recipient.domain1.com
    recipient.domain2.com@domain1.com      recipient.domain2.com
    recipient.domain2.com@domain2.com   recipient.domain2.com
    
  • Change this to
    recipient@domain1.com      recipient.domain1.com
    recipient@domain2.com      recipient.domain2.com
    

    NOTE: the two columns of data MUST be separated by a TAB

  • Upload the changed file to /etc/mail/virtusertable

Create genericstable (optional)

If you want any applications or scripts on your server to send email from recipient@domain2.com, then follow these optional steps.

  • Create a new file /etc/mail/genericstable which contains
    recipient.domain1.com   recipient@domain1.com
    recipient.domain2.com   recipient@domain2.com
    

Create generics-domains (optional)

  • Create a new file /etc/mail/generics-domains which contains
    domain1.com
    domain2.com
    

Modify sendmail.mc (optional)

  • Add these two lines to /etc/mail/sendmail.mc
    FEATURE(`genericstable', `dbm /etc/mail/genericstable')dnl Added two lines
    GENERICS_DOMAIN_FILE(`/etc/mail/generics-domains')dnl
    

Activate changes

  • Login to your server using SSH
  • Generate a new copy of virtusertable.db by entering the command makemap hash /etc/mail/virtusertable </etc/mail/virtusertable
    (If you are not hosted by Westhost, then check your /etc/mail/sendmail.cf file for the type of database used by virtusertable. This will be either hash (as above) or dbm)
  • If you completed the optional steps above, you will also need to
    • generate a copy of genericstable.db by entering the command makemap hash /etc/mail/genericstable </etc/mail/genericstable on one line.
    • regenerate your sendmail configuration file if you made the optional changes by typing m4 /etc/mail/sendmail.mc >/etc/mail/sendmail.cf (again on one line)
  • Logout of your SSH session (logout)

Configure your email client

  • You will need to set up two accounts
  • For domain1.com
    • Your Name: Recipient One
    • EMail Address: recipient@domain1.com
    • Incoming Mail Server: pop.domain1.com
    • Outgoing Mail Server: smtp.domain1.com
    • User Name: recipient.domain1.com
    • Password: domain1
    • NOTE: User Name and Password are case sensitive. In other words, recipient.domain1.com is not the same as Recipient.Domain1.Com
    • Under More Settings:
      • Click on Outgoing Server
      • Select “My outgoing server requires authentication”
      • Select “Use same settings as my incoming mailserver”
  • For domain2.com
    • Your Name: Recipient Two
    • EMail Address: recipient@domain2.com
    • Incoming Mail Server: pop.domain2.com
    • Outgoing Mail Server: smtp.domain2.com
    • User Name: recipient.domain2.com
    • Password: domain2
    • NOTE: User Name and Password are case sensitive. In other words, recipient.domain2.com is not the same as Recipient.Domain2.Com
    • Under More Settings:
      • Click on Outgoing Server
      • Select “My outgoing server requires authentication”
      • Select “Use same settings as my incoming mailserver”

Distribution lists

These can be created by editing /etc/mail/aliases