mod-gzip – How To Download pages faster and Save Server Bandwidth

I recently ran these pages through a Web Optimiser and the speeds were not that impressive, so I have decided to implement mod_gzip. This will compress the textual portion of these pages so that they download quicker.

The procedure will NOT follow the familiar procedure for those who follow this site, so read on…

  • Download the source from Sourceforge.net using the command wget, for example wget http://easynews.dl.sourceforge.net/sourceforge/mod-gzip/mod_gzip-1.3.26.1a.tgz. Choose the mirror nearest to your server. If you are running Apache 2.0, then you will need to donload the module from here instead.
  • Unzip this file and extract the contents so that we can get on with the build gunzip -c mod_gzip-1.3.26.1a.tgz | tar x
  • Change into the source directory cd mod_gzip-1.3.26.1a.
  • The directions for building mod_gzip are included at the end of the ChangeLog file.

    There are two ways to build mod_gzip:

    1. statically compiled into Apache and
    2. a DSO-File for mod_so.

    The DSO-Version is much easier to build. Just type

    make APXS=/path/to/apxs
    make install APXS=/path/to/apxs
    /path/to/apachectl restart

    The apxs script is normaly located inside the bin directory of Apache.

    The path to APXS is /usr/local/apache/bin/apxs or /usr/sbin/apxs, so we will use the command
    make APXS=/usr/sbin/apxs >log.make 2>err.make
    NOTE:You will get an error complaining about ok_to_send being unused. This Warning can be safely ignored.
    mod_gzip.c: In function `mod_gzip_sendfile2':
    mod_gzip.c:5262: warning: unused variable `ok_to_send'

  • The final step of the build process is to install the module. This will copy the library to the relevant place and edit your httpd.conf file.
    make install APXS=/usr/sbin/apxs >log.install 2>err.install
    The installer inserted two lines into the httpd.conf file, which are commented out. However, it did not quite get it right on my system, so it is just as well that they are commented out! The lines read

    <IfDefine HAVE_FRONTPAGE_SPHERA>
    LoadModule frontpage_module modules/mod_frontpage_sphera.so
    #LoadModule gzip_module /usr/lib/apache/mod_gzip.so
    </IfDefine>

    and should have read


    <IfDefine HAVE_FRONTPAGE_SPHERA>
    LoadModule frontpage_module modules/mod_frontpage_sphera.so
    </IfDefine>
    <IfDefine HAVE_GZIP>
    LoadModule gzip_module /usr/lib/apache/mod_gzip.so
    </IfDefine>

    The variable HAVE_GZIP is automatically created by the start up process. The installer also made the same mistake with the AddModule directives which read
    <IfDefine HAVE_FRONTPAGE_SPHERA>
    AddModule mod_frontpage_sphera.c
    #AddModule mod_gzip.c
    </IfDefine>

    instead of

    <IfDefine HAVE_FRONTPAGE_SPHERA>
    AddModule mod_frontpage_sphera.c
    </IfDefine>
    <IfDefine HAVE_GZIP>
    AddModule mod_gzip.c
    </IfDefine>

  • There are still a couple of changes to make to the httpd.conf file according to Apache’s own website. The following needs to be added to the configuration file; I added it just after the AddModule (within the define block)
    mod_gzip_on Yes
    mod_gzip_can_negotiate Yes
    mod_gzip_dechunk Yes
    mod_gzip_minimum_file_size 600
    mod_gzip_maximum_file_size 0
    mod_gzip_maximum_inmem_size 100000
    mod_gzip_keep_workfiles No
    mod_gzip_temp_dir /usr/local/apache/gzip
    mod_gzip_item_include file \.html$
    mod_gzip_item_include file \.txt$
    mod_gzip_item_include file \.jsp$
    mod_gzip_item_include file \.php$
    mod_gzip_item_include file \.pl$
    mod_gzip_item_include mime ^text/.*
    mod_gzip_item_include mime ^application/x-httpd-php
    mod_gzip_item_include mime ^httpd/unix-directory$
    mod_gzip_item_include handler ^perl-script$
    mod_gzip_item_include handler ^server-status$
    mod_gzip_item_include handler ^server-info$
    mod_gzip_item_exclude file \.css$
    mod_gzip_item_exclude file \.js$
    mod_gzip_item_exclude mime ^image/.*

    Additional filetypes can be defined. For example, there is no need to compress PDF files as they are already compressed, so we could also add
    mod_gzip_item_exclude mime ^application/pdf$
  • You will also need to create the mod_gzip_temp_dir specified in httpd.conf. mkdir /usr/local/apache/gzip
  • Next restart Apache and check that everything is still working. If you get the error
    Syntax error on line xxx of /etc/httpd/conf/httpd.conf:
    Cannot add module via name 'mod_gzip.c': not in list of loaded modules
    then you forgot top remove the # from the LoadModule command or mod_glib.so is not in the correct directory.

Optimising your broadband connection

I had some problems achieving the expected throughput from my broadband supplier. I have a nominal 24 Mbps ADSL2+ connection and should be achieving downloads of 14 Mbps. These are the steps to follow to improve your downloads.

First, establish whether the problem is upstream or downstream of your router.

  • Find out how far you are from your telephone exchange. The only way to get a true value is by running the line test facility, but BT have restricted this to authorised personnel. So, in the meantime raise a ticket with your provider and get them to find out. Alternately, you can get a rough idea by plugging your phone number and postcode into the Broadband Checker over at DSLZoneUK and you’ll get a pretty map. . I cannot find this on SamKnows anymore…
    Example: 460m
  • Calculate the route using Multimap to drive it for you. This will give you a rough guide of your possible cable length; they don’t always follow the most logical routing and if you’re in a rural environment, the cables will cut across fields.

    Example: 700m
  • Download and run DMT from http://dmt.mhilfe.de/. This page is in German, but just search for your modem and download the right version; this is v7.31 for the Speedtouch 780. You might be able to use this link for a Google translation of their site.
  • Look in the bottom right of the third graph (Channel Characteristics). This will give you the estimated loop length. This should be similar to your twice the cable length.

    Example: 830m-2015m
  • Go to router’s web interface and check what your current Sync speed and Line Attenuation are.

    Example: Bandwidth (Up/Down) [kbps/kbps]:394 / 18,106 —> 18106 kbps –> 17.7 Mbps (18106/1024)
    Line Attenuation (Up/Down) [dB]: 7.5 / 19.5 —> 19.5
  • Look up your expected Sync speed using either this interactive tool or the graphs provided by afster or Internode graph. Take 10% off the figure provided by the interactive tool to get a realistic sync speed.

    Example: Approx 18.2 Mbits/sec according to average graph
  • IF your sync speed is significantly lower than the expected value OR your Line Length is higher than predicted, THEN start looking at wiring issues. If not, then check OS parameters.

This graph shows the real life results from hundreds of users, relating their sync speed to the downstream attenuation reported by their router. Those above the green line are lucky :)

Next optimise your PC’s performance. You will need to be logged on as a user with Adminstrative privileges. If you only have one account, then you will have the required privileges.

  • Download and run TCP Optimiser from SpeedGuide.net. Click on Optimal settings, Apply changes and exit.
  • Reboot your computer.
  • Verify that your Network card is configured for maximum throughput (100 Mbps Full Duplex). To do this, click on
    • Start
    • Control Panel
    • Network and Internet Connections
    • Network Connections
    • Select your network card and click on Change Settings of this Connection or right click on the card and select Properties…
    • Click on the Configure… button next to the Network card name
    • Click on the Advanced tab.
    • Set the appropriate property to 100 Mbps Full-Duplex. On my network card, this is called Media Type
      There are other tweaks that you can do here, such as disable Rx and Tx Checksum Offload. I am not sure of the advisability of doing this.
    • Click on OK
  • Run Microsoft’s Guided Help to maximise your PC’s performance. WARNING! This will disable all themes and turn your PC into retro mode ;) Further details can be found in this knowledgebase article
  • Minimise the window during downloads where possible. It is possible that your graphics processor cannot keep up with the download :)
  • Check CPU usage during downloads and investigate any processes which are consuming too much. Anti Virus packages such as Kaspersky have been known to cause problems during downloads, so try disabling them TEMPORARILY while running any download tests.
    • Right click on a blank area of the taskbar at the bottom of your screen
    • Click on Task Manager
    • Click on the Processes tab and then double click on the column labelled CPU. If the CPU column is not displayed, then click on View and Select columns… then enable the CPU Usage option.
    • The process using the most processor time should be at the bottom of the list
  • Check your memory usage during downloads and investigate any processes which are consuming too much. If the Memory Usage column is not displayed in the task manager, then click on View then Select columns… and enable the Memory Usage option.

All speed checks should be run using a properly shielded Ethernet connection, even if your Wireless connection claims an ‘Excellent’ signal strength. If everything is OK, then we should be able to get download speeds at upto 80% of our sync speed.

NOTE: Speed Tests are to be used as a guideline only. As one commentator said

Speedtest.net shows me at 9000kbps, the Be* test puts me at 9500kbps, Thinkbroadband puts me at 16000kbps and the Virgin Media FTP puts me at about 8000kbps.

All of which are dramatically wrong, since I am synced at over 24000kbps and can download at 2MB/sec from some FTP’s on a single thread, let alone multi threading.

Securing and Configuring the SpeedTouch 780 WL

I have just received a SpeedTouch 780 WL Residential ADSL Router with VOIP from Be* Un Limited.

The router comes preconfigured with a username of Administrator and a blank password. In addition, there are also a number of other users preconfigured for use by Be* Technical support.
There are two ways to access the router; you may either use the graphical interface or the command line interface. The graphical interface is simpler to use, but is limited in its functionality.

  1. Connect the router to your computer

    You will need to use a cable for this first stage of the configuration.This is for your own security. Also, we are going to be changing some wireless parameters and we don’t want you to be locked out.

  2. Backup your current setup

    • Open your browser and navigate to http://192.168.1.254/. This is the default address for your router’s configuration interface. You can also access the router over a secure link by accessing https://192.168.1.254/, https://bebox/, https://speedtouch/ and https://dsldevice/. Note that the SpeedTouch uses a self signed certificate, which your browser will not recognise by default and may issue some security alerts if using HTTPS; you can proceed safely. You should always use HTTPS when accessing your router’s interface wirelessly.
    • If necessary, login using the username Administrator and a blank password.
    • You should be presented with your router’s home page. Your current username is shown in square brackets thus [Administrator] between the two horizontal lines at the top of the page. This region is known as the “Notification Area”
    • Click on the the Speedtouch button in the left hand column and then click on the Configuration hyperlink. At the bottom of the page you will see Save or Restore Configuration. Click it.
    • Click on the Backup Configuration Now button and save user.ini to your hard drive
  3. Change the default user and password

    • Click on the Toolbox in the lefthand menu bar
    • Click on User Management. You will be presented with a list of current users.
    • Click on Change My Password and enter a new password for the Administrator account. We will disable this account later, but let’s make life difficult for any hacker. When you click on the Change Password button, you will be presented with a login screen. Login as Administrator with the password you have just created.
    • Now create a new user by clicking on Add a new user. Create a new user with Administration Privileges set to Administrator. Click the Apply button to create the new user. The default password will be the same as their name.
    • Click on Switch to another user. If you are not presented with a login prompt, then click the link again. This time, login as the new user you created. Notice that the username has changed in the Notification Area.
    • Click on Change My Password again, but this time enter a new password for the new user’s account. Click on the Change Password button, and login as the new user, with the password you have just created.
    • Click on Administrator in the list of usernames and change their Administration Privileges to User (the most restricted) and click on Apply. There are eight predefined privilege levels.
      Role Access Rights
      root Any service and any access from LAN/WAN/LOCAL
      SuperUser Any service and any access from LAN/WAN/LOCAL
      TechnicalSupport Any service and any access from WAN
      Administrator Any service and any access from LAN/Local. No access from WAN
      PowerUser GUI (Service/overview page) via http/https from LAN origin
      WAN_Admin Only WAN related configurations from any Channel/Origin
      LAN_Admin Only LAN related configurations from any Channel/Origin
      User GUI (Overview page/Remote Assistance) via http/https from LAN origin
    • That is as much as you can do from the GUI interface. You can delete the Be* backdoors and the Administrator account if use the CLI
  4. Configuring Wireless Access

    IF YOU ARE NOT USING WIRELESS, THEN MAKE SURE YOU DISABLE IT!

    • Click on Home Network in the lefthand menu.
    • Click on WLAN:BeBox under Wireless in the list of interfaces
    • Click on Configure in the Notification Area
    • If you are not using Wireless access, then clear the check mark against Interface Enabled and click the Apply button.
    • If you are using Wireless access, then
      • Change the Network Name (SSID) from BeBox
      • Clear the check mark against Broadcast Network Name
      • Set Allow New Devices to New stations are allowed (via registration)
      • Set Encryption to Use WPA-PSK Encryption
      • The default WPA-PSK Encryption Key is a ten digit Hexadeciaml number (64 bit encryption). It should be be set to a random, 26 digit, Hexadecimal number for maximum security (128 bit encryption) (Example:FADDC2077AF10406E866984C9E). A Hexadecimal character is any of the numbers 0-9 and the letters A-F. Download this Excel spreadsheet to generate a key for you if you want to. Make a note of this number as you will need it later.
      • Set the WPA-PSK version to WPA2.
      • Click the Apply button to save your changes.
    • Now you need to configure your PC.
      • Click on the Start button
      • Click on Control Panel
      • If you are using Category View, then click on Network and Internet Connections
      • Click on Network Connections
      • Right Click on your Wireless Network Card and select Properties
      • If you are not going to use your Wireless link to connect to another computer on your home network, then deselect Client for Microsoft Windows and File and Printer Sharing for Microsoft Networks on the General Tab.
      • Select the Wireless Networks tab
      • Click on the Add button in the Preferred Networks pane
      • Enter the Network Name (SSID) of your Wireless router
      • Select WPA2-PSK from the drop down list for Network Authentication. If WPA2-PSK is not an option, then either your Wireless card does not support WPA2-PSK and you will need to use WPA-PSK instead OR you need to download the WPA2 update from Microsoft. Try the download first.
      • Set Data Encryption to TKIP
      • Enter the same 26 digit Hexadecimal number as you generated earlier in the Network Key and Confirm Network Key fields.
      • Click on OK to confirm your changes.
    • Even if you have done everything properly, you will still not be able to connect Wirelessly to your router until you have registered your Wireless card with it. The simplest way to do this is to press the button on the front of your router when trying to connect to it. You will then have one minute to connect. Once your card has been registered, you will not need to repeat this.
      The alternative way to register your card is via the GUI interface. Just click on Home Network, WLAN:YourSSID and then click Search for Wireless Devices when you try to connect to the router.
  5. Checking your security

    • Go to Gibson Research and run a ShieldsUP! test. The link is near the bottom of the page in the “Hot Spots” section.
    • Follow the instructions and run the Common Ports or All Service Ports ShieldsUP! service
    • If everything is OK, then you will get a TruStealth PASSED stamp. I had to close port 0 in order not to reply to pings. Replying to a ping (or ICMP request) is not a problem in itself, however it lets a hacker doing a port scan know that there is a potential target there.
    • In order to close a port, you will have to use the CLI to configure your router.
      • Connect to your router by typing telnet 192.168.1.254 from the command prompt of your computer
      • Login using your router’s username and password
      • Enter the command service system ifdelete name=PING_RESPONDER group=wan
      • Save the changes by entering the command saveall
      • End your session by typing exit
      • Other ports you should close (unless you NEED to access your router remotely)
        Port Command to delete
        21 service system ifdelete name=TELNET group=wan
        23 service system ifdelete name=FTP group=wan
        443 service system ifdelete name=HTTPs group=wan

        You can get a full list of services by issuing the command service system list

Installing and Running Rsync

DRAFT POSTING

This post is very long, so I have broken it down into multiple pages.
Page 1 – Installing rsync daemon on your server
Page 2 – Installing rsync client on your PC
Page 3 – Scheduling rsync automatically
Page 4 – Errors I encountered and their solutions


Download and build rsync from source

  • Start a SSH session to your server
  • Download the source from samba wget http://samba.anu.edu.au/ftp/rsync/rsync-2.6.9.tar.gz
  • Extract the source code gunzip -c rsync-2.6.9.tar.gz | tar x
  • Change into the directory so that we can build the software cd rsync-2.6.9
  • You will need the GCC compiler installed if you have not already done so.
  • Run the auto configure script so that the server can work out if everything necessary is present ./configure –prefix=/usr/mylocal –with-included-popt >log.config 2>err.config
  • Check that no errors were logged by the configure script less err.config
  • Build the application make >log.make 2>err.make
  • Check that no errors were logged during the build process less err.make
  • Install the application make install >log.install 2>err.install
  • Check that no errors were logged during the installation less err.install

Configure the rsync daemon

We cannot use the default port (873) on Westhost as this is used by Westhost’s own application for nightly backups. Nor can we use ports below 1024 as these require us to run as root. However, ports 8730-8732 are unassigned by IANA (Internet Assigned Numbers Authority), so we can use any of these.

  • Create a directory for your configuration files. mkdir /etc/rsyncd
  • Create a new configuration file pico /etc/rsyncd.conf. The configuration file consists of a general configuration section which applies to every module and one (or more) modules. Each module starts with its name in square brackets, for example [rsyncd_module].
    This sample configuration file will allow upto 3 rsync clients to read the contents of /ftp/pub/rsync PROVIDED that they authenticate themselves by providing the correct credentials AND their IP Address is in the range specified by hosts allow. The connection will close down after 5 minutes (300 seconds) of inactivity.

    #Global definitions
    #Message of the Day
    motd file = /etc/rsyncd/rsyncd.motd
    log file = /var/log/rsyncd.log
    pid file = /var/run/rsyncd.pid
    lock file = /var/run/rsyncd.lock
    port = 8730

    #Module Options begin here
    [rsyncd_module]
    path = /ftp/pub/rsyncd
    comment = My Very Own Rsync Server. This area is ReadOnly
    max connections = 3
    timeout = 300
    uid = myuserid
    gid = vuser
    read only = yes
    list = yes
    auth users = rsync_user
    secrets file = /etc/rsyncd/secrets
    hosts allow = xxx.xxx.xxx.xxx/xx
    hosts deny = *

    Replace the module name, myuserid, vuser and rsync_user as necessary for your system.
    If you want to limit those who have access to your rsync server, then specify the “auth users” and “secrets file”. If you want to restrict where they can access the rsync server from, then you will also need the “hosts allow” and “hosts deny” variables. The value for “hosts allow” should be set to your IP address if you use a static IP (example 10.0.0.1/32) or the range if you use a dynamic IP (example 10.0.0.1/16). Multiple possibilities can be separated by a space. If you don’t understand this bit, then leave “hosts allow” and “hosts deny” out of your configuration for now.

  • Create the secrets file pico /etc/rsyncd/secrets. Format is username:password in plain text, one user per line. If you want to communicate over SSH, then ensure that one of the users is your account’s user id; it need not have the same password here as for logging in.
  • The secrets file must not be readable by other users, so change the access rights by using the command chmod 600 /etc/rsyncd/secrets
  • Create your Message Of the Day file pico /etc/rsyncd/rsyncd.motd. This text will be displayed when a connection is made to your server.