Sun 29 Jul 2007
Custom Error Page returns status 200
Posted by Richard under Linux , Configuration , SoftwareNo Comments
I was recently playing with Google’s webmaster tools and ran into a problem with my custom error page. Although I have had it set up for some time, it was apparently incorrectly configured. Google complained ‘We’ve detected that your 404 file (file not found) error page returns a status of 200 (Success) in the header. ” which means that the page was found and they’re not happy.
In order to troubleshoot this, I connected to my web server using telnet. This enabled me to see the exact data that Apache is delivering in reply to the request. This is what you need to do
- Open a Command prompt window (run cmd.exe on a PC)
- Connect to the web server (Apache) using telnet. If your web server does not run on port 80, then use the correct value. telnet mydomain.com 80
- Ask for the header of a non existent page by typing HEAD invalidpage.html HTTP/1.0. Although the cursor moves, NO TEXT WILL APPEAR ON THE SCREEN.
- Press the Enter key twice and wait for a reply
The reply you get should look something like this
HTTP/1.1 404 Not Found
Date: Fri, 30 Mar 2007 11:30:22 GMT
Server: Apache/2.0.52 (Red Hat)
Last-Modified: Fri, 30 Mar 2007 09:56:45 GMT
ETag: “13148ae-4fd-e39fb140;741a7f80″
Accept-Ranges: bytes
Content-Length: 1277
Connection: close
Content-Type: text/html
Connection to host lost.
When I tested my site, I found that it was responding
HTTP/1.1 302 Found
Date: Fri, 30 Mar 2007 11:33:22 GMT
Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_ssl/2.8.12 OpenSSL/0.9.6b PHP/4.4.2
Location: http://xxx.xxx.xxx.xxx/404.html
Connection: close
Content-Type: text/html; charset=iso-8859-1
Connection to host lost.
So my server was returning a status code of 302 which is a temporary redirect. When Google followed this to the new location, it received the 200 status code as the ultimate file exists. I tried adding the line
<meta http-equiv="Status" content="404 Condition Intercepted" />
in the head section of my error file as suggested on another site, but it made no difference. The real problem was that I had failed to notice the comments in the Apache manual
Note that when you specify an ErrorDocument that points to a remote URL (ie. anything with a method such as http in front of it), Apache will send a redirect to the client to tell it where to find the document, even if the document ends up being on the same server. This has several implications, the most important being that the client will not receive the original error status code, but instead will receive a redirect status code. This in turn can confuse web robots and other clients which try to determine if a URL is valid using the status code.
I wanted to use the same error page for multiple domains hosted on my VPS and so had hardcoded the IP Address into the URL. The line in my Apache configuration file read ErrorDocument 404 http://xxx.xxx.xxx.xxx/404.html. I therefore changed it to ErrorDocument 404 /errorp/404.html and created an alias for /errorp/ Alias /errorp/ “/path/to/custom_errors/”. My custom error page now responds correctly 
HTTP/1.1 404 Not Found
Date: Fri, 30 Mar 2007 12:43:22 GMT
Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_gzip/1.3.26.1a mod_ssl/2.8.12
OpenSSL/0.9.6b PHP/4.4.2
Last-Modified: Fri, 30 Mar 2007 12:12:48 GMT
ETag: “2c6439c-261c-460d8b60;46a7f31a”
Accept-Ranges: bytes
Content-Length: 9756
Connection: close
Content-Type: text/html
If you have to use http: in your Custom Error Page's definition, then you are better off using a redirect in a local file. This will preserve the error status code. Your local error file should look something like this
<HTML><HEAD>
<META http-equiv=”refresh” content=”0; URL=http://www.example.com”>
</HEAD>
<div style=”visibility:hidden”>
;Lets insert some random text here in order to keep Microsoft Internet
;Explorer happy. In their wisdom, Microsoft have decided that any custom
;error page with less than five hundred and twelve characters of text
;in it will be totally ignored and they will display their Friendly HTTP Error
;message instead. This is detailed in knowledge base article Q294807
;at http://support.microsoft.com/default.aspx?scid=kb;en-us;Q294807
;Well, I think that that is enough from me for now or maybe I should
;just type a little bit more to make sure that I have typed enough…
</div></BODY></HTML>
Further details can be found in 
