--------------------
WEB SERVER LOG FILES
--------------------
This is directed at those of you who extract statistics for your Web pages
from the Web server's log files.
 
Once again the crontab script that resets the log files at the end of each
month has failed, and the log files for August are lost. Our apologies to
all you statistics keepers out there, but I think this time we have a
workable solution that will prevent the problem in future. 
 
The failure seems to be related to the size of the log files, which are
located in the /carl/httpd/logs/ directory.  The huge size of the
access_log file is the main problem.  By about 11:30 PM on August 31, the
access_log for August had grown to over 222 megabytes due to the huge
number of hits on our Web server.  BCPL has become a very popular Web
site, due in no small part to the excellent Web pages published by some of
our account holders. 
 
The huge size of access_log also causes other problems.  As access_log
grows larger, Statbot and other statistics collection utilities take
longer and longer to run.  This places a heavy load on our UNIX host's
processor.  Toward the end of the month when access_log has grown to
enormous size, one person running Statbot can easily bring our processor
idle time down to zero for an extended period of time. Two or more people
running Statbot at the same time can bring the system to its knees. 
 
Also, in order to preserve enough room on the /carl hard disk to
accommodate the inexorable growth of access_log, it has become necessary
to zero out the rest of the log files (agent_log, error_log, and
referer_log)  on a daily basis toward the end of each month. 
 
As a solution to all three problems we are revising the management
schedule for the Web server logs.  Effective September 1 the logs will be
restarted at the beginning of every week instead of at the beginning of
every month.  At 12:01 AM every Sunday the following will occur: 
 
o Rename access_log to access_log.0 and create new empty access_log.
o Rename agent_log to agent_log.0 and create new empty agent_log.
o Rename error_log to error_log.0 and create new empty error_log.
o Rename referer_log to referer_log.0 and create new empty referer_log. 
 
Thus a log file with a ".0" extension will always contain statistics for
the preceding week, and a log file without the extension will always
contain statistics for the current week.  This will keep the log files
down to a reasonable size, with the following benefits: 
 
o It will reduce the risk of losing the logs when the weekly crontab
  script runs.
o It will reduce the run time for Statbot and other statistics utilities.
o It will eliminate the need to constantly monitor disk space on the /carl
  file system.
 
The new schedule will make it necessary for many of you to revise your
statistics gathering routines.  We apologize for that, but there seems to
be no alternative.  If anyone can suggest another solution to the problem,
we will be more than happy to consider it! 
 
If you have questions about this or about anything else pertaining to your
BCPL Internet account, please contact the BCPL Help Desk by telephone
(887-3297) or by e-mail (oz@mail.bcpl.lib.md.us).