Date: Wed, 10 Jul 2002 00:19:12 -0400 (EDT) From: BCPL.NET SysAdmin To: BCPL.NET News Subject: BCPL.NET NEWS: Triple Whammy Tuesday --------------------- TRIPLE WHAMMY TUESDAY --------------------- Tuesday July 9 was NOT one of the better days in the history of BCPL.NET Internet Services or Baltimore County Public Library computer operations. I'll try to keep this short, but it won't be easy. Whammy One ---------- Shortly after midnight Tuesday morning I installed a cluster of software patches on our main UNIX server (mail.bcpl.net). The installation went well, so I called it a night at about 1:30 AM. What I didn't notice was that the patch installer changed the permissions on several directories in such a way that the mail server software could not deliver mail to customers' mailboxes. Instead it was queued for future delivery. I discovered and fixed the permissions problem at about 6:00 AM, but by then the mail queue was so huge the mail server bogged down trying to process it. By shortly after noon it became clear that the mail server was losing the battle. I saved the existing mail queue under a new name, started a new queue to allow prompt processing of new incoming mail, and started a separate instance of the mail server software to continue processing the old mail queue. This means that any mail received for you by the server since the queue was split should have been delivered to your mailbox fairly quickly. Any mail received for you from about midnight through the time the queue was split may have been delivered to your mailbox by now, or may still be in the old queue waiting to be processed. Whammy Two ---------- At 1:45 PM our building had several short power outages in quick succession. The last one fried the refrigerator-sized uninterruptable power supply in our computer room. Everything powered by that UPS shut down, bringing BCPL.NET Internet access and all of the library's computer operations to a dead stop. By about 2:35 PM we had bypassed the UPS and brought everything back to life. Well, almost everything... Whammy Three ------------ One of BCPL.NET's dialup access servers didn't survive the power fluctuations. A replacement will arrive on Thursday or Friday, but until then our modem pool down to 874 usable modems instead of the normal 920. An Ethernet switch also fried, isolating parts of the network for a few minutes until we swapped in a spare switch. Worst of all, the Tandem mainframe that runs the library's circulation system, catalog, borrower database, and materials database objected to the sudden loss of power. It took the Library's ILS staff several hours of hard work to coax it back to useful life. It was NOT a good Tuesday. I hope yours was a LOT better! Chip -- BCPL.NET INTERNET SERVICES 320 York Road Towson, MD 21204-5179 U.S.A. CONTACTS: -------- Web Site: http://www.bcpl.net Administration & Policy: ispadmin@bcpl.net 410-887-6180 Sales, Renewals, Account Status: accounts@bcpl.net 410-887-4172 Technical Support (Help Desk): help@bcpl.net 410-887-3297 Usenet News Newsgroup Requests news-admin@bcpl.net 410-887-6180 E-Mail & Newsgroup Abuse Reports: abuse@bcpl.net 410-887-6180 Domain Name Service Issues: dnsadmin@bcpl.net 410-887-6180 FAX: 410-887-2091