Date: Fri, 11 Oct 2002 23:30:20 -0400 (EDT) From: BCPL.NET SysAdmin To: BCPL.NET News Subject: BCPL.NET NEWS: Recent Mail Problems ------------------------------- THE RECENT MAIL SYSTEM PROBLEMS ------------------------------- As many of you noticed, we had some serious mail system problems from late night Tuesday October 8 through mid afternoon Wednesday October 9. I apologize for not posting this explanation earlier, but I wanted to wait until we were sure the problem had been solved. Here's what happened: We have all seen a rapid increase in the volume of spam (junk e-mail) over the past few months. This week it was especially bad. One BCPL.NET customer who shall remain unnamed was receiving thousands of spam messages per day. That's the undisputed BCPL.NET record as far as we know. He got so fed up with it that he decided to fight back by setting up the "vacation" program to respond automatically to every message he received. Trouble is, he didn't stop to consider the fact that almost all spam is sent under phony "From:" addresses, so most of his auto-responses were undeliverable. They just caused tons of "message undeliverable" reports to be sent back to him, virtually one report for every spam he received.. In short, he was in effect mail-bombing himself. That, plus an unusually high volume of mail being received by the system, overloaded the mail server's delivery queue so badly that the server slowed to a crawl. I didn't become aware of the problem until some time Wednesday morning. The customer's self-inflicted mail-bombing was easily stopped, but cleaning up the aftermath took a while. It didn't help at all that an Ethernet switch on our network died in the midst of all this. It didn't affect access to BCPL.NET in any way, but did divert my attention away from the mail system cleanup for a while. By mid afternoon on Wednesday the mail system was working smoothly again. As far as I can tell no mail was lost, but some of your mail may not have reached your mailbox until a long time after it was sent to you. This was because I had to halt the original over-stuffed mail queue, start a new queue to handle current messages, and postpone processing of the mail in the old queue until later on. If mail intended for you was in that old queue, delivery may have been delayed for several hours. There was a beneficial side effect of this mess. While sorting things out we were able to identify a large number of mass mail sites that were sending a non-stop barrage of spam to BCPL.NET customers. We are now blocking mail from those addresses, and the result has been very gratifying. For the past several months our mail server has been rejecting 3000 to 4000 messages per day as a result of our earlier spam blocking efforts. On Thursday, the first full day after the additional mass-mail sites were blocked, the mail server rejected an astounding 37,055 messages. Friday's statistics look like they will be similar, with almost 35,000 blocked as of 10:30 PM. I'm sure a lot of spam is still slipping through, but this has put a big dent in it. Here's hoping for a quiet weekend! Chip -- BCPL.NET INTERNET SERVICES 320 York Road Towson, MD 21204-5179 U.S.A. BCPL.NET CONTACTS: See http://www.bcpl.net/contacts/