| |
 Tech Note: Fighting Spam with the FirstClass mail server
FirstClass 7.1 marks the introduction of several new facilities to help the admin and end user deal with the problem of unsolicited email (SPAM). With all the facilities put together they create a very powerful, and very flexible method of fighting spam from the admin level down to the end user level. The following document outlines each of these facilities in the order in which they get processed by your FirstClass server, and shows how all the facilities work together to reduce the amount of SPAM that arrives in a FirstClass mailbox. For more info on configuring / customizing the set up on your FirstClass server, I refer you to the "Internet Services Administrator's Guide / Managing system security" documentation located in your "Admin Help / Online Books" folder.
 • Filter documents - Reject connections based on Filters
The first line of defense is that of using your Filters folder in conjunction with the "Reject connections based on Filters" option located on the "Internet Services / Basic Internet Setup" form, "UCE/Spam" tab.
 If you have this option checked and have a list of IP addresses in your Filters folder, then any connection from those IP addresses will be rejected by FirstClass. This option is very efficient since no processing needs to be done on the message. As soon as the sending server makes the connection to your FirstClass server, we know the IP address it is connecting from and can do a search of your filters for a match then terminate the connection if a match is made.
To use this you need to know the IP addresses which the Spam mail is coming from. This can be obtained by opening a piece of spam mail and then choosing the "View" menu and "Show Internet Header". The first "Received:" header will show the IP address the messages has arrived from. The below example shows a piece of spam arriving from the IP address of: 157.156.176.106. You could then place that IP address into any of your filter files to block it. After making any changes to the Filters folder you will need to click on the "Get Config" button located on the "Internet Services Monitor" for the changes to take effect.
Return-Path: <Amparobqvo@online-shop-exchange.com>
Received: from mdoqh ([157.156.176.106])
by mail.centrinity.com (FirstClass Mail Server v7.1) with ESMTP
(Sender: Amparobqvo@online-shop-exchange.com)
transient id 19406; Sat, 24 May 2003 17:18:46 -0400
NOTES:
1.) This method will only be effective if your SMTP traffic comes directly into your FirstClass mail server with out first being routed through another mail server. If all your mail is routed through another mail server before arriving at the FirstClass mail server, then it will of course always show as being received from the IP address of that one mail server.
2.) If you already have a hardware firewall, use it to protect your Internet Services machine, since it offloads the effort to a dedicated resource. But for quick blocking of a bad address or for sites that don't run a firewall, this is a good option.
• Reverse DNS Lookup - Reject unknown domain names
The next line of defense is the "Reject unknown domain names" option located on the "Internet Services / Basic Internet Setup" form, "UCE/Spam" tab.
 Rejects mail from any IP address unknown to your server. This option causes your server to perform a Domain Name Server (DNS) lookup of the IP address of any SMTP server or POP3 client that is trying to send it mail. If your DNS server cannot find the domain name, it rejects the mail. This is known as a reverse DNS lookup.
NOTES:
1.) Be aware that selecting this may cause your server to reject mail from legitimate sites with IP addresses that cannot be resolved back to their domain name by your DNS.
2.) Since this option relies on querying the DNS server on each inbound SMTP connection, make sure your DNS servers are functioning well in order to maintain good performance.
• RBL (Realtime Blackhole List) lookup - Reject based on RBL hosts(s)
The next facility or next line of defense is the Real Blackhole List (RBL) hosts option, located on the "Internet Services / Basic Internet Setup" form, "UCE/Spam" tab.
 This option will disallow connections from any IP addresses listed in the host RBL server(s).
RBL host name fields contain the URL's of the RBL service(s) you use.
Note
There are several Real Blackhole List (RBL) services you can use with varying degrees of aggressive IP lists. The following site contains a list of RBL servers and links to their sites.
We recommend you choose one or two good RBL services that are not too aggressive in their IP lists. For more information on configuring a RBL setup, see the FirstClass Internet Services Administrator's Guide.
The help text fields contain messages the sender receives that describes why the connection was refused and contains a link to the RBL site where corrective action can be taken. You will only have one message per RBL site.
An example of this is, "Your mail has been found on our RBL service list and will not be delivered. Go to rbl.spamcop.org for more information".
• X-RBL-Warning header instead of Nondelivery Notice (NDN)
 If this option is left unchecked then the FirstClass server sends an NDN followed by the information entered in the "Help Text" field for the particular "RBL host name #" on which the sender's IP address was located.
If checked, this option will not cause the message to be rejected with a NDN, but rather cause a "X-RBL-Warning:" header to be inserted into the internet header of the message informing the user that the sender's IP address has been found on a RBL service list. The recipient should implement personal mail rules (see online help) to handle future emails from this sender. Note: If you wish to use the X-RBL-Warning header option you need to still have the "Reject based on RBL host(s)" option checked as well.
You can, optionally, enter up to three RBL host servers in the UCE/Spam tab. If the sender's IP is not found on the first RBL, then the second host is tried an so on. If the IP address is not found on all three hosts or if all three hosts did not respond then the message will continue to be processed as normal.
• SMTP rules - rules.MailRules document
The next facility is the rules.MailRules file, located in the Filters folder, which allows the admin to use a scripting language to customize how FirstClass processes incoming SMTP mail messages. This section outlines what happens to a message as it gets processed by the default rules.mailrules document.
If you wish to read a more detailed document which explains in detail how to modify your rules.mailrules document in order to create custom rules then please see the following link: http://www.firstclass.com/CentrinityPerspectives/Terry_Whyte/ISMailRules
The rules.mailrules script uses a points system to determine the outcome of the inbound message. As various tests are performed against the message, a number of points are assigned to the message's Spam Level. The final rules then determine the message's destiny based the Spam Level which the message has received.
Default variable settings:
$CrosspostLimit=15 AND $CrosspostIncr=5
$XpostSpamlevel=20 AND $XpostIncrSpamlevel=5
If an inbound message arrives, cross posted to more than 15 (CrosspostLimit=15) email addresses, it will be given a Spam level of 20 (XpostSpamlevel=20), plus an additional score of 5 (XpostIncrSpamlevel=5) for each additional 5 (CrosspostIncr=5) email addresses past the original 15.
ie. A message addressed to 20 people would score a spam level of 25, while one addressed to 30 would score a spam level of 35, and so on.
$XtremeCausesNDN=0
By default no Spam Level will cause a message to be rejected with an NDN.
$LowSpamMin=10 AND $LowSpamMax=25
By default a message which scores between 10 & 25 points will be marked as a Low Spam Level.
$MedSpamMax=50
By default a message which scores between 25 & 50 points will be marked as a Medium Spam Level.
$HighSpamMax=100
By default a message which scores between 50 & 100 points will be marked as a High Spam Level.
By default a message which scores over 100 points will be marked as an Extreme Spam Level.
Default Rules.MailRules:
Trusted Addresses:
• Checks to see if the Sender address which the message is arriving from is trusted in any of your filter files. If you have the address trusted, (ie. entered as +TestUser@domainname.com, or +domainname.com), then it will not apply any mailrules to the message.
For FirstClass 8.x or later, in the Rules.Mailrules, the following lines need to have the comment "#" removed to activate this blessing for addresses or domains.
# The MAIL FROM address is easily spoofed, so less trustworthy
#
#^: IF (@IsTrustedAddress($Sender)) DONE
#
# The From header is easily spoofed, so less trustworthy
#
#From: IF (@IsTrustedAddress($From)) DONE
• Checks to see if the IP address the message is arriving from is trusted in any of your filter files. If you have the IP address trusted, (ie. entered as +199.198.197.196, or +199.198.197.*), then it will not apply any mailrules to the message.
Spammer Addresses:
• Checks to see if the IP address the message is arriving from is entered in any of your filter files as a spammer. If you have the IP address entered as a spammer, (ie. entered as 199.198.197.196, or 199.198.197.*), then it will then generate an NDN message and stop processing the mail rules.
Subject Rules:
• Checks to see if the message subject contains any of the words or phrases you have entered in the "rules.subjectblock" file located in the "Filters" folder. By default the "rules.subjectblock" file contains the following; "XXX", "Hot teen", and "ADV:". If it finds a match it will add 100 points to the Spam Level, and set the message as having failed the "SUBJECTBLOCK" test.
• Checks to see if the message subject contains 6 or more consecutive spaces.
(ie. "For You: XGFDCXRHT.exe") If it finds 6 or more consecutive spaces it will add 50 points to the Spam Level, and set the message as having failed the "SUBJECT_HAS_SPACES" test.
• Checks to see if the message subject contains only capital letters. If it does, it will add 25 points to the Spam Level, and set the message as having failed the "SUBJECT_ALL_CAPS" test.
"-ERRORS_TO;" Rule:
• Checks to see if there is an "Errors-To:" entry in the message's internet header. If it finds one it assumes it is a returned error message rather then spam. It then decreases the spam level by 20 points, and sets the message as having failed the "-ERRORS_TO;" test.
From Rules:
• Checks to see if the "From:" address contains a group of letters followed by a group of numbers followed by group of letters followed by a group of numbers (ie. "ghhgf432gvfgf455@spammersareus.com"). If it finds this type of formatting in the "From:" address it will add 25 points to the Spam Level, and set the message as having failed the "FROM_SUSPICIOUS" test.
• Checks to see if the "From:" address is found in any of your filter files entered as a spammer. If you have the address entered as a spammer, (ie. entered as TestUser@spammersareus.com, or @spammersareus.com), then it will add 101 points to the Spam Level, and set the message as having failed the "FROM_IN_SPAM_FILTERS" test.
Message-ID Rules:
• If a "Message-ID:" entry exists in the header, it checks to see if the field contains an "@" character. If this characters is NOT found in the "Message-ID:" field it will add 51 points to the Spam Level, and set the message as having failed the "INVALID_MSGID" test.
• If a "Message-ID:" entry exists in the header, it checks to see if the field contains any suspicious characters. If any are found in the "Message-ID:" field it will add 51 points to the Spam Level, and set the message as having failed the "INVALID_MSGID_2" test.
Crossposting Rules:
• Checks to see if the total number of addresses combined from the BCC, To, and Cc fields equal or exceed the crossposting limit which by default is set to 15. If the crosspost limit is exceeded it will by default set the Spam Level to 20 plus an additional 5 points for each additional 5 email addresses past the limit of 15 addresses. It will also set the message as having failed the "CROSSPOST_EXCEEDED" test.
Former built-in Junk Mail Rules:
• If a "X-Mailer:" entry exists in the header, it checks to see if the field contains any of the following; "Extractor", "Floodgate", "Group Mail", "Millennium Mailer", or "AutoMail". If it does contain any of these entries it will add 75 points to the Spam Level, and set the message as having failed the "X-MAILER" test.
• Checks to see if a message is addressed using only the BCC field. If it is, it will add 75 points to the Spam Level, and set the message as having failed the "NO_RECIPIENTS" test.
• Checks message header to see if it contains a "Message-ID:" entry. If the phrase "Message-ID:" is not found in the header it will add 51 points to the Spam Level, and set the message as having failed the "NO_MESSAGE_ID" test.
NOTE: Though the RFC's state that a message's internet header should contain a "Message-ID:", there are various mail servers / SMTP mail clients which do not follow this rule, and thus mail from them would by default fail this test and appear as Junk mail.
Default Rules based on Spam Levels:
Once the above rules have run, the message is then processed based on the total Spam Level it has been assigned.
• If the Spam Level is greater than the maximum High Spam Level which by default is 100, and the $XtremeCausesNDN variable is set to 1, then it will reject the message with the following NDN: "NDN 550 'Sorry, your message has triggered a SPAM block, please contact the postmaster" If this NDN gets sent then no more rules processing is done.
• If the Spam Level is greater than the maximum High Spam Level (default=100) then insert the following phrase into the message internet header: "X-SPAM-Warning: EXTREME".
• If the Spam Level is greater than the maximum Medium Spam Level (default=50) but less than than the maximum High Spam Level (default=100) then insert the following phrase into the message internet header: "X-SPAM-Warning: HIGH".
• If the Spam Level is greater than the maximum Low Spam Level (default=25) but less than than the maximum Medium Spam Level (default=50) then insert the following phrase into the message internet header: "X-SPAM-Warning: MEDIUM".
• If the Spam Level is greater than or equal to the minimum Low Spam Level (default=10) but less than than the maximum Low Spam Level (default=25) then insert the following phrase into the message internet header: "X-SPAM-Warning: LOW".
• If the Spam Level is greater than the maximum Medium Spam Level (default=50) then set the message priority as "Junk". This will be done by preceding the subject with the "Junk:" tag. If a user has the "Junk mail handling:" preference set to "Delete Silently", then they will never see this message in their mail box.
• If the Spam Level is greater than or equal to the minimum Low Spam Level (default=10) then insert the following phrase into the message internet header: "X-SPAM-Level:"<Spam Level>.
• If the Spam Level is greater than or equal to the minimum Low Spam Level (default=10) then insert the following phrase into the message internet header: "X-SPAM-Tests:"<Spam tests which the message failed>.
The following rules give the end user a visual indication of the level of Spam which has been set for the message.
• If the Spam Level is greater than the maximum Medium Spam Level (default=50) then set the message icon as the High Spam Level icon ID 23048.
 • If the Spam Level is greater than the maximum Low Spam Level (default=25) but less than than the maximum Medium Spam Level (default=50) then then set the message icon as the Medium Spam Level icon ID 23049.
 • If the Spam Level is greater than or equal to the minimum Low Spam Level (default=10) but less than than the maximum Low Spam Level (default=25) then set the message icon as the Low Spam Level icon ID 23050.
 If a user was to see a message with one of the above icons, viewing the internet header for the message will show exactly what the Spam Level was calculated at, and why it was marked as spam.
• Spam logging
There may be the occasional situation where a legitimate piece of inbound e-mail gets blocked by the above anti-spam facilities, and an end user will be asking why they can't receive mail from a particular person. In this case it would be nice to know why the particular message failed. Spam logging allows for you to track this info down.
Whenever Internet Services rules on the UCE/Spam tab take effect and block a message from arriving in a user's Mailbox, an entry is added to your server statistics file, for example:
Spam 27 10/22/2001 12:27:54 PM localhost 127.0.0.1 terry@127.0.0.1 Sender Extra
where,
Spam: keyword
The keyword for these entries is always "Spam"
27: FirstClass user ID
10/22/2002: the date
12:27:54 PM: the time
localhost: the host name
127.0.0.1: the IP address
terry@127.0.0.1: the mail address
Sender: the reason code
The reason code can be one of:
• MailRule
• Sender
• RBL
• ReverseDNS
• Host
• RelayReject.
Extra: the extra information
Extra information will contain:
• MailRule type - any NDN text sent
• RelayReject type - the address being relayed
Notes:
1.) RelayReject is not logged when all relaying is disabled.
2.) For info on how to enable Statistics logging please see the "Admin Help / Admin Help" folder and the "Server Maintenance / Server Monitoring" online help document.
3.) As long as you have a valid "Statistics folder path:" filled in, and logging is enabled, then the "Spam" logging will occur. There is no option / checkbox to turn the "Spam" logging on or off.
• Client side mail rules (end user configurable)
Junk mail handling:
The "Junk mail handling:" preference is located under the FirstClass client "Edit" menu, "Preferences..." form on the "Messaging/Mail Rules" tab.
If a user has the "Junk mail handling:" preference set to "Delete Silently", then any message marked as Junk will be deleted automatically. By default this applies to all messages with a spam level of HIGH which means they scored greater than 50 on the spam level.
 Mailbox Rules...:
This document only explains how to tie the Mailbox Rules... preference in with the system level rules.mailrules. For more info on using the "Mailbox Rules..." preference please see the FirstClass client online help.
The "Mailbox Rules..." preference is located under the FirstClass client "Edit" menu, "Preferences..." form on the "Messaging/Mail Rules" tab.
 As stated in the "rules.MailRules/Default Rules based on Spam Levels:" section of this document, various header entries are inserted into the message header when the message fails any Spam tests as defined by rules.MailRules. You can view these inserted headers by opening a spam message and selecting the "View" menu, "Show Internet Header" option.
The following sample is taken from the internet header of a spam message which failed the rules.mailrules SUBJECTBLOCK test and was there fore marked with a spam level of High.
X-SPAM-Warning: HIGH
X-SPAM-Level: 100
X-SPAM-Tests: SUBJECTBLOCK;
X-FC-Icon-ID: 23048
Using the client "Mailbox Rules...", we can create user specific mailrules which act on these inserted headers. Below are sample Mailbox rules which use these inserted headers to remove all Spam marked with a rating of Medium or above, by either moving the message to a folder and setting it's expiry to 1 week, or by simply deleting the message upon receipt. The following 4 examples also show the flexibility of the mail rules, using 4 different methods to accomplish the same result, getting Spam out of your Mailbox.
    You can also create a Mailbox rule which acts against Spam which failed a specific Rules.mailrules test such as the SUBJECTBLOCK, as shown in the example below.
 If the administrator has chosen the "X-RBL-Warning header instead of Nondelivery Notice (NDN)" option in order to let the users decide what Spam to keep or not, then you can also use a Mailbox rule to determine if a received message was considered Spam by a RBL host, by checking for the presence of the "X-RBL-Warning:" header, as shown in the example below.
 So in closing we hope this document achieved it's intended objective of showing how the default 7.1 FirstClass server Anti-Spam defenses work, and how all the facilities tie together to produce a powerful and very flexible method of reducing the Spam load on your email server and in the end, the amount of Spam arriving in your user account's mailbox.
Version:
Based on version 7.1 FirstClass Server.
| ||