Spam Filters Explained

Spam is a very real problem that manypeople have to deal with on a daily basis. For those that have decided to dosomething about it and start to investigate the options available in spamfiltering, this article provides a brief introduction to your options and thetypes of spam filters available.

Despite the bewildering array of spamfilters available today, all claiming to the best one “of its kind” there arereally just five filtering methodologies in general use today and all productsrely on one, or a combination of these:

Content-Based Filters

“In the beginning, there werecontent-based filters.”

These filters scan the contents of the andlook for tell-tale signs that the message is spam. In the early days ofspamming it was quite simple to look out for “Kill Words” such as
”Lose Weight” and mark a message as spam if it was found.

Very soon though, spammers got wise to thisand started resorting to all kinds of tricks to get their message past thefilters. The days of “obfuscation” had begun.
We started getting messages containing the phrase “L0se Welght” (Notice thezero for “o” and “l” for “i”) and even more bizarre – and sometimes quite ingenious– variations.

This rendered basic content-based filterssomewhat ineffective, although there are one or two on the market now that areclever enough to “see through” theses attempts and still provide good results.

Bayesian Based Filters

“The Reverend Bayes comes to the rescue”

Born in London 1702, the son of a minister, Thomas Bayesdeveloped a formula which allowed him to determine the probability of an eventoccurring based on the probabilities of two or more independent evidentiaryevents.

Bayesian filters “learn” fromstudying known good and bad messages. Each message is split into single “wordbytes”, or tokens and these tokens are placed into a database along with howoften they are found in each kind of message.

When a new message arrives to be tested by the filter, the new message is also split into tokens and each token is looked up in the database. Extrapolating results from the database and applying a form of the good reverend’s formula, know as the “Naive Bayesian” formula, the message is given a “spamicity” rating and can be dealt with accordingly.

Bayesian filters typically arecapable of achieving very good accuracy rates (>97% is not uncommon), andrequire very little on-going maintenance.

Whitelist/Blacklist Filters

“Who goes there, friend or foe?”

This very basic form of filtering is seldomused on its own nowadays, but can be useful as part of a larger filteringstrategy.

A “whitelist” is nothing more than a listof e-mail addresses from which you wish to accept communications. A whitelistfilter would only accept messages from these people and all others would berejected

A “blacklist”, conversely, is a list ofe-mail addresses - and sometimes IP Addresses (computer identificationaddresses) - from which communications will not be accepted.

While this may seem like a good idea fromthe outset, a whitelist methodology is too restrictive for most people and, asvirtually all spam e-mails carry a forged “from” address, there is little pointin collecting this address to ban it in future as it is very unlikely to be thesame next time.

There are bodies on the internet thatmaintain a list of known “bad” sources of e-mail. Many filters today have theability to query these servers to see if the message they are looking at comes froma source identified by this Internet-based blacklist, or RBL. While being quiteeffective, they do tend to suffer from “false positives” where good messagesare incorrectly identified as spam. This happens often with newsletters.

Challenge/Response Filters

“Open sesame!”

Challenge/Response filters arecharacterised by their ability to automatically send a response to a previouslyunknown sender asking them to take some further action before their messagewill be delivered. This is often referred to as a "Turing Test" -named after a test devised by British mathematician Alan Turing to determine ifmachines could “think”.

Recent years have seen the appearance ofsome internet services which automatically perform this Challenge/Responsefunction for the user and require the sender of an e-mail to visit their website to facilitate the receipt of their message.

Critics of this system claim it to be toodrastic a measure and that it sends a message that "my time is moreimportant than yours" to the people trying to communicate with you.

For some low traffic e-mail users though,this system alone may be a perfectly acceptable method of completelyeliminating spam from their inbox - one step above the "Whitelist"system outlined above.

Community Filters

“A united front”

These types of filters work on theprincipal of "communal knowledge" of spam. When a user receives aspam message, they simply mark it as such in their filter. This information issent to a central server where a “fingerprint” of the message is stored.

After enough people have “voted” thismessage to be spam, then it is stopped from reaching all the other people inthe community.

This type of filtering can prove to bequite effective, although it stands to reason that it can never be 100%effective as a few people have to receive the spam for it to be “flagged” inthe first place. Just like its similar cousin the Internet black list (RBL), thissystem also can suffer from “false positives”, or messages incorrectlyidentified as spam.

Hopefully you are now armed with a little moreinformation to be able to make an informed decision on the best spam filter foryou.

For further information, consider reading thereviews and articles found at http://www.whichspamfilter.com