How Spam and Phishing Filters Work

Shortly after the internet was invented, the first spam email was sent by Nac to Krag, who was checking his email in the cave next door, encouraging him to deposit a substantial portion of his cache of woolly mammoth steaks in a drop box located (mysteriously) outside the entrance to Nac’s cave.

Okay, that bit of spam trivia is not true, but this is: Current estimates are that spam constitutes 45% of ALL emails sent. That works out to about 14.5 billion pieces of pointless drivel related to scams and frauds of all types jamming up the internet every single day.

In response, automated email filters – like the AI-driven email spam detection introduced by Gmail in 2015 – have moved to the front lines of the war on junk email.

Around 150-300 billion emails are sent over the internet each day, and roughly 85% of those are immediately marked as spam…

The problem for legitimate online entrepreneurs and website owners who communicate with customers by email is how to avoid getting messages zapped by filters or, worse, find themselves blacklisted from sending emails at all.

The best way for email marketers to avoid these filters is to understand how they work, and there’s no time like the present. First, let’s get clear on what these terms – spam and phishing – actually mean.

What is email spam and phishing?

While you’ve probably never been more excited to learn about email spam than you are now, I’d be remiss to plow forward in our discussion of spam and phishing filters without a quick review of what spam and phishing actually are, just so we’re all starting from the same spot.

Email spam

More than a peculiar lunch meat that comes in a can, spam is the term used to describe all those emails you receive and never wanted. To be precise, spam email is when the same message (likely inappropriate and/or irrelevant) is sent to hundreds, thousands, or even millions of people for the sole purpose of enriching the sender at the expense of the recipient.

A perennial favorite is a financial scheme floated by an evasive Nigerian Prince (or government official, astronaut lost in space, or any of a dozen different personas) that promises to net you millions for little effort. Here is a copy of one such email:

Source: Ubergizmo

Without wasting time on the details, just be aware that if you comply with the requests of one Dr. Bakare Tunde, cousin of Nigerian Astronaut, Air Force Major Abacha Tunde, you will never, ever see a penny… but will stand to lose quite a lot (upwards of $3,000,000 – the value of his missing trust inheritance).

Take, for example, Kansas handyman Fred Haines, who lost $110,000 between 2005 and 2008 to a Nigerian inheritance scam. For a self-employed individual like Mr. Haynes, losing $110,000 is devastating – yet, it’s a drop in the bucket compared to the total amount of lost money due to scams. Customers in the UK lost nearly $650 million to bank scams in just the first half of 2018 alone. The scariest part about these scam statistics is that they are increasing dramatically each year: a report from the FBI revealed that email scams increased by 2,370% between January 2015 and December 2016.

The key here is that the spam is sent out mostly indiscriminately. In 2017, the Department of Homeland Security reported to CNBC that email scammers are increasingly targeting the elderly. Yet, there is no qualified list of recipients or strategy used to determine who will receive the message. It’s all about numbers: send out to as many people as possible and hope that enough will respond to make the effort worth it.

Email phishing

This type of scheme refers to the practice of sending mass emails that purport to be from reliable companies in order to induce you to give up information like bank accounts, credit card numbers, passwords, etc. One of the more memorable email phishing scams of 2018 centered around the Moscow World Cup. Last summer, scammers phishing emails to soccer fans that included enticing, but totally fake, free trips to Moscow.

A variation on phishing is known as spear phishing, in which a single individual in an organization is targeted and the email tailored to sucker them into doing something stupid.

Why do spammers and phishers do it? Easy. It pays well, beats getting a real job, and there is an inexhaustible supply of gullible people online itching to get ripped off.

When you get a message that seems a bit fishy, do a bit of research before opening it and clicking any links in the email. For instance, if you get an email from someone you’ve never met, asking for personal information, put the sender’s address in spam databases like Spamhaus.org or DNSStuff.com.

You can also check the reputation of the sender’s email address with SenderScore.org. If the report isn’t 100% positive, don’t reply and certainly don’t click any links in the email.

What are email service provider spam filters (and why do they matter?)

The first category of email spam filters (when we say “filter” think “software”) are those created and installed by an ESP like Gmail or Outlook on their servers. Think of these as your “first line of defense” against spam and phishing.

These filters are essentially algorithmic software — now powered by AI — which have one mission in life and that’s to “read” and analyze all incoming and outgoing emails, according to a set of rules, and decide if it is spam or not.

To be specific, the algorithms look for:

• Questionable content (related to adult material, pharmaceuticals, etc.)
• Trigger words or phrases (like ‘free’ or ‘bully bully’). Here’s a list of 455+ trigger words.
• Known viruses or malware, in emails both to and from you.
• Bulk mailing tactics, such as sending many emails at the same time, using proxies to hide the sender IP address, or cloaking the domain of the sender.
• Email domains from known spammers.
• IP addresses that have never been used to send emails or that have been used to send questionable senders.
• Engagement with emails (opens and clicks), especially on large list of tens of thousands sent daily.
• Individual engagement (at least Gmail does it)
• Number of invalid email addresses (bounce rate)
• Emails sent to an old address that has been converted to a honeypot (Gmail Yahoo, Outlook)
• SPF & DKIM signatures
• DMARC, which is a policy in case SPF or DKIM aren’t valid. For example, if Yahoo receives an email from will@gmail.com but the email was sent from a non-Gmail server, it will simply not deliver the email. This is becoming more and more common.

At the end of the analysis, every email is assigned a spam score that reflects how likely it is to be spam. If the filter is suspicious but not entirely sure, the message might end up in the recipient’s spam folder for later review. If the filter is pretty darn sure of the spamminess, it won’t be delivered at all.

Since most email marketers and website owners prefer to have their messages actually delivered, it’s worth the time involved to learn how to avoid giving ESP spam filters anything to complain about.

Commercial spam filters

Considering the scope of the spam/phishing email problem (remember the number 14.5 billion pieces of junk email DAILY), it should be no surprise that businesses and sometimes even individuals install commercial filters on their networks or devices to catch unwanted emails that are missed by the ESP server filters.

While an ESP filter is a good first step, the reality is that a business will rapidly find its email system unusable without a secondary filter in place. And since sending spam is a multi-billion dollar industry — costing individuals and businesses more than $12 billion between 2013 and 2018 — it should be no surprise that the commercial spam filter industry has become a titan in its own right. Examples of well-known commercial products are SpamHaus, SpamCop, Cloudmark, and VadeSecure, to name a few.

Commercial filters work generally the same as ESP filters. A company should select their particular software depending on their email use.

For example, a content filter would be a good idea if employees receive a lot of newsletters and articles. If security is a serious concern, opt for a permission-based service (requires an outside sender to be vetted by the intended recipient).

Common email spam triggers

You might not be surprised to learn that the exact process by which any spam or phishing filter evaluates emails is a proprietary trade secret. They don’t want every Tom, Dick, and Jane spammer to know what the algorithm is looking for – or they’ll just tweak their output to beat the system.

Still, experts acknowledge the following categories and importance as industry truisms when it comes to determining whether or not an email message is spam.

Reputation

This is a HUGE factor. Spam filters look at the number of outstanding complaints against a sender, how old the domain is and how long it has been in use and the reputation of the IP addresses from the sending server.

Since spammers tend to get blacklisted a lot, they’re constantly buying and setting up new domains or using new IP addresses that have never been used to send emails before. Keep in mind that including links in emails you send puts you at the mercy of the reputation of the web host on whose servers the domain is hosted. Avoid cheap hosting sites like these, which regularly put thousands of domains on the same shared servers.

For example, if you send out a travel newsletter with a link to a rental car company offer, better check that their domain has a good rep because you could suffer the consequences by association if not.

Engagement

How do you typically respond to spam? Chances are you ignore it, delete it, move it to a junk folder, or report it. Filters know this and punish the senders of messages that get treated in this manner.

Gmail was the first to introduce engagement as a factor to determine if the recipient should receive the email or not, or in which tab it should be classified. This behavior is user-specific and not applied across all accounts. Other Email Service Providers (ESP) have followed since.

Positive engagement, measured by a message being viewed or clicked on, helps the filters decide that it was useful and the sender is a good egg. Properly segmenting your list is almost guaranteed to increase email engagement, according to a recent study by Mailchimp.

Authentication

Filters look for evidence that an email sender is actually authorized to send emails using the domain the message is associated with.

While we could spend a week explaining the intricacies of how this is verified (check out this article on DomainKeys Identified Email if you’re interested and have the time), suffice it to say for our purposes that a common hacker tactic is to misuse domains that don’t belong to them. If the spam filter can identify that you’re an authorized user, bully for you.

Formatting

Pay attention to how you format your emails because it matters to spam filters. Large fonts, gaudy colors, and too high of an image to text ratio are red flags. Using a single large image is a no-no. Another sin is to copy and paste from a Word document into the email. The problem is that Bill Gates’ pride and joy word processing program introduces a lot of junk code that is incompatible with most email programs.

Word triggers

If you use the wrong words in your email copy, expect a quick trip to the spam folder. The usual suspects would include “free,” “low interest loans,” “cheap medicines,” “best casino payouts,” and many, many more. There are lists online of words to avoid. Here is one. Keep in mind these aren’t set in stone and are just guesses based on observations. Once again, I don’t know the exact algorithm for these filters.

The blacklist is not the end

It happens. You got on the wrong side of a spam filter and ended up on the proverbial blacklist (you can use MailPoet’s mail testing tool called Mail-Tester to check if you’ve been blacklisted). While it’s a hassle, it doesn’t mean your days of sending email are over and online entrepreneurial dreams shot to heck.

For a step-by-step guide to getting de-blacklisted, this guide is a good start or contact your email sending provider. From our own experience at MailPoet, this is a job in itself. You first need to know where you’re listed (Mail-Tester’s list if not exhaustive and excludes all the major ESPs) and you’ll need to write a reassuring message to get delisted.

For our purposes, we’d like to focus on one preventative measure. Don’t send emails unless every member on your list of recipients has agreed to receive them. It’s that simple. Spam is defined (partly) as contact a receiver didn’t agree to. Get them on record as having agreed and you should have no worries if you get reported to a blacklist.

This is why the mailing list you’re growing should require a double-opt-in mechanism to prove that every listing is okay to receive email from you.   

Phishing filters: A fish of a different flavor

Filters that attempt to block phishing attempts work a little differently than those intended to detect straightforward spam.

Since the lifespan of a phishing website is measured in days – typically one or two at most – filtering them relies on maintaining a real-time database that your browser checks when you head online. It’s important to report phishing attempts because it improves the quality of the database.

A common phishing method for stealing a customer’s credit card data. Source: Tripwire

More reports equals better information and fewer successful phishes. With such a limited lifespan, blacklisting a phishing website is sort of pointless. Instead, these types of filters rely on methods like comparing a URL you visit to legitimate ones, such as your bank or online uber-seller Amazon.

While you might be in a hurry and not notice subtle differences that would indicate a phishing website, your trusty phishing filter notices and will pop up a warning that you should depart the website without clicking anything.

Virtual private networks – the miracle spam solution? Not really

While there are plenty of reasons for anyone who goes online to protect their privacy and anonymity with a VPN, this service won’t help much in the battle against spam.

A VPN encrypts your data and runs it through a separate server than your ISP. This makes it difficult for hackers to locate critical financial information like your credit card number whilst making an online payment.

As we’ve discussed, spam arrives via an entirely different process. Despite what some VPN services might claim, simply encrypting your internet connection won’t save you viruses and malware embedded in spam emails. You need the proper filters in place, like we’ve discussed, in order to block that the messages entirely.

The bottom line

As we’ve explored in this guide, email and spam is an ongoing problem for nearly everyone who uses email today. And it’s becoming more of a problem as scammers find new ways to defraud individuals and separate them from their money.

For legitimate business owners and email marketers (that’s you… right?), this presents a bit of a problem. When sending emails en masse, triggering the wrath of ESPs or commercial spam filters is going to happen. So be prepared to quickly course correct when it does so that you can continue using email to effectively market your products or services.

The best way to avoid spam filters? Only send emails via reputable email sending services. The folks who know the business of email inside and out, and who have deliverability experts on hand to help you resolve your deliverability issues, which will inevitably come up trumps.

Further, choose white-hat tactics. These are the only ones that never go out of fashion. Read a few good blogs that let you keep your finger on the pulse of anything that changes in hat spam filters look for.

Other than that, make it a habit to only send out useful emails to people who have agreed to receive them and you will have no problems.