.
Email is not a messaging protocol. It’s a TODO list. Or at least my inbox is a TODO list and email is the protocol for putting stuff on it. Here’s the problem. It is a s**tty TODO list. Any one of you can put something on my TODO list. And I don’t want that.
I’m open to different solutions to this problem but I suspect tweaking the inbox is not going to be enough. You’re going to have to make a new protocol. It can degrade to the old protocol but I think you should make a TODO list protocol instead of a messaging protocol. As a messaging protocol it ought to give more power to the recipient. I ought to be able to control who can put things in my inbox. When someone can’t put things in my inbox my server should just say “Sorry not accepting any more”
– Paul Graham, PyCon 2012
Microsoft’s plan to end spam:
Spammers send millions of messages every day to be profitable because response rates are so low, so their computers spend only a small fraction of a second processing each message. In a spammer’s economic model, spending even five or 10 seconds per message could be prohibitively expensive. Smaller organizations, however, that send low volumes of e-mail generally have an abundance of computer processing power available. They can afford to spend a few seconds on each message.
I had an email problem. My inbox was always full of emails. I was very slow in responding to personal emails. How slow? Ask my friends. It wasn’t unusual for them to receive an email response from me over a year after they sent the email as I played catchup.
I get too much email. Most of us do. The situation was out of control, and had been that way for years. I remember the good old days in the 90’s when, for the first few years of my email experience, almost all emails were personal, and a joy to engage in. I wanted that experience back. I wanted receiving someone’s email to lighten up my day. I didn’t want it to be surrounded by non-personal junk email.
Below is a list of failed attempts to bring my email under control: [1]
- Manual processing. Too much mental pain to classify each email as spam or not.
- Spamassassin. This worked for many years, but then a few years ago, the accuracy went down like crazy. I gave up trying to debug why. And even a perfect spam filter would not eliminate emails from Target, etc.
- Every 3-6 months, go through my emails, find the unsubscribe link, and hope it works. It often didn’t. And sooner or later I’d be back on some organization’s list because I needed something from them and they wanted my address.
- Use a separate forwarding email address whenever I signed up for something, and have a rule to filter those emails to a separate folder. Invariably stuff would still find its way into my inbox.
- Set up a blacklist. Useless against spam.
- Get a new email address and start over. This was off the table. I’m partial to my email address, and paid money for my own domain. I will not let spammers and marketeers win!
- Inbox Zero, the GTD way. Not a bad solution, but was still too large an overhead.
It wasn’t just that these solutions weren’t good enough. It was that I kept spending time tending to my inbox. And the end result was I still found myself replying to people months later.
Paul Graham had it right. If you think about it, it’s pretty lousy that anyone in the whole world can trivially put items into my inbox, and with almost no effort.
So I came up with a solution. Instead of me spending time, make the sender work to get my attention. Make them earn the privilege of getting emails into my inbox.
Make ‘Em Work!
The solution is ridiculously simple. Forget clever ways to filter emails. Don’t use a blacklist. Instead, use a whitelist, and make the sender work to get into the whitelist.
When someone new sends me an email, my mail filter automatically sends their email into quarantine, and sends them this email:
Hello,
You recently tried sending an email to myemail@domain.com with subject:
Discount! Ends Soon!
His spam checker is unable to determine if your email is spam, and has quarantined your email. To ensure he receives your email, please go to the following site:
http://mysite.com/slkjed399saljdla
and follow the instructions. Hopefully, once you do this your email address will be whitelisted and your future emails will pass through unfettered.
Thanks,
Your polite spam checker.
When they go to the website, they see this:
Greetings!
You recently tried to send an email to XXXX, but his spam filter could not determine if your email was spam. Your email has been stored in a holding cell. To ensure your email is liberated and added to his inbox, please enter your email address in the field below and click Submit. Hopefully, you will need to do this only once.
Email address:
Once they’ve confirmed it, my system takes all quarantined emails from them and delivers them into my inbox. What’s more, they are now in the whitelist. Future emails from them will go straight to my inbox.
A simple one-time pain for the sender is all it took to kill all non-personal emails (spam, corporate emails, etc). My inbox only has personal emails in it. I’ve used this system for over a year. It’s rare that a non-personal email ends up in my inbox. I can’t emphasize how nice that is.
Some Technical Details
I’ll give a big picture view of how I set it up. For most readers, my specific implementation will not work for you, and you’ll need to adapt to your setup. It works a lot easier if you download all your emails and use some software to interact with emails (as opposed to a web interface like Gmail).
I use notmuch for my email. It’s similar to Gmail - you don’t have folders, you have tags (what Gmail calls labels). Emails in my inbox merely have the inbox tag applied to them. Emails in quarantine have the quarantine tag, and not the inbox tag.
The nice thing about notmuch is that it has Python bindings, and you can add a Python script as a hook every time notmuch checks for new emails. My Python script is simple: For every new email, check if the sender is in the whitelist. If not, send them an email pointing them to a web site and quarantine the email.
I’ve configured keybindings in the notmuch interface for the following:
- Quarantine this person (i.e. -inbox +quarantine). It also adds the person to the whitelist.
- De-quarantine the person: This will do the opposite of the above.
That’s pretty much it.
On the web server, I have a simple Flask app running. The URL is unique to the sender. When they enter their email address, a sanity check is done against the URL. If it passes, it sends an email to me. My notmuch hook script scans for these emails, and does the dequarantining procedure.
The whole thing probably took a few hours to write.
Anticipated Queries
Here I address concerns some may have:
Do people bother filling out that form?
Yes, surprisingly. Even more surprisingly, people behind some of the companys’ mass emails occasionally take the time to fill it out. As an example, I get regular emails about crimes logged in my neighborhood. They are annoying. Yet every few months someone from that service goes through the hoops to get into my inbox - only to have me quarantine them again.
I think the only humans who didn’t fill it out were job recruiters. There’s a bit of a disconnect when they say they’ve looked at my profile and think I’d be a great candidate for their company, but they don’t seem to value me enough to spend a few seconds dealing with my filter.
Is your workflow really that simple?
No - the above is a bit of a simplification. I actually have three tiers:
Inbox: These are reserved for personal emails only. If an email lands here, someone intentionally wrote it for me. My goal is that this is the primary folder I’ll look at daily.
Low Priority: These are emails I’d like to keep abreast of (e.g. community organizations I’m a member of, etc). These are usually not personal emails, but they are of some value. They are in a separate whitelist.
Ideally, I would like to process these once a week.
Quarantine: Most of these emails can go to Hell. There are a few that are of occasional use to me (e.g. certain emails with certain deals), so I will examine the quarantined items once in a while.
Additionally, there are some email addresses I want to keep in the quarantine area, but I value the people/organization sending them and don’t want to annoy them with my emails. So I have another list for quarantined people who do not get annoying emails.
Don’t let the complexity fool you. The basic idea is still simple. And the beauty of it is that it is easy to extend to custom flows as I have done.
Aren’t you concerned about missing out on important emails?
No - I check the quarantine folder regularly - usually daily. I’ll catch important emails that may have slipped through (and if need be, dequarantine them).
Of course, as time goes by, I rarely need to dequarantine people - they’ve already made their way into my whitelist.
And finally, I’ll be frank. A clean inbox is worth missing some emails. Think of the prevailing situation: Where mildly important emails are not acted on because they’re buried under all that spam.
Doesn’t checking the quarantine folder so often defeat the purpose?
Before I began, I would have thought that as well. However, simply having separate inbox and quarantine folders makes a world of a difference.
And when I’m not in the mood, I just don’t check it that often. I recently went almost a week without looking at it.
How long do you keep quarantined emails?
My plan was to have a script that deletes anything older than 6 months. I’ve been too lazy to write that script, so at the moment I still have all of them.
6 months should be plenty of time to go find any important emails you may have missed. Chances are if it’s older than that, it’s not useful any more.
I can’t believe all your friends went through the hassle of getting on to your whitelist
I didn’t believe they would either, so I didn’t make them go through it. Before my system went into production, I wrote a script that went through all my emails in the last 20 years and put every person’s email address into the whitelist. [2]
Yes, that includes all the corporate emails and the spam emails that were still in my mail folders. Once I went into production, anytime some email showed up in my Inbox that shouldn’t, I hit a keystroke to remove them from my whitelist. It was very little work.
Why not just use Google Inbox’s bundles?
The Web has gone full circle. When I started using the web, I signed up with a free email service provider that had free email forwarding. At some point, they asked for money and I was a poor student. So I signed up for another one. Same story a year or two later. Same story with hosting online photos. Same story with hosting web sites. I’d settle down somewhere, only to be uprooted. Same story with Internet telephony.
Finally, in 2004, I gave up on free services, and started paying for them. I got my own domain as well. I hosted my own image gallery. I vowed never to be lured by free Internet offers. And I did not invest heavily in systems that didn’t let me download my content offline.
Then came Gmail, and Flickr, and so many other services. All my friends assured me they were there to stay. I mean, can you imagine Gmail being discontinued?
As far too many people have noticed, Google has been killing a lot of their products of late. Far too many people I know say they don’t want to rely on any Google service for their business because they now have a solid track record of killing their own products. Flickr was decaying under Yahoo, and have now been bought again. Let’s see where they go.
As long as there is IMAP and SMTP, and as long as the world continues to use the usual email protocols, my solution will continue to work.
Any cases where this solution doesn’t work?
Yes. It won’t work if someone is trying to use a service to send you an email, and that service uses a random email address each time. The biggest offender is Craigslist. Every time someone responds using Craigslist, the email address is different.
Look what happened to this poor guy who wanted to buy my car:
First email:
Hey there!
I just saw your ad for the Galant! I am very interested in it. We tried to go look at one like it yesterday but the guy stood us up!
Anyway feel free to call me or mail. We’d b able to come look at it this evening probably.
Thank you so much.
3 days later:
Hey there,
I have tried emailing you before but I keep getting these emails back saying it’s going to your spam folder. Hopefully this one gets through.
I am super interested in the Galant! I would like to come take a look at it in the next couple days.
Feel free to call or text me!
5 hours later:
QUIT FUCKING EMAILING ME
> Hello,
> You recently tried sending an email to myemail@domain.com with subject:
> Mitsubishi Galant?
> His spam checker is unable to determine if your email is spam, and has quarantined your email. To ensure he receives your email, please go to the following site:
> http://mysite.com/slkjed399saljdla
> and follow the instructions. Hopefully, once you do this your email address will be whitelisted and your future emails will pass through unfettered.
> Thanks,
> Your polite spam checker.
I bet I can find a way to circumvent your filter!
I bet you can. I also bet it’s not worth your time, so I do not need to worry about this.
Postscript
So was I able to return to the magical days of early email? Not quite. As much as I like the medium, the rest of the world no longer has the patience to write out well thought out emails. Just yesterday I had a few back and forth emails (initiated by him) with an old friend I had not talked to for months. In 3 emails, he wrote a sum total of 11 words.
Nevertheless, my mental state is still much better with this solution. And I feel totally free to give out my email address and sign up for crap on the Web.
Advertising companies (Google, Facebook, etc) are earning a huge amount of money by monetizing your attention. Your time and attention are valuable. If someone wants it, make them do at least the minimum of effort. No one should be able to pile stuff up into your email account.
People laughed at Microsoft, but they were on to something.
Update: This post hit HN. Read my followup post.
| [1] | I suppose I should point out that I don’t use a major provider like Gmail. I pay for mail hosting, and download emails to my PC. As such, I didn’t use my provider’s spam filtering, and if you read this whole post, you’ll see why spam filtering is not a solution. |
| [2] | This is trivial with the Python standard library. |