From the Mind of Eric Lalonde

A Dumping Ground of Technical Information

Trapping Spammers With the OpenBSD Spam Deferral Daemon

One of the more satisfying aspects of OpenBSD’s spam management is how it stutters conversations with mail exchangers, taking up valuable resources of spammers. If every MTA took an advesarial approach to dealing with untrusted MX hosts, spam would be a lot less profitable.

First, the basics:

Greylisting is a method of defending e-mail users against spam. A mail transfer agent (MTA) using
greylisting will "temporarily reject" any email from a sender it does not recognize. If the
sender re-attempts mail delivery at a later time, the sender may be allowed to continue the
mail delivery conversation.

Spammers deal in volume. For this reason alone, greylisting filters out a lot of adversaries. It is a tactic employed by many MTAs. What separates spamd from others is that it stutters conversations as well. From the man page:

When a sending host talks to spamd, the reply will be stuttered. That is, the response will be
sent back a character at a time, slowly. For blacklisted hosts, the entire dialogue is
stuttered.  For greylisted hosts, the default is to stutter for the first 10 seconds of
dialogue only.

More than once, this twist on greylisting has caused my users to be removed from the recipient list of spam senders. Clearly someone is taking notice. Digging into the spamd code, we see why:

1
2
3
4
5
6
7
8
gethelo(cp->helo, sizeof cp->helo, cp->ibuf);
if (cp->helo[0] == '\0') {
  ...
} else {
  snprintf(cp->obuf, cp->osize,
  "250 Hello, spam sender. "
  "Pleased to be wasting your time.\r\n");
}

and also:

1
2
3
4
5
6
7
if (match(cp->ibuf, "MAIL")) {
  setlog(cp->mail, sizeof cp->mail, cp->ibuf);
  snprintf(cp->obuf, cp->osize,
  "250 You are about to try to deliver spam. "
  "Your time will be spent, for nothing.\r\n");
  ...
}

It’s clear that a few blacklisted spammers have noticed that conversations with my domains take a prohibitively long time, checked their logs, been properly insulted (I hope), and have given up trying to send garbage to my domains.

Spam Deferral Overview

I employ three components in the spam deferral system:

  1. spamd
    • tags new hosts as ‘grey’ in the spam database
    • wastes the time of blacklisted mail exchangers via stuttering
    • updates the packet filter so that whitelisted hosts speak directly to the MTA
  2. spamlogd
    • monitors the pflog(4) interface and detects when grey-listed hosts re-attempt mail delivery
    • promotes hosts from ‘grey’ to ‘white’ when the delivery retry threshold is met
  3. spamdb
    • allows administrators to view and modify entries in the spamd database
    • allows email addresses to be added to spam database for greytrapping consideration

Setup

Our first step is to configure the packet filter such that any MTA connection that is not already whitelisted is redirected to the spamd daemon. The following lines are added to /etc/pf.conf:

1
2
3
4
5
6
7
8
# rules for spamd(8)
table <spamd-white> persist
table <nospamd> persist file "/etc/mail/nospamd"
pass in on egress proto tcp from any to any port smtp \
    rdr-to 127.0.0.1 port spamd
    pass in on egress proto tcp from <nospamd> to any port smtp
    pass in log on egress proto tcp from <spamd-white> to any port smtp
    pass out log on egress proto tcp to any port smtp

The above configuration achieves the following:

  1. The nospamd table contains a list of addresses in CIDR netmask notation. These are addresses of mail exchangers belonging to well-known companies (google, AOL, etc.). I have chosen to exempt these mail exchangers from the greylisting process because these organizations employ a pool of mail exchangers; it is likely that the host which first attempts mail delivery will be different form the host which retries later.
  2. The spamd-white table contains the list of addresses which have been whitelisted by the greylisting process. That is, after an initial temporary rejection, these hosts properly retried mail delivery within a reasonable window, and thus are permitted to speak with the local MTA. This table is maintained by spamd.
  3. Any MTA that my domain contacts in order to deliver outgoing mail is automatically exempt from greylisting. This ensures that any replies to mail sent by my users will be delivered immediately.

Now we must configure spamd and spamlogd to start on boot:

1
2
3
# cat /etc/rc.conf.local
spamd_flags=""
spamlogd_flags=""

I also elected to create a special log file to keep track of events:

1
2
3
4
5
# touch /var/log/spamd
# grep -C1 spamd /etc/syslog.conf
# spamd
!!spamd
daemon.*                          /var/log/spamd

man syslog.conf(5) to understand this configuration.

Greytrapping with spamdb

A grey trap is a bogus email address used to lure spammers into attempting mail delivery. Since the address is bogus, all hosts which attempt delivery to the spam trap address are automatically blacklisted. People tend to embed grey trap email addresses into their websites. Spammers inevitably scrape the html, looking for valid email addresses. Lured by the grey trap, they fall immediately into a blacklist.

Grey traps are added as follows:

1
# spamdb -a -t spamtrap@protoc.org

We can use the spamdb utility to see the current MTA state for each host that has attempted delivery, and grey traps in the system.

1
2
3
4
5
6
# spamdb
WHITE|102.149.202.131|||1419811716|1419814026|1422944374|8|5
WHITE|209.85.192.170|||1419834014|1419834014|1422944414|1|0
SPAMTRAP|posta@protoc.org
SPAMTRAP|pandasticker@protoc.org
SPAMTRAP|spamtrap@protoc.org

In the above output, two hosts are whitelisted. The first entry was whitelisted after properly re-attempting mail delivery. The second is whitelisted because it falls in the list of corporate conglomerate addresses in the nospamd table.

The final three entries are grey trap addresses. These entries inform spamd that if any MTA attempts delivery to any address in this list, they are to be blacklisted.

In /var/log/spamd we see the payoff:

1
2
3
4
spamd[30378]: Trapping 198.49.66.165 for tuple 198.49.66.165 vps.mysoloads.biz <betty@gotodeals.eu> <pandasticker@protoc.org>
spamd[30378]: new greytrap entry 198.49.66.165 from <betty@gotodeals.eu> to <pandasticker@crackhead.org>, helo vps.mysoloads.biz
spamd[20363]: 198.49.66.165: connected (1/1), lists: spamd-greytrap
spamd[20363]: 198.49.66.165: disconnected after 392 seconds. lists: spamd-greytrap

For 392 seconds the spammer conversed with the local MTA at a rate of one character per second. Eventually, the shitheel gave up. Grey listing and trapping are only the first line of defense against spammers. In future posts I’ll dig into what countermeasures are available once mail reaches the MTA.

First Thoughts on the New OpenBSD httpd Webserver

In OpenBSD 5.6, apache has been removed from base entirely, and the community has been advised that nginx will be removed from base in 5.7. In their place, a home-grown webserver, named simply httpd, is now available in the OpenBSD 5.6 base system. The man page provides a brief overview:

 HTTPD(8)                OpenBSD System Manager's Manual               HTTPD(8)     

 NAME     
      httpd - HTTP daemon     

 SYNOPSIS     
      httpd [-dnv] [-D macro=value] [-f config_file]     

 DESCRIPTION     
      The httpd daemon is a HTTP server with FastCGI and SSL support.

      The FastCGI implementation has optional socket support.  httpd can log to
      syslog or per-server files with several standard formats.

      httpd rereads its configuration file when it receives SIGHUP and reopens
      log files when it receives SIGUSR1.

OpenBSD’s httpd has some nice features that caught my attention:

  1. Proper privilege separation - after binding to the http port, the server runs as the lesser-privileged user named www.
  2. Use of chroot to limit the harm that an attacker can do should the server be compromised.
  3. Simple and straightforward configuration file format.

If you’re like me, and you need only to serve up static content, then migrating your website to httpd is a simple matter. The /etc/examples directory contains sample implementations of various services on an OpenBSD system. Not surprisingly, the file we’re interested in is /etc/examples/httpd.conf.

1
# cp /etc/examples/httpd.conf /etc

The file contains a number of examples. For my configuration, I kept only the example that demonstrated a name-based virtual server (In OpenBSD 5.6, this example starts on line 22). All other examples were removed. Next, I replaced ‘example.com’ with the appropriate domain name. The result is the following:

1
2
3
4
5
6
7
8
9
# cat /etc/httpd.conf
ext_addr="egress"

# A name-based "virtual" server on the same address
server "www.protoc.org" {
        listen on $ext_addr port 80

        root "/htdocs/www.protoc.org"
}

It’s important to remember that httpd runs in a chroot. If you don’t know what this means, it’s important to use your google skills to find out before continuing. As the man page states, the default target of the chroot is /var/www. So, in reality, /htdocs directory mentioned above on line 8, is in fact /var/www/htdocs.

Other httpd configuration settings are discussed further in the man page.

Dynamic Content Hosting

But what if you’re not like me, and you need to host both static and dynamic content? The good news for you is that httpd comes with FastCGI support. What is FastCGI?

 FastCGI is a binary protocol for interfacing interactive programs with a web server.
 FastCGI is a variation on the earlier Common Gateway Interface (CGI); FastCGI's main
 aim is to reduce the overhead associated with interfacing the web server and CGI programs,
 allowing a server to handle more web page requests at once.

In order to get your dynamic content loaded into httpd, you’ll need a FastCGI interface to your content. This need is fufilled by an service called slowcgi which speaks the FastCGI protocol.

The exact instructions for loading dynamic content into httpd is use-case dependent, obviously. Below I have outlined a few of the important steps that you can adapt to your situation. The examples below assume we want to allow httpd to invoke perl scripts to perform some action.

  • First thing’s first: Get your script working under your chroot environment.

Do this first. It is important to isolate issues related to your chroot before httpd is even in the picture. This approach makes it easy to isolate issues with your chroot setup from issues with your httpd configuration, or from issues with your httpd <-> slowcgi interaction. Here’s an example script run under chroot:

1
2
3
4
5
6
$ cat /var/www/bin/test.pl
#!/usr/bin/perl
print "Hello!"

# chroot -u www /var/www /bin/test.pl
Hello!

If the above command doesn’t work, then I know perl isn’t installed correctly under chroot. Installing a program (and dependencies) into a chroot is a topic unto itself, and certainly beyond the scope of this blog entry, but it usually starts with something like:

1
2
3
4
5
6
7
8
9
10
$ ldd /usr/bin/perl
/usr/bin/perl:
        Start            End              Type Open Ref GrpRef Name
        000019f2f3d00000 000019f2f4102000 exe  1    0   0      /usr/bin/perl
        000019f575663000 000019f575a75000 rlib 0    2   0      /usr/lib/libpthread.so.18.0
        000019f5ba126000 000019f5ba6a0000 rlib 0    1   0      /usr/lib/libperl.so.15.0
        000019f5dce8c000 000019f5dd2b4000 rlib 0    1   0      /usr/lib/libm.so.9.0
        000019f5852a6000 000019f5856b2000 rlib 0    1   0      /usr/lib/libutil.so.12.1
        000019f5a91bb000 000019f5a96a4000 rlib 0    1   0      /usr/lib/libc.so.77.0
        000019f50af00000 000019f50af00000 rtld 0    1   0      /usr/libexec/ld.so

Perl will expect all of those libraries to exist, and has no idea it is being invoked under a chroot. All of the above files need to be installed in to the chroot destination in order to satisfy perl’s dependencies. Knowing the full extent of dependencies of a chrooted application is often an exercise in trial and error. Google is your friend here.

  • Update httpd.conf

Now that we know the interpreter works, we can move on to updating httpd.conf with FastCGI support:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ cat /etc/httpd.conf
ext_addr="egress"

server "default" {
       listen on $ext_addr port 80

       location "/cgi-bin/*" {
               fastcgi

               # The /cgi-bin directory is outside of the document root
               root "/"
       }
}

The above configuration has been simplified to focus entirely on FastCGI configuration. The changes are straightforward given what we are trying to achieve.

  • Enable slowcgi server
1
2
# vi /etc/rc.d.conf.local
slowcgi=""

Add the above line to inform OpenBSD to start slowcgi at boot.

With slowcgi running, and the chroot correctly configured, httpd should be able to invoke the script. Loading the script in a webpage should generate the expected output.

Debugging

Keep in mind that both httpd and slowcgi can be run in the foreground, from the command line, with the -d option. This is particularly useful since httpd will log directly to the console. If you’re really stuck, open two terminals on your webserver: one running slowcgi, one running httpd, both in the foreground. Any issues concerning interactions of these two services should become obvious quickly.

Other Notes

This guide is meant only as an introduction to httpd. Deploying your content safely and securely is the responsibility of the adminstrator. Many important concepts were not covered, such as file permissions and common security mistakes people make when deploying applications inside a chroot. For those new to web hosting, an in-depth understanding of these issues may save you from some embarassing results.

Also note that httpd was only introduced to base in OpenBSD 5.6. The stated goal of the authors is to get people testing it now before widespread deployment. Hopefully this guide helps you do so.