You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

396 lines
14 KiB

<h1>E-Mail, Advanced Topics</h1>
<p>This tutorial discusses advanced topics concerning e-mail on
SDF: spam filtering, automatic processing, forwarding. It is probably
only useful for ARPA members.</p>
<li><a href="#intro">Introduction, and what this is <em>not</em></a></li>
<li><a href="#forward">Forwarding E-Mail</a></li>
<li><a href="#fetch">Fetching E-Mail</a></li>
<li><a href="#spam">Fighting Spam</a></li>
<li><a href="#whitelist">Personal White List Against Spam</a></li>
<li><a href="#dnsbl">Using Procmail With a DNSBL</a></li>
<li><a href="#ssh-smtp">Use SDF SMTP Server Remotely via SSH</a></li>
<li><a href="#gloss">Glossary</a> - explications of terms</li>
<hr />
<h2><a id="intro" />Introduction, and what you <em>will not</em> find here</h2>
<p>Here we discuss advanced topics for e-mail processing on SDF, mainly
automatic processing of incoming messages with <code>procmail</code>, and
forwarding issues. As <code>procmail</code> is only available to ARPA
members, the contents of this tutorial will be mostly useless to user
<br />
For basic info about e-mail (addresses, reader programs, file size, etc),
please read the corresponding
<a href="">FAQ</a> entry - and if
you still have problems with basic sending and receiving of messages
via SDF, this tutorial is most probably not for you!
<h2><a id="forward" />Forwarding E-Mail</h2>
<p>The <code>sendmail</code> system allows for automatic forwarding of
incoming mail to other addresses or even programs (filters).<br />
The file <code>$HOME/.forward</code> may contain the following types of
<dd>external e-mail address, where a copy of each mail will be sent</dd>
<dd>will pipe the incoming messages through <code>/bin/sh</code> with the
argument <code>filterprogram</code></dd>
<p>However, if you want to do more than forwarding to one address,
use of <code>procmail</code> is highly recommended! In fact, it is
best to set up <a href="#nospam">nospam -e</a>, which will generate
a .forward file with a line calling procmail, and then to modify
the resulting <code>.procmailrc</code>.
<h2><a id="fetch" />Fetching E-Mail</h2>
<p>Use <code>fetchmail</code> to collect messages from accounts
you might have elsewhere. The messages are given to <code>procmail</code>
(see <a href="#forward">Forwarding E-Mail</a> above) for further
<h3><a id="fetch-ex" />Sample <code>.fetchmailrc</code></h3>
proto pop3
user "john.doe"
pass "secret"
is "jdoe" here
mda "/path/to/procmail -f- ~/.procmailrc"
<h2><a id="spam" />Fighting Spam</h2>
<p>To fight unwanted bulk e-mail (spam), there are several possibilities
available on SDF:
<li>As a general remark, think before you publish or tell somebody your
e-mail address! And try to make clear to people that you don't want your
address on a visible list of 500 recipients from somebody's address book; one
should use Bcc: for sending to a lot of people in parallel, not To:!</li>
<li>SDF also supports plus-addressing;
you can use this to join mailing lists and do filters.
This can be used by adding +something before the @ in your email address.</li>
<li>The <a href="">FAQ</a>
has more information about <a href="#nospam">nospam</a>, available on SDF.</li>
<li>The automatic mail processor <code>procmail</code> (see <code>man
procmail</code> for more information)
allows to filter incoming mail following user defined rules; in particular,
you can generate a <a href="#whitelist">white list</a> of sender addresses
you know as good, and put messages from other addresses into a special
folder which you browse through now and then, to check for good messages
and adding the corresponding addresses to your white list.</li>
<h3><a id="nospam" />nospam</h3>
<p><code>nospam</code> is a script which can set up .forward and .procmailrc
files so that certain e-mails will be stored in &quot;bulk folders&quot;.
<code>nospam -i</code> gives some additional information, and <code>nospam
-e</code> will enable this filtering. We will suppose that this has been done
also for the following discussion of <a href="#whitelist">white
<h2><a id="whitelist" />Personal White List Against Spam</h2>
<p><em>White lists</em> contain e-mail addresses which are always
to be accepted as good. It is easy to implement this with
<code>procmail</code>: Supposing that <code>nospam -e</code> has been
executed, so that basic procmail processing is set up, one can add the
following fragment to <code>.procmailrc</code>.</p>
<p><em>Note:</em> For this to be working, the following <em>must</em> have
been set up:
<li>a <code>.forward</code> file which pipes incoming e-mail through
<li>a standard <code>.procmailrc</code> file, where this fragment is attached
or inserted</li>
The easiest method to install these files is by launching
<code>nospam -e</code> as mentioned <a href="#nospam">above</a>, then adding
the fragment to the generated <code>.procmailrc</code>.
<br />
(Of course, it is possible to deactivate or delete all the initial
procmailrc lines generated by nospam, which move messages with
attachments to the bulk folder, if such messages should pass through
the whitelist filter.)
# procmail spamfilter using whitelist (2006 Yargo Bonetti)
# :: use at your own risk and any way you want! ::
# whitelist: file containing one valid e-mail address per line
# (only generic xxx@yyy.zz form, without "Name.." <*> parts)
# spamfilter
FROM=`formail -c -x 'From:'|sed -e 's/.*<\(.*@[^>]*\)>.*/\1/'`
:0 hb:
* ! ? grep -i -F -e "$FROM" $WHITELIST >/dev/null
<p>This will compare the address in the From: field of incoming
messages to the ones saved in the file $MAILDIR/.whitelist (of
course, name and location can be changed), and when an address is
<em>not found</em> in this file, the message will be saved in the
folder $MAILDIR/quarantine (which can be modified as well) and not show up in
the normal Inbox (and not be processed further). Now and then, one can look
through the quarantine folder for &quot;good messages&quot;, and add the
corresponding addresses to the .whitelist file.</p>
<p>If you want to use the mail directories proposed by nospam, you may
prepend the above noted fragment by</p>
<p>which will result in storing the &quot;possibly bad&quot; messages in
<code>~/mail/quarantine</code> and expect the whitelist in
<p>A sample .whitelist may look like this:</p>
<p>All messages not coming from these three addresses would end up in the
&quot;quarantine&quot; folder defined in the .procmailrc fragment.</p>
<h2><a id="dnsbl" />Using Procmail With a DNSBL</h2>
<p>DNSBL stands for "DNS blacklist", and it is essentially a collection of IP
addresses known to send out spam and other bad stuff. Two popular DNSBLs are
<a href="">SpamCop</a> and
<a href="">Spamhaus</a>. Typically the mail server
will check the source of incoming mail with the DNSBL before even accepting
the message. SDF does not do this, but using Procmail to check a DNSBL at
the user level is easy. In this tutorial we will use Spamhaus.</p>
<p>(Thank you <a href="">Benya</a> for
the original instructions.)</p>
<p>First make sure that <code>~/.forward</code> is set-up to forward incoming
mail to Procmail. Then we'll edit <code>~/.procmailrc</code>. The beginning
of this file should declare basic variables such as <code>SHELL=/bin/sh</code>,
<code>LOGFILE=$HOME/.procmaillog</code>, <code>ORGMAIL</code>,
<code>DEFAULT</code>, and <code>MAILDIR</code>. <code>ORGMAIL</code> and
<code>DEFAULT</code> should point to your inbox, and <code>MAILDIR</code>
should point to the directory that contains your saved mail.</p>
<p>Next Procmail should extract the IP address from which the message was
sent. This is done using <code>formail</code> to get the headers,
<code>grep</code> to find the correct line, and <code>sed</code> to find the
actual IP address, which is then saved as <code>SENDERIP</code>.</p>
<pre>SENDERIP = `formail -c -XReceived | grep "by" | \
grep -v "from" | \
sed "s/^Received: from .*\[\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\].*by*$/\1/"`</pre>
<p>Then we'll begin the actual Procmail recipe, which will only be executed
after checking to make sure that the <code>SENDERIP</code> variable exists
and is in the correct format. Anything written to the <code>LOG</code> variable
will be inserted into the log file. (New lines must be explicitly stated.)</p>
* SENDERIP ?? ^^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*^^
LOG = "The sender's IP address is "
LOG = "
<p>The next step is to reverse the IP address because Spamhaus wants it that
way. Procmail will then use <code>host</code> to determine the IP address that
[reversed IP address] resolves to.</p>
<pre> SENDER_REVERSED = `expr "$SENDERIP" | \
sed "s/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/"`
sed "s/^.*\(127\.0\.0\.[0-9]*\)$/\1/"`</pre>
<p>If Spamhaus returns an IP address between and, then
we know that this particular email is of dubious origin. In that case we can
put it in the folder <code>$MAILDIR/Spam</code>. Otherwise, the message will
get written to your inbox by default.</p>
<pre> :0
* KNOWNOFFENDER ?? ^^127.0.0.[0-9]*^^
LOG = "This sender is a known source of spam.
LOG = "This sender is not a known source of spam.
<p>This completes the Procmail recipe.</p>
<h2><a id="ssh-smtp" />Use SDF SMTP Server Remotely via SSH</h2>
<p>Using SSH's port-forwarding capabilities, MetaARPA members can create
an encrypted tunnel which makes the SDF SMTP mail server appear local,
avoiding need for authentication or additional encryption.</p>
<li>SDF MetaARPA membership
<li>SSH plus suitable MUA (email program)
<li>public key authentication (optional)
<h3>Tunnel Construction:</h3>
<p>Port-forwarding can occur on ports 53, 110, 143, 443, and 993 on most SDF hosts.</p>
Use port <em>443</em> on SDF host <em>"otaku"</em>, tunnel local port
<em>2525</em> to port <em>25</em> on SDF host <em>"mx"</em> (SMTP server); restrict<br>
ssh to IPv4, no remote commands, run in background after connecting:</p>
% <b>ssh -4 -fN -p 443 -L 2525:mx:25</b>
<p>Note if public key authentication isn't setup a password prompt occurs.</p>
<h3>Tunnel Test:</h3>
<p><i>telnet(1)</i> can be used; type <em>"quit"</em> to exit session:</p>
% <b>telnet localhost 2525</b>
Connected to localhost.
Escape character is '^]'.
220 ESMTP Sendmail 8.14.5/8.14.3; Tue 13 Dec 2011 07:45:59 GMT
221 2.0.0 closing connection
Connection closed by foreign host.
<h3>Tunnel Usage:</h3>
<p>Configure MUA to use <em>""</em> (localhost), port <em>2525</em> for out-going
messages. Also ensure the <em>From:</em> and <em>ReplyTo:</em> addresses are routeable.
Handling of rejected email is MUA-dependent - some do queuing, others
write to <em>$HOME/dead.letter</em> or <em>/dev/null</em>:</p>
Configure and test the <i>Heirloom Mailx</i> MUA for user <em>"frog"</em>
on localhost <em>"mud.bog"</em> <br>to use <em>"localhost:2525"</em> for
SMTP (off-site email):</p>
# <b><i>$HOME/.nailrc</i></b><code>
set smtp="localhost:2525"
set from="You &#60;;"
set replyto="You &#60;;" </code>
% <b>hmail -v</b>
Subject: tunneled SMTP test
test 123
Resolving host localhost . . . done.
Connecting to . . . connected.
220 ESMTP Sendmail 8.14.5/8.14.3; Tue, 13 Dec 2011 08:21:22 GMT
>>> HELO mud.bog
250 Hello [], pleased to meet you
>>> MAIL FROM:<frog@mud.bog>
250 2.1.0 <frog@mud.bog>... Sender ok
>>> RCPT TO:<>
250 2.1.5 <>... Recipient ok
>>> DATA
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 pBD8LM7d000515 Message accepted for delivery
>>> QUIT
221 2.0.0 closing connection
<h3>Tunnel Teardown:</h3>
<p>The SSH tunnel can be torn down by identifying the appropriate PID
(process ID) and sending it a <em>kill</em> signal, via the
<em>kill(1)</em> command or similar tool. The following illustrates the
teardown process using <em>sockstat(1)</em> to determine the PID on the
<i>NetBSD</i> operating system:</p>
% <b>sockstat -np 2525</b>
frog ssh 24362 7 tcp *.*
% <b>kill -HUP 24362</b>
Killed by signal 1.
<h3>Tunnel Maintenance:</h3>
<p>Network connections can fail for various reasons; a keep-alive script can help:<br>
(script contributed by SDF user <em>nullogic</em> - thanks!)</p>
# Keep alive script for local ssh tunnel from home to SDF SMTP
SSH_ARGS='-N -f -L'
while [ 1 ]; do
SSH_PID=`ps aux | sed -n -e /sed/d -e /ssh\ \-N/p | wc -m`
if [ $SSH_PID -eq 0 ]; then
-p$SSH_PORT &>/dev/null
DATE_TIME=`date '+%y.%m.%d %T'`
echo "$DATE_TIME SSH Tunnel restarted.">>$LOG_FILE
sleep $SLEEP
<p>Writing the PID to a file at the time of tunnel creation can simplify the ID process.</p>
<li>Setting up <a href="">Public Key Authentication</a>
<li>OpenSSH FAQ: How do I use <a href="">Port Forwarding?</a>
<h2><a id="gloss" />Glossary</h2>
<hr />
<cite>$Id: e-mail-advanced.html,v 1.17 2012/01/06 21:53:01 jgw Exp ruscorp $</cite>