396 lines
14 KiB
HTML
396 lines
14 KiB
HTML
<h1>E-Mail, Advanced Topics</h1>
|
|
<p>This tutorial discusses advanced topics concerning e-mail on
|
|
SDF: spam filtering, automatic processing, forwarding. It is probably
|
|
only useful for ARPA members.</p>
|
|
|
|
<h3>Contents</h3>
|
|
<ol>
|
|
<li><a href="#intro">Introduction, and what this is <em>not</em></a></li>
|
|
<li><a href="#forward">Forwarding E-Mail</a></li>
|
|
<li><a href="#fetch">Fetching E-Mail</a></li>
|
|
<li><a href="#spam">Fighting Spam</a></li>
|
|
<li><a href="#whitelist">Personal White List Against Spam</a></li>
|
|
<li><a href="#dnsbl">Using Procmail With a DNSBL</a></li>
|
|
<li><a href="#ssh-smtp">Use SDF SMTP Server Remotely via SSH</a></li>
|
|
<!--
|
|
<li><a href="#gloss">Glossary</a> - explications of terms</li>
|
|
-->
|
|
</ol>
|
|
<hr />
|
|
|
|
<h2><a id="intro" />Introduction, and what you <em>will not</em> find here</h2>
|
|
<p>Here we discuss advanced topics for e-mail processing on SDF, mainly
|
|
automatic processing of incoming messages with <code>procmail</code>, and
|
|
forwarding issues. As <code>procmail</code> is only available to ARPA
|
|
members, the contents of this tutorial will be mostly useless to user
|
|
members.
|
|
<br />
|
|
For basic info about e-mail (addresses, reader programs, file size, etc),
|
|
please read the corresponding
|
|
<a href="http://sdf.lonestar.org/index.cgi?faq?EMAIL">FAQ</a> entry - and if
|
|
you still have problems with basic sending and receiving of messages
|
|
via SDF, this tutorial is most probably not for you!
|
|
</p>
|
|
|
|
<h2><a id="forward" />Forwarding E-Mail</h2>
|
|
<p>The <code>sendmail</code> system allows for automatic forwarding of
|
|
incoming mail to other addresses or even programs (filters).<br />
|
|
The file <code>$HOME/.forward</code> may contain the following types of
|
|
lines:
|
|
<dl>
|
|
<dt>somebody@somewhere.com</dt>
|
|
<dd>external e-mail address, where a copy of each mail will be sent</dd>
|
|
<dt>|<code>filterprogram</code></dt>
|
|
<dd>will pipe the incoming messages through <code>/bin/sh</code> with the
|
|
argument <code>filterprogram</code></dd>
|
|
</dl>
|
|
</p>
|
|
<p>However, if you want to do more than forwarding to one address,
|
|
use of <code>procmail</code> is highly recommended! In fact, it is
|
|
best to set up <a href="#nospam">nospam -e</a>, which will generate
|
|
a .forward file with a line calling procmail, and then to modify
|
|
the resulting <code>.procmailrc</code>.
|
|
|
|
<h2><a id="fetch" />Fetching E-Mail</h2>
|
|
<p>Use <code>fetchmail</code> to collect messages from accounts
|
|
you might have elsewhere. The messages are given to <code>procmail</code>
|
|
(see <a href="#forward">Forwarding E-Mail</a> above) for further
|
|
processing.</p>
|
|
|
|
<h3><a id="fetch-ex" />Sample <code>.fetchmailrc</code></h3>
|
|
<pre>
|
|
poll pop.provider.net
|
|
proto pop3
|
|
user "john.doe"
|
|
pass "secret"
|
|
is "jdoe" here
|
|
mda "/path/to/procmail -f- ~/.procmailrc"
|
|
ssl;
|
|
</pre>
|
|
|
|
<h2><a id="spam" />Fighting Spam</h2>
|
|
<p>To fight unwanted bulk e-mail (spam), there are several possibilities
|
|
available on SDF:
|
|
<ol>
|
|
<li>As a general remark, think before you publish or tell somebody your
|
|
e-mail address! And try to make clear to people that you don't want your
|
|
address on a visible list of 500 recipients from somebody's address book; one
|
|
should use Bcc: for sending to a lot of people in parallel, not To:!</li>
|
|
<li>SDF also supports plus-addressing;
|
|
you can use this to join mailing lists and do filters.
|
|
This can be used by adding +something before the @ in your email address.</li>
|
|
<li>The <a href="http://sdf.lonestar.org/index.cgi?faq?EMAIL?07">FAQ</a>
|
|
has more information about <a href="#nospam">nospam</a>, available on SDF.</li>
|
|
<li>The automatic mail processor <code>procmail</code> (see <code>man
|
|
procmail</code> for more information)
|
|
allows to filter incoming mail following user defined rules; in particular,
|
|
you can generate a <a href="#whitelist">white list</a> of sender addresses
|
|
you know as good, and put messages from other addresses into a special
|
|
folder which you browse through now and then, to check for good messages
|
|
and adding the corresponding addresses to your white list.</li>
|
|
</ol>
|
|
</p>
|
|
|
|
<h3><a id="nospam" />nospam</h3>
|
|
<p><code>nospam</code> is a script which can set up .forward and .procmailrc
|
|
files so that certain e-mails will be stored in "bulk folders".
|
|
<br>
|
|
<code>nospam -i</code> gives some additional information, and <code>nospam
|
|
-e</code> will enable this filtering. We will suppose that this has been done
|
|
also for the following discussion of <a href="#whitelist">white
|
|
lists</a>.</p>
|
|
|
|
<h2><a id="whitelist" />Personal White List Against Spam</h2>
|
|
<p><em>White lists</em> contain e-mail addresses which are always
|
|
to be accepted as good. It is easy to implement this with
|
|
<code>procmail</code>: Supposing that <code>nospam -e</code> has been
|
|
executed, so that basic procmail processing is set up, one can add the
|
|
following fragment to <code>.procmailrc</code>.</p>
|
|
|
|
<p><em>Note:</em> For this to be working, the following <em>must</em> have
|
|
been set up:
|
|
<ul>
|
|
<li>a <code>.forward</code> file which pipes incoming e-mail through
|
|
<code>procmail</code></li>
|
|
<li>a standard <code>.procmailrc</code> file, where this fragment is attached
|
|
or inserted</li>
|
|
</ul>
|
|
The easiest method to install these files is by launching
|
|
<code>nospam -e</code> as mentioned <a href="#nospam">above</a>, then adding
|
|
the fragment to the generated <code>.procmailrc</code>.
|
|
<br />
|
|
(Of course, it is possible to deactivate or delete all the initial
|
|
procmailrc lines generated by nospam, which move messages with
|
|
attachments to the bulk folder, if such messages should pass through
|
|
the whitelist filter.)
|
|
</p>
|
|
|
|
<pre>
|
|
# procmail spamfilter using whitelist (2006 Yargo Bonetti)
|
|
# :: use at your own risk and any way you want! ::
|
|
# whitelist: file containing one valid e-mail address per line
|
|
# (only generic xxx@yyy.zz form, without "Name.." <*> parts)
|
|
WHITELIST=$MAILDIR/.whitelist
|
|
# spamfilter
|
|
FROM=`formail -c -x 'From:'|sed -e 's/.*<\(.*@[^>]*\)>.*/\1/'`
|
|
:0 hb:
|
|
* ! ? grep -i -F -e "$FROM" $WHITELIST >/dev/null
|
|
$MAILDIR/quarantine
|
|
</pre>
|
|
|
|
<p>This will compare the address in the From: field of incoming
|
|
messages to the ones saved in the file $MAILDIR/.whitelist (of
|
|
course, name and location can be changed), and when an address is
|
|
<em>not found</em> in this file, the message will be saved in the
|
|
folder $MAILDIR/quarantine (which can be modified as well) and not show up in
|
|
the normal Inbox (and not be processed further). Now and then, one can look
|
|
through the quarantine folder for "good messages", and add the
|
|
corresponding addresses to the .whitelist file.</p>
|
|
|
|
<p>If you want to use the mail directories proposed by nospam, you may
|
|
prepend the above noted fragment by</p>
|
|
|
|
<pre>
|
|
MAILDIR=$HOME/mail
|
|
</pre>
|
|
|
|
<p>which will result in storing the "possibly bad" messages in
|
|
<code>~/mail/quarantine</code> and expect the whitelist in
|
|
<code>~/mail/.whitelist</code>.</p>
|
|
|
|
<p>A sample .whitelist may look like this:</p>
|
|
|
|
<pre>
|
|
my.friend@his.domain.com
|
|
president@whitehouse.gov
|
|
myself@gmail.com
|
|
</pre>
|
|
|
|
<p>All messages not coming from these three addresses would end up in the
|
|
"quarantine" folder defined in the .procmailrc fragment.</p>
|
|
|
|
<h2><a id="dnsbl" />Using Procmail With a DNSBL</h2>
|
|
|
|
<p>DNSBL stands for "DNS blacklist", and it is essentially a collection of IP
|
|
addresses known to send out spam and other bad stuff. Two popular DNSBLs are
|
|
<a href="http://www.spamcop.net">SpamCop</a> and
|
|
<a href="http://www.spamhaus.org">Spamhaus</a>. Typically the mail server
|
|
will check the source of incoming mail with the DNSBL before even accepting
|
|
the message. SDF does not do this, but using Procmail to check a DNSBL at
|
|
the user level is easy. In this tutorial we will use Spamhaus.</p>
|
|
|
|
<p>(Thank you <a href="http://www.benya.com/procmail/#dnsbl">Benya</a> for
|
|
the original instructions.)</p>
|
|
|
|
<p>First make sure that <code>~/.forward</code> is set-up to forward incoming
|
|
mail to Procmail. Then we'll edit <code>~/.procmailrc</code>. The beginning
|
|
of this file should declare basic variables such as <code>SHELL=/bin/sh</code>,
|
|
<code>LOGFILE=$HOME/.procmaillog</code>, <code>ORGMAIL</code>,
|
|
<code>DEFAULT</code>, and <code>MAILDIR</code>. <code>ORGMAIL</code> and
|
|
<code>DEFAULT</code> should point to your inbox, and <code>MAILDIR</code>
|
|
should point to the directory that contains your saved mail.</p>
|
|
|
|
<p>Next Procmail should extract the IP address from which the message was
|
|
sent. This is done using <code>formail</code> to get the headers,
|
|
<code>grep</code> to find the correct line, and <code>sed</code> to find the
|
|
actual IP address, which is then saved as <code>SENDERIP</code>.</p>
|
|
|
|
<pre>SENDERIP = `formail -c -XReceived | grep "by sdf.lonestar.org" | \
|
|
grep -v "from sdf.lonestar.org" | \
|
|
sed "s/^Received: from .*\[\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\].*by sdf.lonestar.org.*$/\1/"`</pre>
|
|
|
|
<p>Then we'll begin the actual Procmail recipe, which will only be executed
|
|
after checking to make sure that the <code>SENDERIP</code> variable exists
|
|
and is in the correct format. Anything written to the <code>LOG</code> variable
|
|
will be inserted into the log file. (New lines must be explicitly stated.)</p>
|
|
|
|
<pre>:0
|
|
* SENDERIP ?? ^^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*^^
|
|
{
|
|
LOG = "The sender's IP address is "
|
|
LOG = $SENDERIP
|
|
LOG = "
|
|
"</pre>
|
|
|
|
<p>The next step is to reverse the IP address because Spamhaus wants it that
|
|
way. Procmail will then use <code>host</code> to determine the IP address that
|
|
[reversed IP address].zen.spamhaus.org resolves to.</p>
|
|
|
|
<pre> SENDER_REVERSED = `expr "$SENDERIP" | \
|
|
sed "s/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/"`
|
|
|
|
KNOWNOFFENDER = `host "$SENDER_REVERSED".zen.spamhaus.org | \
|
|
sed "s/^.*\(127\.0\.0\.[0-9]*\)$/\1/"`</pre>
|
|
|
|
<p>If Spamhaus returns an IP address between 127.0.0.0 and 127.0.0.9, then
|
|
we know that this particular email is of dubious origin. In that case we can
|
|
put it in the folder <code>$MAILDIR/Spam</code>. Otherwise, the message will
|
|
get written to your inbox by default.</p>
|
|
|
|
<pre> :0
|
|
* KNOWNOFFENDER ?? ^^127.0.0.[0-9]*^^
|
|
{
|
|
LOG = "This sender is a known source of spam.
|
|
"
|
|
|
|
:0:
|
|
Spam
|
|
}
|
|
|
|
LOG = "This sender is not a known source of spam.
|
|
"
|
|
}</pre>
|
|
|
|
<p>This completes the Procmail recipe.</p>
|
|
<br>
|
|
|
|
<h2><a id="ssh-smtp" />Use SDF SMTP Server Remotely via SSH</h2>
|
|
|
|
<h3>Introduction:</h3>
|
|
<p>Using SSH's port-forwarding capabilities, MetaARPA members can create
|
|
an encrypted tunnel which makes the SDF SMTP mail server appear local,
|
|
avoiding need for authentication or additional encryption.</p>
|
|
|
|
<ul><b>Prerequisites:</b>
|
|
<li>SDF MetaARPA membership
|
|
<li>SSH plus suitable MUA (email program)
|
|
<li>public key authentication (optional)
|
|
</ul>
|
|
|
|
<h3>Tunnel Construction:</h3>
|
|
<p>Port-forwarding can occur on ports 53, 110, 143, 443, and 993 on most SDF hosts.</p>
|
|
|
|
<blockquote>
|
|
<p><b>ex:</b><br>
|
|
Use port <em>443</em> on SDF host <em>"otaku"</em>, tunnel local port
|
|
<em>2525</em> to port <em>25</em> on SDF host <em>"mx"</em> (SMTP server); restrict<br>
|
|
ssh to IPv4, no remote commands, run in background after connecting:</p>
|
|
|
|
<pre>
|
|
% <b>ssh -4 -fN -p 443 -L 2525:mx:25 sdf_user@otaku.sdf.org</b>
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p>Note if public key authentication isn't setup a password prompt occurs.</p>
|
|
|
|
<h3>Tunnel Test:</h3>
|
|
<p><i>telnet(1)</i> can be used; type <em>"quit"</em> to exit session:</p>
|
|
|
|
<blockquote>
|
|
<pre>
|
|
% <b>telnet localhost 2525</b>
|
|
Trying 127.0.0.1...
|
|
Connected to localhost.
|
|
Escape character is '^]'.
|
|
220 sdf.lonestar.org ESMTP Sendmail 8.14.5/8.14.3; Tue 13 Dec 2011 07:45:59 GMT
|
|
<b>quit</b>
|
|
221 2.0.0 sdf.lonestar.org closing connection
|
|
Connection closed by foreign host.
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<h3>Tunnel Usage:</h3>
|
|
<p>Configure MUA to use <em>"127.0.0.1"</em> (localhost), port <em>2525</em> for out-going
|
|
messages. Also ensure the <em>From:</em> and <em>ReplyTo:</em> addresses are routeable.
|
|
Handling of rejected email is MUA-dependent - some do queuing, others
|
|
write to <em>$HOME/dead.letter</em> or <em>/dev/null</em>:</p>
|
|
|
|
<blockquote>
|
|
<p><b>ex:</b><br>
|
|
Configure and test the <i>Heirloom Mailx</i> MUA for user <em>"frog"</em>
|
|
on localhost <em>"mud.bog"</em> <br>to use <em>"localhost:2525"</em> for
|
|
SMTP (off-site email):</p>
|
|
|
|
<pre>
|
|
# <b><i>$HOME/.nailrc</i></b><code>
|
|
set smtp="localhost:2525"
|
|
set from="You <someone@sdf.org>"
|
|
set replyto="You <someone@sdf.org>" </code>
|
|
|
|
% <b>hmail -v someone@sdf.org</b>
|
|
Subject: tunneled SMTP test
|
|
test 123
|
|
.
|
|
Resolving host localhost . . . done.
|
|
Connecting to 127.0.0.1:2525 . . . connected.
|
|
220 sdf.lonestar.org ESMTP Sendmail 8.14.5/8.14.3; Tue, 13 Dec 2011 08:21:22 GMT
|
|
>>> HELO mud.bog
|
|
250 sdf.lonestar.org Hello IDENT:root@otaku.sdf.org [192.94.73.6], pleased to meet you
|
|
>>> MAIL FROM:<frog@mud.bog>
|
|
250 2.1.0 <frog@mud.bog>... Sender ok
|
|
>>> RCPT TO:<someone@sdf.org>
|
|
250 2.1.5 <someone@sdf.org>... Recipient ok
|
|
>>> DATA
|
|
354 Enter mail, end with "." on a line by itself
|
|
>>> .
|
|
250 2.0.0 pBD8LM7d000515 Message accepted for delivery
|
|
>>> QUIT
|
|
221 2.0.0 sdf.lonestar.org closing connection
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<h3>Tunnel Teardown:</h3>
|
|
<p>The SSH tunnel can be torn down by identifying the appropriate PID
|
|
(process ID) and sending it a <em>kill</em> signal, via the
|
|
<em>kill(1)</em> command or similar tool. The following illustrates the
|
|
teardown process using <em>sockstat(1)</em> to determine the PID on the
|
|
<i>NetBSD</i> operating system:</p>
|
|
|
|
<blockquote>
|
|
<pre>
|
|
% <b>sockstat -np 2525</b>
|
|
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
|
|
frog ssh 24362 7 tcp 127.0.0.1.2525 *.*
|
|
|
|
% <b>kill -HUP 24362</b>
|
|
Killed by signal 1.
|
|
</pre>
|
|
</blockquote>
|
|
|
|
|
|
<h3>Tunnel Maintenance:</h3>
|
|
<p>Network connections can fail for various reasons; a keep-alive script can help:<br>
|
|
(script contributed by SDF user <em>nullogic</em> - thanks!)</p>
|
|
|
|
<blockquote>
|
|
<pre><code>
|
|
#!/bin/bash
|
|
# sdftun.sh: Keep alive script for local ssh tunnel from home to SDF SMTP
|
|
|
|
SSH_USER=
|
|
SSH_HOST='faeroes.freeshell.org'
|
|
SSH_PORT=443
|
|
SSH_ARGS='-N -f -L'
|
|
TAG_HOST=mx
|
|
TAG_PORT=25
|
|
LIS_PORT=8080
|
|
LOG_FILE=~/l0g/sdftun.log
|
|
SLEEP="30m"
|
|
|
|
while [ 1 ]; do
|
|
SSH_PID=`ps aux | sed -n -e /sed/d -e /ssh\ \-N/p | wc -m`
|
|
if [ $SSH_PID -eq 0 ]; then
|
|
ssh $SSH_ARGS $LIS_PORT:$TAG_HOST:$TAG_PORT $SSH_USER@$SSH_HOST \
|
|
-p$SSH_PORT &>/dev/null
|
|
DATE_TIME=`date '+%y.%m.%d %T'`
|
|
echo "$DATE_TIME SSH Tunnel restarted.">>$LOG_FILE
|
|
fi
|
|
sleep $SLEEP
|
|
done
|
|
</code></pre>
|
|
</blockquote>
|
|
|
|
<p>Writing the PID to a file at the time of tunnel creation can simplify the ID process.</p>
|
|
|
|
<ul><b>References:</b>
|
|
<li>Setting up <a href="http://sdf.org/?tutorials/SSH-SDF#public_key">Public Key Authentication</a>
|
|
<li>OpenSSH FAQ: How do I use <a href="http://openssh.com/faq.html#2.11">Port Forwarding?</a>
|
|
</ul>
|
|
|
|
<!--
|
|
<h2><a id="gloss" />Glossary</h2>
|
|
-->
|
|
|
|
<hr />
|
|
<cite>$Id: e-mail-advanced.html,v 1.17 2012/01/06 21:53:01 jgw Exp ruscorp $</cite>
|