Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Combating spam, or how I befriended the Killer ...

Combating spam, or how I befriended the Killer Rabbit of Caerbannog

You might have received an unwanted email at some point. We all have. According to some studies, between 80 and 90 percent of all email is spam. Those of us with accounts at established email providers can rely on their hosts' filters to keep their inboxes manageable. What if you're hosting your email on your own, though? Off-the-shelf open source solutions are there when you need them, but that's not where the fun is. Combining existing tools, building your own classifiers, and seeing them work in practice is far more exciting. Let me tell you a story, one with rabbits.

Video: https://janstepien.com/combating-spam-july-2017/

Jan Stępień

July 27, 2017
Tweet

More Decks by Jan Stępień

Other Decks in Technology

Transcript

  1. 0 0.25 0.5 0.75 1 Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec Spam ratio, 2017
  2. Return-Path: <[email protected]> X-Original-To: [email protected] Delivered-To: [email protected] Received: from ktxd8z.happysillyfeetman.net (ktxd8z.happysillyfeetman.net

    [213.5.68.131]) by r245-52.iq.pl (Postfix) with ESMTP id DCE364A61FC4 for <[email protected]>; Wed, 14 Oct 2015 08:46:12 +0200 (CEST) Received: from 05f541e2.ktxd8z.happysillyfeetman.net (amavisd, port 7307) by ktxd8z.happysillyfeetman.net with ESMTP id 05LVF541OGE2; for <[email protected]>; Tue, 13 Oct 2015 23:46:07 -0700 To: <[email protected]> Date: Tue, 13 Oct 2015 23:46:07 -0700 Message-ID: <[email protected]> From: "FlirtLife" <[email protected]> Subject: ATTN: You have (1) New Message Content-Language: en-us MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: multipart/alternative; boundary="----=Part.592.3860.1444805167" ------=Part.592.3860.1444805167 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="UTF-8" You received a new message on Flirtlocal.com From: adorableAira I need a spark in my sexual life...i want someone who can... ?
  3. a s 0.7 + b s 0.4 + c h

    0.2 + t = 0.1
  4. a s 0.7 + b s 0.4 + c h

    0.2 – t = 0.3
  5. a s 0.7 + b s 0.4 – c h

    0.2 – t = 0.5
  6. a s 0.7 – b s 0.4 – c h

    0.2 – t = 0.9
  7. Return-Path: <[email protected]> X-Original-To: [email protected] Delivered-To: [email protected] Received: from ktxd8z.happysillyfeetman.net by

    r245-52.iq.pl (Postfix) with ESMTP id DCE364A61FC4 for <[email protected]>; Wed, 14 Oct 2015 08:46:12 +0200 (CEST) To: <[email protected]> Date: Tue, 13 Oct 2015 23:46:07 -0700 Message-ID: <530723999953959530767719911262707@...> From: "FlirtLife" <[email protected]> Subject: ATTN: You have (1) New Message Content-Language: en-us MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: multipart/alternative; boundary="----=Part.592.3860.1444805167" ------=Part.592.3860.1444805167 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="UTF-8" You received a new message on Flirtlocal.com From: adorableAira I need a spark in my sexual life...i want someone who can... bogolexer
  8. head:alternative mime:Content-Transfer mime:bit mime:Content-Type mime:text mime:plain mime:charset mime:UTF-8 You received

    new message Flirtlocal.com From adorableAira need spark sexual life...i rtrn:FlirtLife rtrn:happysilly.net head:X-Original-To head:jan head:stepien.cc head:Delivered-To head:jan head:stepien.cc rcvd:from rcvd:r245-52.iq.pl rcvd:Postfix rcvd:with rcvd:ESMTP rcvd:for rcvd:jan rcvd:stepien.cc rcvd:Wed rcvd:Oct rcvd:CEST to:jan to:stepien.cc head:Date head:Message-ID from:FlirtLife from:FlirtLife from:happysilly.net subj:ATTN subj:You subj:have subj:New subj:Message head:Content-Language head:en-us head:MIME-Version head:Content-Transfer head:bit head:Content-Type head:multipart
  9. Check for presence of tokens to:stępień 0.52 …and to:jan 0.63

    rcvd:tlsv1.2 0.66 …and to:stępień 0.84 …and 3 more tokens 0.89
  10. lz4 50 lines of each email concatenate them together and

    compress with lz4 …for both spam and ham
  11. false positive rate 0 0 1 1 That’s not a

    problem true positive rate
  12. Technical details Postfix as my MTA procmail to call the

    filter bogolexer and VW to score and procmail to deliver
  13. for message in $(find archive/ -type f); do bogolexer -p

    < $message \ | tr '[:upper:]' '[:lower:]' \ | egrep -vi '(bogosity|bogofilter|vowpal-wabbit)' \ | sort | uniq | tr "\n" " " done vw bag.gz -b 15 --passes 8 --cache_file m-cache -f model
  14. bogolexer -p \ | tr '[:upper:]' '[:lower:]' \ | sort

    | uniq | tr "\n" " " \ | vw -t -i model -p /dev/stdout 0.964 Vowpal-Wabbit-Says:
  15. cmd="< I filter.sh | grep ^Vowpal-Wabbit-Says \ | sed 's|^Vowpal-Wabbit-Says:

    ||; s|$| I|;’" find archive/ -type f -print0 \ | xargs -0 -P 4 -I I sh -c "$cmd" | tee results grep -v archive/Junk/ results | sort -r | head grep archive/Junk/ results | sort | head