SpamFilter seems like to be ineffective

MrTalent's Debian config questions go here!

Moderators: Admins, Forum Moderator

SpamFilter seems like to be ineffective

Postby vmaillot » Mon Feb 20, 2006 5:23 pm

Hi !
I really want to thank you mr88talent because your "howto" is wonderfull !
Excepted that it seems like to not working for me :(

I've strictly followed your howto, but when it comes the time to test the eicar virus, it starts to show me that it's not well working --> the mail pass through the spam filter. ok it's not a virus filter but the eicar must be filtered by our machine ... By extension, the spam are not stoped : i've tried with the GFI's email tests (http://www.gfi.com/emailsecuritytest) and I received ALL the mails :cry:
So I can conclude that I've a problem...

I've triple checked every parameters, I've reinstalled from scratch, and I've checked once more every conf file, without any success.

would you please help me ?

------
here is my configuration :

spamfilter box (web02-gti) = 192.168.68.9
exchange server = 192.168.68.2

------
here is the mail.log extract concerning mail sending test (from GFI):

Code: Select all
web02-gti:/usr/local/src# tail -f /var/log/mail.log
Feb 20 14:56:33 web02-gti amavis[2527]: Found decoder for    .cab  at /usr/bin/cabextract
Feb 20 14:56:33 web02-gti amavis[2527]: No decoder for       .tnef tried: tnef
Feb 20 14:56:33 web02-gti amavis[2527]: Internal decoder for .tnef
Feb 20 14:56:33 web02-gti amavis[2527]: Found decoder for    .exe  at /usr/bin/unrar; /usr/bin/lha; /usr/bin/arj
Feb 20 14:56:33 web02-gti amavis[2527]: Using internal av scanner code for (primary) ClamAV-clamd
Feb 20 14:56:33 web02-gti amavis[2527]: Found primary av scanner BitDefender at /usr/bin/bdc
Feb 20 14:56:33 web02-gti amavis[2527]: Found secondary av scanner ClamAV-clamscan at /usr/bin/clamscan
Feb 20 14:56:33 web02-gti amavis[2527]: Creating db in /var/lib/amavis/db/; BerkeleyDB 0.26, libdb 4.3
Feb 20 14:56:36 web02-gti postfix/postfix-script: starting the Postfix mail system
Feb 20 14:56:36 web02-gti postfix/master[2600]: daemon started -- version 2.1.5
Feb 20 14:57:24 web02-gti postfix/smtpd[2507]: connect from localhost.localdomain[127.0.0.1]
Feb 20 14:57:24 web02-gti postfix/smtpd[2507]: 8042E56A79: client=localhost.localdomain[127.0.0.1]
Feb 20 14:57:24 web02-gti postfix/cleanup[2500]: 8042E56A79: message-id=<20060220195317.906C456A52@web02-gti.domain1.com>
Feb 20 14:57:24 web02-gti postfix/qmgr[2014]: 8042E56A79: from=<external_private_adress@other_domain.com>, size=2084, nrcpt=1 (queue active)
Feb 20 14:57:24 web02-gti postfix/smtpd[2507]: disconnect from localhost.localdomain[127.0.0.1]
Feb 20 14:57:24 web02-gti postfix/smtp[2501]: 906C456A52: to=<vmaillot@domain1.com>, relay=127.0.0.1[127.0.0.1], delay=7, status=sent (250 2.6.0 Ok, id=02496-01, from MTA([127.0.0.1]:10025): 250 Ok: queued as 8042E56A79)
Feb 20 14:57:24 web02-gti postfix/qmgr[2014]: 906C456A52: removed
Feb 20 14:57:24 web02-gti postfix/smtp[2508]: 8042E56A79: to=<vmaillot@domain1.com>, relay=192.168.68.2[192.168.68.2], delay=0, status=sent (250 2.6.0  <20060220195317.906C456A52@web02-gti.domain1.com> Queued mail for delivery)
Feb 20 14:57:24 web02-gti postfix/qmgr[2014]: 8042E56A79: removed


-----

master.cf

Code: Select all
#
# Postfix master process configuration file.  For details on the format
# of the file, see the Postfix master(5) manual page.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       -       -       -       smtpd
#submission inet n      -       -       -       -       smtpd
#       -o smtpd_etrn_restrictions=reject
#       -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#smtps    inet  n       -       -       -       -       smtpd
#  -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes
#submission inet n      -       -       -       -       smtpd
#  -o smtpd_etrn_restrictions=reject
#  -o smtpd_enforce_tls=yes -o smtpd_sasl_auth_enable=yes
#628      inet  n       -       -       -       -       qmqpd
pickup    fifo  n       -       -       60      1       pickup
        -o content_filter=
        -o receive_override_options=no_header_body_checks
cleanup   unix  n       -       -       -       0       cleanup
qmgr      fifo  n       -       -       300     1       qmgr
#qmgr     fifo  n       -       -       300     1       oqmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
smtp      unix  -       -       -       -       -       smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       -       -       -       smtp
        -o fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache
#
smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
#
127.0.0.1:10025 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o mynetworks=127.0.0.0/8
        -o strict_rfc821_envelopes=yes
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
#
# See the Postfix UUCP_README file for configuration details.
#
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -d -t$nexthop -f$sender $recipient
scalemail-backend unix  -       n       n       -       2       pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}


-------
main.cf

Code: Select all
# See /usr/share/postfix/main.cf.dist for a commented, more complete version

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

alias_maps = hash:/etc/aliases
myorigin = domain1.com
myhostname = web02-gti.domain1.com
mynetworks = 127.0.0.0/8, 192.168.68.0/24
message_size_limit = 104857600
local_transport = error:no local mail delivery
mydestination =
local_recipient_maps =
virtual_alias_maps = hash:/etc/postfix/virtual
relayhost = [192.168.68.2]
relay_recipient_maps = hash:/etc/postfix/relay_recipients
transport_maps = hash:/etc/postfix/transport
relay_domains = domain1.com, domain2.ca, domain3.com, domain4.com
recipient_delimiter =
smtpd_helo_required = yes
smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/sender_access, reject_non_fqdn_sender, reject_unknown_sender_domain
smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_unauth_pipelining
smtpd_data_restrictions = reject_unauth_pipelining
header_checks = pcre:/etc/postfix/header_checks
body_checks = pcre:/etc/postfix/body_checks
content_filter = smtp-amavis:[127.0.0.1]:10024


-------

/etc/postfix/relay_recipients
Code: Select all
@domain1.com 1
@domain2.com 1
@domain3.com 1
@domain4.com 1

------
/etc/postfix/transport
Code: Select all
@domain1.com smtp:[192.168.68.2]
@domain2.com smtp:[192.168.68.2]
@domain3.com smtp:[192.168.68.2]
@domain4.com smtp:[192.168.68.2]

------

amavisd.conf is here : http://www.gtinnovatech.com/amavisd.txt



-----
edit = some orthography errors wiped out (sorry for my horrible english :wink: )
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Mon Feb 20, 2006 7:07 pm

This is strange.

amavis should show some info like this:
Code: Select all
Feb 20 15:41:00 sfm amavis[29442]: (29442-01) Passed CLEAN, LOCAL [192.168.0.102] [192.168.0.102] <88fan@example.net> -> <garyv@example.com>, Message-ID: <658236748.20060220154054@example.com>, mail_id: 211I5utMhnxj, Hits: 10.179-20, 5317 ms


Does
grep amavis /var/log/mail.log
show anything?


in amavisd.conf, there is a mistake...
Code: Select all
@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
#                   192.168.68.0/16);


remove the comment, this should be:
Code: Select all
@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
                   192.168.68.0/16);


Fix this, reload amavisd-new, send one plain test file through. I would like to see the log from beginning to end for that one test email.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby mr88talent » Mon Feb 20, 2006 7:19 pm

Another mistake. You need to uncomment this line:
Code: Select all
# $myhostname = 'web02-gti.domain1.com';

There should be no error messages when amavisd-new is started.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Tue Feb 21, 2006 10:45 am

Thanks for your so-fast answer !
After the modifications, I've got that :

Code: Select all
web02-gti:~# grep amavis /var/log/mail.log
Feb 21 09:32:09 web02-gti amavis[1652]: starting.  /usr/sbin/amavisd-new at web02-gti.gtinnovatech.com amavisd-new-2.3.2 (20050629), Unicode aware
Feb 21 09:32:09 web02-gti amavis[1652]: Perl version               5.008004
Feb 21 09:32:10 web02-gti amavis[1653]: Module Amavis::Conf        2.042
Feb 21 09:32:10 web02-gti amavis[1653]: Module Archive::Tar        1.23
Feb 21 09:32:10 web02-gti amavis[1653]: Module Archive::Zip        1.14
Feb 21 09:32:10 web02-gti amavis[1653]: Module BerkeleyDB          0.26
Feb 21 09:32:10 web02-gti amavis[1653]: Module Compress::Zlib      1.34
Feb 21 09:32:10 web02-gti amavis[1653]: Module Convert::TNEF       0.17
Feb 21 09:32:10 web02-gti amavis[1653]: Module Convert::UUlib      1.051
Feb 21 09:32:10 web02-gti amavis[1653]: Module DBI                 1.46
Feb 21 09:32:10 web02-gti amavis[1653]: Module DB_File             1.808
Feb 21 09:32:10 web02-gti amavis[1653]: Module MIME::Entity        5.417
Feb 21 09:32:10 web02-gti amavis[1653]: Module MIME::Parser        5.417
Feb 21 09:32:10 web02-gti amavis[1653]: Module MIME::Tools         5.417
Feb 21 09:32:10 web02-gti amavis[1653]: Module Mail::Header        1.62
Feb 21 09:32:10 web02-gti amavis[1653]: Module Mail::Internet      1.62
Feb 21 09:32:10 web02-gti amavis[1653]: Module Mail::SPF::Query    1.997
Feb 21 09:32:10 web02-gti amavis[1653]: Module Mail::SpamAssassin  3.000003
Feb 21 09:32:10 web02-gti amavis[1653]: Module Net::Cmd            2.26
Feb 21 09:32:10 web02-gti amavis[1653]: Module Net::DNS            0.48
Feb 21 09:32:10 web02-gti amavis[1653]: Module Net::SMTP           2.29
Feb 21 09:32:10 web02-gti amavis[1653]: Module Net::Server         0.87
Feb 21 09:32:10 web02-gti amavis[1653]: Module Razor2::Client::Version 2.67
Feb 21 09:32:10 web02-gti amavis[1653]: Module Time::HiRes         1.59
Feb 21 09:32:10 web02-gti amavis[1653]: Module Unix::Syslog        0.100
Feb 21 09:32:10 web02-gti amavis[1653]: Amavis::DB code    loaded
Feb 21 09:32:10 web02-gti amavis[1653]: Amavis::Cache code loaded
Feb 21 09:32:10 web02-gti amavis[1653]: SQL base code      NOT loaded
Feb 21 09:32:10 web02-gti amavis[1653]: SQL::Log code      NOT loaded
Feb 21 09:32:10 web02-gti amavis[1653]: SQL::Quarantine    NOT loaded
Feb 21 09:32:10 web02-gti amavis[1653]: Lookup::SQL  code  NOT loaded
Feb 21 09:32:10 web02-gti amavis[1653]: Lookup::LDAP code  NOT loaded
Feb 21 09:32:10 web02-gti amavis[1653]: AM.PDP prot  code  NOT loaded
Feb 21 09:32:10 web02-gti amavis[1653]: SMTP-in prot code  loaded
Feb 21 09:32:10 web02-gti amavis[1653]: ANTI-VIRUS code    loaded
Feb 21 09:32:10 web02-gti amavis[1653]: ANTI-SPAM  code    loaded
Feb 21 09:32:10 web02-gti amavis[1653]: Unpackers  code    loaded
Feb 21 09:32:10 web02-gti amavis[1653]: Found $file            at /usr/bin/file
Feb 21 09:32:10 web02-gti amavis[1653]: No $dspam,             not using it
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .mail
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .asc
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .uue
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .hqx
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .ync
Feb 21 09:32:10 web02-gti amavis[1653]: No decoder for       .F    tried: unfreeze, freeze -d, melt, fcat
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .Z    at /bin/uncompress
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .gz
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .bz2  at /usr/bin/bzip2 -d
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .lzo  at /usr/bin/lzop -d
Feb 21 09:32:10 web02-gti amavis[1653]: No decoder for       .rpm  tried: rpm2cpio.pl, rpm2cpio
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .cpio at /usr/bin/pax
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .tar  at /usr/bin/pax
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .deb  at /usr/bin/ar
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .zip
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .rar  at /usr/bin/unrar
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .arj  at /usr/bin/arj
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .arc  at /usr/bin/nomarch
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .zoo  at /usr/bin/zoo
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .lha  at /usr/bin/lha
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .cab  at /usr/bin/cabextract
Feb 21 09:32:10 web02-gti amavis[1653]: No decoder for       .tnef tried: tnef
Feb 21 09:32:10 web02-gti amavis[1653]: Internal decoder for .tnef
Feb 21 09:32:10 web02-gti amavis[1653]: Found decoder for    .exe  at /usr/bin/unrar; /usr/bin/lha; /usr/bin/arj
Feb 21 09:32:10 web02-gti amavis[1653]: Using internal av scanner code for (primary) ClamAV-clamd
Feb 21 09:32:10 web02-gti amavis[1653]: Found primary av scanner BitDefender at /usr/bin/bdc
Feb 21 09:32:10 web02-gti amavis[1653]: Found secondary av scanner ClamAV-clamscan at /usr/bin/clamscan
Feb 21 09:32:10 web02-gti amavis[1653]: Creating db in /var/lib/amavis/db/; BerkeleyDB 0.26, libdb 4.3


Email test :
- this is the debug for amavisd-new --> http://www.gtinnovatech.com/amavisdnew_debug.txt

- this is the tail -f /var/log/mail.log -->
http://www.gtinnovatech.com/tail_maillog.txt
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Tue Feb 21, 2006 11:33 am

You should not be starting spamd. It needs to be disabled.
vi /etc/default/spamassassin

and set:
ENABLED=0

then reboot. I'm curious, did you enable this at some point? AFAIK it is not enabled by default, so if you did enable it, I am curious why you would.

Before you reboot, edit:
vi /var/dcc/whitecommon
and comment out the line with optimus22.ietf.org on it (line 33). This will get rid of another one of the error messages.

I don't see where amavisd-new is logging, possibly because you are running it in debug mode (or because spamd is running). Since amavisd-new appears to be OK now, after it reboots it should start normally. Once it reboots, once again do 'tail -f /var/log/mail.log' and send a single test message through, and show the results. This time it should show the one amavis line. If it does, try sending the eicar.txt sample through. If you would like, use 'top' and enter a '>' to sort by program size to see if amavisd-new is running. You should see something like this near the top:

Code: Select all
6944 amavis    18   0 56660  47m  11m S  0.0 38.3   0:43.49 amavisd-new
6943 amavis    16   0 56588  47m  11m S  0.0 38.2   0:10.98 amavisd-new
6682 amavis    16   0 55416  46m  10m S  0.0 37.3   0:16.16 amavisd-new
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Tue Feb 21, 2006 11:54 am

Oops. I thought it was an error from me that spamd was disabled.
I enabled it because at start it says :
"SpamAssassin Mail Filter Deamon: disabled, see /etc/default/spamassassin"

"top" shows effectively amavisd-new running.

and a tail mail.log gives that :

Code: Select all
web02-gti:~# tail -f /var/log/mail.log
Feb 21 10:40:03 web02-gti amavis[1653]: Creating db in /var/lib/amavis/db/; BerkeleyDB 0.26, libdb 4.3
Feb 21 10:40:08 web02-gti dccifd[1710]: 1.3.29 listening to /var/dcc/dccifd
Feb 21 10:40:13 web02-gti postfix/postfix-script: warning: /var/spool/postfix/etc/passwd and /etc/passwd differ
Feb 21 10:40:13 web02-gti postfix/postfix-script: starting the Postfix mail system
Feb 21 10:40:13 web02-gti postfix/master[1824]: daemon started -- version 2.1.5

//// --------- i've installed logcheck, so an email is sent to logcheck & to root :
Feb 21 10:40:28 web02-gti postfix/pickup[1842]: ECD2A56AA6: uid=108 from=<logcheck>
Feb 21 10:40:29 web02-gti postfix/cleanup[2661]: ECD2A56AA6: message-id=<20060221154028.ECD2A56AA6@web02-gti.domain1.com>
Feb 21 10:40:29 web02-gti postfix/qmgr[1843]: ECD2A56AA6: from=<logcheck@domain1.com>, size=22685, nrcpt=1 (queue active)
Feb 21 10:40:29 web02-gti postfix/smtp[2665]: ECD2A56AA6: to=<root@domain1.com>, orig_to=<root>, relay=192.168.68.2[192.168.68.2], delay=1, status=sent (250 2.6.0  <20060221154028.ECD2A56AA6@web02-gti.domain1.com> Queued mail for delivery)
Feb 21 10:40:29 web02-gti postfix/qmgr[1843]: ECD2A56AA6: removed

//// --------- i've redirected the port 25 on my spamfilterbox :
Feb 21 10:44:52 web02-gti postfix/smtpd[2705]: connect from c-67-172-26-250.hsd1.pa.comcast.net[67.172.26.250]
Feb 21 10:45:17 web02-gti postfix/smtpd[2707]: connect from c-69-139-156-251.hsd1.md.comcast.net[69.139.156.251]
Feb 21 10:45:20 web02-gti postfix/smtpd[2708]: connect from unknown[219.128.61.241]
Feb 21 10:45:21 web02-gti postfix/smtpd[2707]: A61E756AA6: client=c-69-139-156-251.hsd1.md.comcast.net[69.139.156.251]
Feb 21 10:45:21 web02-gti postfix/cleanup[2709]: A61E756AA6: message-id=<c7eb47ac27f.c7eb47ac27fc277705fd4@mastercard.com>
Feb 21 10:45:21 web02-gti postfix/qmgr[1843]: A61E756AA6: from=<ikrijgzexq@mastercard.com>, size=1372, nrcpt=1 (queue active)
Feb 21 10:45:21 web02-gti postfix/smtpd[2707]: disconnect from c-69-139-156-251.hsd1.md.comcast.net[69.139.156.251]
Feb 21 10:45:25 web02-gti postfix/smtpd[2705]: NOQUEUE: reject: RCPT from c-67-172-26-250.hsd1.pa.comcast.net[67.172.26.250]: 450 <stoffele@bleuchannel.com>: Sender address rejected: Domain not found; from=<stoffele@bleuchannel.com> to=<spicyb@domain1.com> proto=SMTP helo=<bleuchannel.com>
Feb 21 10:45:25 web02-gti postfix/smtpd[2705]: lost connection after RCPT from c-67-172-26-250.hsd1.pa.comcast.net[67.172.26.250]
Feb 21 10:45:25 web02-gti postfix/smtpd[2705]: disconnect from c-67-172-26-250.hsd1.pa.comcast.net[67.172.26.250]
Feb 21 10:45:36 web02-gti postfix/smtpd[2708]: disconnect from unknown[219.128.61.241]
Feb 21 10:45:37 web02-gti dccifd[1710]: stat(log directory /var/dcc/log): No such file or directory
Feb 21 10:45:39 web02-gti postfix/smtpd[2716]: connect from localhost.localdomain[127.0.0.1]
Feb 21 10:45:39 web02-gti postfix/smtpd[2716]: 4262356A99: client=localhost.localdomain[127.0.0.1]
Feb 21 10:45:39 web02-gti postfix/cleanup[2709]: 4262356A99: message-id=<c7eb47ac27f.c7eb47ac27fc277705fd4@mastercard.com>
Feb 21 10:45:39 web02-gti postfix/qmgr[1843]: 4262356A99: from=<ikrijgzexq@mastercard.com>, size=2520, nrcpt=1 (queue active)
Feb 21 10:45:39 web02-gti postfix/smtpd[2716]: disconnect from localhost.localdomain[127.0.0.1]
Feb 21 10:45:39 web02-gti amavis[1681]: (01681-01) Passed SPAM, [69.139.156.251] [69.139.156.251] <ikrijgzexq@mastercard.com> -> <yasso@domain1.com>, Message-ID: <c7eb47ac27f.c7eb47ac27fc277705fd4@mastercard.com>, mail_id: ivxWt5zuXbcb, Hits: 22.903, 17578 ms
Feb 21 10:45:39 web02-gti postfix/smtp[2710]: A61E756AA6: to=<yasso@domain1.com>, relay=127.0.0.1[127.0.0.1], delay=22, status=sent (250 2.6.0 Ok, id=01681-01, from MTA([127.0.0.1]:10025): 250 Ok: queued as 4262356A99)
Feb 21 10:45:39 web02-gti postfix/qmgr[1843]: A61E756AA6: removed
Feb 21 10:45:39 web02-gti postfix/smtp[2717]: 4262356A99: to=<yasso@domain1.com>, relay=192.168.68.2[192.168.68.2], delay=0, status=bounced (host 192.168.68.2[192.168.68.2] said: 550 5.1.1 User unknown (in reply to RCPT TO command))
Feb 21 10:45:39 web02-gti postfix/cleanup[2709]: 89CC156AAA: message-id=<20060221154539.89CC156AAA@web02-gti.domain1.com>
Feb 21 10:45:39 web02-gti postfix/qmgr[1843]: 89CC156AAA: from=<>, size=4410, nrcpt=1 (queue active)
Feb 21 10:45:39 web02-gti postfix/qmgr[1843]: 4262356A99: removed
Feb 21 10:45:39 web02-gti postfix/smtp[2717]: 89CC156AAA: to=<ikrijgzexq@mastercard.com>, relay=192.168.68.2[192.168.68.2], delay=0, status=bounced (host 192.168.68.2[192.168.68.2] said: 550 5.7.1 Unable to relay for ikrijgzexq@mastercard.com (in reply to RCPT TO command))
Feb 21 10:45:39 web02-gti postfix/qmgr[1843]: 89CC156AAA: removed


I'm sorry for the "single" test, but as soon as I redirect the 25 port on my spamfilter box, a huge packet of spam comes into .... I've tried to limit the size of this output.
But as you can see, not all the spams may be filtered.
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Tue Feb 21, 2006 12:12 pm

Code: Select all
Feb 21 10:45:39 web02-gti amavis[1681]: (01681-01) Passed SPAM, [69.139.156.251] [69.139.156.251] <ikrijgzexq@mastercard.com> -> <yasso@domain1.com>, Message-ID: <c7eb47ac27f.c7eb47ac27fc277705fd4@mastercard.com>, mail_id: ivxWt5zuXbcb, Hits: 22.903, 17578 ms


At least I see this amavis entry now. I think you are functioning correctly this point. I would try sending the eicar virus. I think you will find that it does get stopped.

You can simply grep the log file to see if it worked:
grep INFECTED /var/log/mail.log

You should also get a notification sent to postmaster.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby mr88talent » Tue Feb 21, 2006 12:17 pm

Code: Select all
Feb 21 10:40:13 web02-gti postfix/postfix-script: warning: /var/spool/postfix/etc/passwd and /etc/passwd differ

also, run:
LINUX2
to fix this.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Tue Feb 21, 2006 12:29 pm

Perfect. It well works !

Just a small question : To receive all spam notification, I've to uncomment $spam_admin "postmaster\@$mydomain"; , that's it ?
In a later time, I'm going searching why not all spam are filtered.

But I thank you very much for your help.
I appreciate enormously what you've done !
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Tue Feb 21, 2006 12:48 pm

Yes, this will send a detailed report to you for each spam, but it will not send a report when the mail is not spam, so I don't think it will help you figure out why a message is not spam. However, the X-Spam headers are useful in both cases.

You can also get a more detailed report in the header, but again, this detail will only show up in spam, not ham:

$sa_spam_report_header = 1;

This link shows how you can have all mail get the detailed header report:
http://marc.theaimsgroup.com/?l=amavis-user&m=113985038223442&w=2

As the link says, you can also lower $sa_tag2_level_deflt during testing. This is one way to get the spam report sent to the admin, but of course you will be marking ham as spam, so this should not be used on a normal basis.

Also, when you get some time, read:
http://www200.pair.com/mecham/spam/addi ... tings.html

Your English is fine...
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby Netopia » Tue Feb 21, 2006 1:16 pm

WOW....

I've never run Debian and don't know the intricacies of the differences... but when I read a thread like this one, something inside me wants to put you up for a national medal or something!

Mr88Talent, you are truly a bleesing on this board, and I bet others too!

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Tue Feb 21, 2006 1:24 pm

mr88talent => thanks again. For the additional settings, it will be really welcomed. yep !

Joe =>which distrib do you use ?
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Tue Feb 21, 2006 1:28 pm

Fedora Core 4

I've been playing (litterally... just playing around) with Redhat since about Version 4.2 (before Fedora Core existed) so I feel a bit more familiar with it. In the presense of people like Mr88Talent though... I feel like a new born who doesn't even know how to crawl yet! :D

Est-elle française votre langue principale?

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby mr88talent » Tue Feb 21, 2006 1:32 pm

You are much too kind :)
I am simply much too familiar with my doc. I've been maintaining it for over two years now. It's a good hobby. I appreciate it when people tell me they had a problem. Of course it helps me tweak the document.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Tue Feb 21, 2006 5:48 pm

Joe => Yes i'm a canadian "made in france" :D
Personnaly, I've always "played" in mandrake/mandriva then I've got like a linux base. But this howto has also shown me how to manipulate a debian.

mr88talent => Is there any way to say to logcheck to send me his report only once a day ? (because actually, it's each hour ... And I've found no entry in crontab, and no time configuration in /etc/logcheck/logcheck.conf)
In any case, the custom rules file you use (http://www.rulesemporium.com/rules/mangled.cf ) reduces more appreciably the number of spam. That's nice ! :)
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Tue Feb 21, 2006 5:54 pm

I've not posted a "what it's done for me" yet, but with the FC4 setup (which I'm going to guess is at least pretty similar in function to the Debian) and with the attition of RulesDuJour and some extra RBL's. my spam has at this point dropped to (VRAIMENT!) litterally ZERO.

One user who usually has had 700-1000 spams waiting for him on Monday mornings had a whopping SIX and they were all tagged!

:shock: :D :shock: :D :shock: :D :D :shock:

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Tue Feb 21, 2006 6:00 pm

Wouahou ... and do you use the header_checks & body_checks of postfix presented in this section --> http://www200.pair.com/mecham/spam/spamfilter20050626.html#antispam ?

Because I'm affraid that these two options are dangerous because there's no way to log the spam they wipe .... yerk ...
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Tue Feb 21, 2006 6:20 pm

Yes, you can edit:
/etc/cron.d/logcheck

But instead I would recommend making entries to have it ignore messages that are not important.

If it is a "System Events" message, you would add entries to the proper program in:
/etc/logcheck/ignore.d.server/
I tend to place most stuff in:
/etc/logcheck/ignore.d.server/postfix
(I'm not sure it matters if the log entry was actually created by Postfix, the different files are a way to keep things organized).

If it is a "Security Events" message, you need to add it to a file in:
/etc/logcheck/violations.ignore.d/
I usually add stuff to:
/etc/logcheck/violations.ignore.d/logcheck-postfix

I have been unable to have it ignore the text string 'attack', but I have not played with it that much. If a log entry contains any of the keywords located in a file in cracking.d or violations.d then it will be logged as a "Security Events".

Heh! That's it. The keyword 'attack' is in /etc/logcheck/cracking.d/logcheck
so I need to create a new 'logcheck-postfix' file in the cracking.ignore.d directory and put text like 'attackingthedevil.co.uk' in it.

You don't have to place full regex expressions in these files like the existing sample entries, just make sure the text you enter is specific enough.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby mr88talent » Tue Feb 21, 2006 6:22 pm

BTW, I set my $log_level = 0; in amavisd.conf to avoid a lot of noise from amavisd-new.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby mr88talent » Tue Feb 21, 2006 6:28 pm

Code: Select all
echo "attackingthedevil.co.uk" >> /etc/logcheck/cracking.ignore.d/logcheck-postfix
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby Netopia » Tue Feb 21, 2006 6:37 pm

I don't know what the different settings are for Debian, but here's stuff from my main.cf



Code: Select all
smtpd_client_restrictions =
        check_client_access hash:/etc/postfix/client_access,
        reject_rbl_client bl.spamcop.net,
        reject_rbl_client sbl.spamhaus.org,
        reject_rbl_client dnsbl.njabl.org,
        reject_rbl_client spam.dnsrbl.org
        reject_rbl_client relays.ordb.org,
        reject_rbl_client opm.blitzed.org,
        reject_rbl_client list.dsbl.org,
        reject_rbl_client cbl.abuseat.org,
        reject_rbl_client dul.dnsbl.sorbs.net,
        #reject_rbl_client dun.dnsrbl.net,
        reject_rbl_client vox.schpider.com

smtpd_helo_restrictions =
        check_helo_access hash:/etc/postfix/helo_access,
        reject_invalid_hostname,
         #reject_non_fqdn_hostname       

smtpd_sender_restrictions =
        check_sender_access hash:/etc/postfix/sender_access,
        reject_non_fqdn_sender,
        reject_unknown_sender_domain,
        reject_rhsbl_sender relays.ordb.org,
        reject_rhsbl_sender opm.blitzed.org,
        #reject_rhsbl_sender dun.dnsrbl.net

smtpd_recipient_restrictions =
        reject_non_fqdn_recipient,
        reject_unknown_recipient_domain,
        permit_mynetworks,
        reject_unauth_destination,
        check_recipient_access hash:/etc/postfix/recipient_access,
        reject_unverified_recipient

smtpd_data_restrictions = reject_unauth_pipelining


I had to comment out dun.dnsrbl.net and non_fqdn_hostname because they were both causing too many problems with rejecting mail we needed.

In my main.cf, I have it set to mark spam at 4.5 and to quarantine it at 8 (it will still send out messages to the sender up to 15... I've got to be careful for a while until I'm sure of what's going on).

I've hand looked through almost 3,000 quarantined emails since Saturday and NONE have been false positives.

I also "prelearned" my SA with about 8,000 or so ham and spam before going live. For the ham, I just raided people's email at work and copied large volumes.

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby Netopia » Tue Feb 21, 2006 6:38 pm

BTW.... by FAR, the two services that seem to catch the most email per day for me are SpamCop and Spamhaus.

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby mr88talent » Tue Feb 21, 2006 6:49 pm

As far as header_checks and body_checks go, they are very expensive. They are not exactly dangerous if you REJECT the mail. REJECT-ing mail (550) does at least provide feedback to the sender. They are expensive because every line in the header is compared to every expression in the header_checks file, and every line in the body is compared to every expression in the body_checks file. Generally speaking, it is my opinion you should only use them when there is no alternative. I have on occasion used body_checks to search for some text I find in some images.

Code: Select all
/2ODxB5okIUkbKILBgXFLknxBrnR0qUORSgtxnB1Jd/ REJECT You tried to send an offensive picture
Note that the text I chose does not contain any special regex characters and I chose enough to make it unique. If you choose too much it won't work. This does get logged and is noted in the daily pflogsumm report. This one worked for about two weeks.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby Netopia » Tue Feb 21, 2006 8:18 pm

Man... that seems a little extreme! I don't mean scanning for objectionable material, but the resources needed to do so per piece of mail!

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby mr88talent » Tue Feb 21, 2006 9:44 pm

I have read that simply enabling content filtering (even with empty body_checks/header_checks files) will reduce Postfix throughput by 40%. However, even at that, Postfix throughput will still be 10 times greater than amavisd-new/spamassassin. So, if a good header_check or body_check is used to reject a message that amavisd-new would have had to process, then it might be processing power well spent. On the other hand, if it is possible to live without them, these checks should be commented out in main.cf, or at least stale entries removed from the files.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby Netopia » Wed Feb 22, 2006 12:58 pm

I had to comment out dun.dnsrbl.net ...


Issue solved. STUPID ME! Turns out that "dun.dnsrbl.net" is no longer running, which accounts for why I got so mainy failures to it. Why it didn't fail on EVERY SINGLE email, I don't know... but at least I know why it failed at all.

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Wed Feb 22, 2006 1:31 pm

oh my god :?

After 1 day of running, there's a major problem :
the mail queue ! argh ...

Some emails are jammed into this queue even if they are ham, and specially even if the expeditor is in the whitelist (/etc/spamassassin/local.cf --> whitelist_from *@my_client1.com etc ... )

qshape shows me the number of these email decreasing very slowly.
and mailq shows me these email waiting for distribution ...

what I've done again ? :? :shock:
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Wed Feb 22, 2006 1:43 pm

I don't know what's going on with your system... because I myself and new to this. I do want to tell you not to give up hope though. I had a couple of things with mine that I needed to fix and now it's running like a race horse and devouring spam like it was a tray of hors devours!

Stick with it...

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby mr88talent » Wed Feb 22, 2006 1:48 pm

What are some of the reasons given for the mail being deferred? Are these mails that your next-hop server has bounced or rejected, or is this mail waiting to be delivered to your internal mail server? In other words, is the mail waiting to get in, or waiting to get out? Is a lot of this mail addressed to invalid users? Privately email me some of the mailq output, send it to:

lists at johnmecham dot com
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby mr88talent » Wed Feb 22, 2006 2:04 pm

let me see system load. Do 'top' and '>' and show me this much:

Code: Select all
top - 11:02:33 up 10 days, 45 min,  1 user,  load average: 1.02, 0.28, 0.15
Tasks:  78 total,   4 running,  74 sleeping,   0 stopped,   0 zombie
Cpu(s):  47.7% user,  13.7% system,  38.6% nice,   0.0% idle
Mem:    516424k total,   491512k used,    24912k free,    85056k buffers
Swap:   963860k total,     5820k used,   958040k free,   195688k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26657 amavis     9   0 51848  50m  30m S  0.0 10.0   0:07.99 amavisd-new
26710 amavis     9   0 51780  50m  30m S  0.0 10.0   0:07.18 amavisd-new
26665 amavis     9   0 48976  47m  30m S  0.0  9.5   0:03.23 amavisd-new
29371 amavis     9   0 47536  46m  30m S  0.0  9.2   0:04.39 amavisd-new
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Wed Feb 22, 2006 2:08 pm

I must confess you that I've just been installing apache2, so the process are on top.

Code: Select all
top - 13:07:42 up  4:06,  2 users,  load average: 1.53, 1.58, 1.79
Tasks:  66 total,   3 running,  63 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.7% us,  0.3% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    516408k total,   446572k used,    69836k free,     8852k buffers
Swap:   594364k total,   237752k used,   356612k free,    23608k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
5340 root      16   0 14884 1432  12m S  0.0  0.3   0:00.18 apache2
5904 www-data  16   0 14888 1416  12m S  0.0  0.3   0:00.01 apache2
5905 www-data  16   0 14884 1412  12m S  0.0  0.3   0:00.00 apache2
5906 www-data  16   0 14884 1412  12m S  0.0  0.3   0:00.00 apache2
5907 www-data  16   0 14884 1412  12m S  0.0  0.3   0:00.00 apache2
5908 www-data  16   0 14884 1412  12m S  0.0  0.3   0:00.00 apache2
5909 www-data  17   0 14884 1412  12m S  0.0  0.3   0:00.00 apache2
5910 www-data  17   0 14884 1412  12m S  0.0  0.3   0:00.00 apache2
8774 amavis    25   0  264m 160m  11m R 49.7 31.8   3:21.14 amavisd-new
8929 amavis    25   0  265m 161m  11m R 49.1 31.9   2:22.63 amavisd-new
1655 amavis    16   0  262m  48m  10m S  0.0  9.6   2:39.14 amavisd-new
4695 mysql     16   0  115m 1888 8920 S  0.0  0.4   0:00.80 mysqld
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Wed Feb 22, 2006 2:28 pm

You amavisd-new child processes are absolutely hugh. You have also dipped into swap space quite a bit. Amavisd-new runs very slowly if it has to use swap space. I would look through the mail.log and see if you find evidence of amavisd-new timing out. Most likely amavisd-new is timing out and the mail is deferred due to this.

Code: Select all
8774 amavis    25   0  264m 160m  11m R 49.7 31.8   3:21.14 amavisd-new
8929 amavis    25   0  265m 161m  11m R 49.1 31.9   2:22.63 amavisd-new


This may be from too many SARE RDJ rule sets. Please show output from:
ls -l /etc/spamassassin

I want to see what you are using, then:
move the RDJ rule sets out of the /etc/spamassassin directory (make a directory like /etc/rdj and move them there). Then 'amavisd-new reload'.

Then show me 'top' again.

The next thing I would be concerned with is rejecting mail to invalid users. It is dictionary attacks that will kill any server that does not reject mail to invalid users. I would work on getting every single one of your valid email addresses into the relay_recipients file as outlined in the doc.

I also suggest getting policyd-weight installed as per the separate document. This will help tone things down quite a bit.

Once we get the size of amavisd-new down, we will take another look at how much ram is available.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Wed Feb 22, 2006 2:29 pm

I've stoped the apache2 service.
The queue seems like to reduce...

does that mean that I can't have apache running on this spambox ?
(pII 350, 256Mo RAM, 1 x 20Go SCSI)
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Wed Feb 22, 2006 2:35 pm

Better than 'amavisd-new reload', we need to make sure amavisd-new has stopped all of its processes, so do:
amavisd-new stop
then:
ps aux | grep amavisd
make sure no processes of amavisd are still running, if they are, run:
killall amavisd-new

then amavisd-new start
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Wed Feb 22, 2006 2:38 pm

too much RDJ rules ? possible :?

Code: Select all
web02-gti:~# ls -l /etc/spamassassin
total 19132
-rw-r--r--  1 root root    31854 2005-06-01 20:00 70_sare_adult.cf
-rw-r--r--  1 root root     3839 2005-06-01 20:00 70_sare_bayes_poison_nxm.cf
-rw-r--r--  1 root root    24298 2005-10-05 16:00 70_sare_evilnum0.cf
-rw-r--r--  1 root root     1574 2005-06-01 20:00 70_sare_evilnum1.cf
-rw-r--r--  1 root root     6970 2005-06-01 20:00 70_sare_evilnum2.cf
-rw-r--r--  1 root root    45933 2005-12-26 19:00 70_sare_genlsubj0.cf
-rw-r--r--  1 root root    75052 2005-12-26 19:00 70_sare_genlsubj1.cf
-rw-r--r--  1 root root    17533 2005-12-26 19:00 70_sare_genlsubj2.cf
-rw-r--r--  1 root root    49125 2005-12-26 19:00 70_sare_genlsubj3.cf
-rw-r--r--  1 root root   187643 2005-12-26 19:00 70_sare_genlsubj.cf
-rw-r--r--  1 root root    31990 2005-12-26 19:00 70_sare_genlsubj_eng.cf
-rw-r--r--  1 root root     4295 2005-12-26 19:00 70_sare_genlsubj_x30.cf
-rw-r--r--  1 root root   118966 2005-11-30 07:00 70_sare_header0.cf
-rw-r--r--  1 root root   136590 2005-10-29 20:00 70_sare_header1.cf
-rw-r--r--  1 root root    69870 2005-10-29 20:00 70_sare_header2.cf
-rw-r--r--  1 root root    59996 2005-10-30 06:00 70_sare_header3.cf
-rw-r--r--  1 root root   384645 2005-10-30 06:00 70_sare_header.cf
-rw-r--r--  1 root root     5898 2005-10-29 20:00 70_sare_header_eng.cf
-rw-r--r--  1 root root     6422 2005-10-29 20:00 70_sare_header_x30.cf
-rw-r--r--  1 root root     4448 2005-06-01 20:00 70_sare_highrisk.cf
-rw-r--r--  1 root root    32286 2005-10-05 19:00 70_sare_html0.cf
-rw-r--r--  1 root root    40967 2005-10-05 19:00 70_sare_html1.cf
-rw-r--r--  1 root root     7866 2005-10-05 19:00 70_sare_html2.cf
-rw-r--r--  1 root root    14160 2005-10-05 19:00 70_sare_html3.cf
-rw-r--r--  1 root root    42182 2005-10-05 19:00 70_sare_html4.cf
-rw-r--r--  1 root root    95279 2005-10-05 19:00 70_sare_html.cf
-rw-r--r--  1 root root     3161 2005-10-05 19:00 70_sare_html_eng.cf
-rw-r--r--  1 root root     2780 2005-10-25 17:00 70_sare_html_x30.cf
-rw-r--r--  1 root root    51886 2005-10-01 16:00 70_sare_obfu0.cf
-rw-r--r--  1 root root   106627 2005-10-01 16:00 70_sare_obfu1.cf
-rw-r--r--  1 root root     6129 2005-10-01 16:00 70_sare_obfu2.cf
-rw-r--r--  1 root root    13766 2005-10-01 16:00 70_sare_obfu3.cf
-rw-r--r--  1 root root   158513 2005-10-01 16:00 70_sare_obfu.cf
-rw-r--r--  1 root root    12739 2005-12-27 07:00 70_sare_oem.cf
-rw-r--r--  1 root root    18190 2005-12-12 05:00 70_sare_random.cf
-rw-r--r--  1 root root    89030 2005-11-25 12:00 70_sare_specific.cf
-rw-r--r--  1 root root    18590 2005-12-13 05:00 70_sare_spoof.cf
-rw-r--r--  1 root root    25124 2005-11-12 05:00 70_sare_unsub.cf
-rw-r--r--  1 root root    17879 2005-10-04 18:00 70_sare_uri0.cf
-rw-r--r--  1 root root    24248 2005-10-10 18:00 70_sare_uri1.cf
-rw-r--r--  1 root root     8502 2005-10-04 18:00 70_sare_uri3.cf
-rw-r--r--  1 root root     5053 2005-10-04 18:00 70_sare_uri_eng.cf
-rw-r--r--  1 root root     8273 2006-02-16 08:00 70_sc_top200.cf
-rw-r--r--  1 root root    17992 2005-06-01 20:00 71_sare_redirect_pre3.0.0.cf
-rw-r--r--  1 root root    13211 2005-06-01 20:00 72_sare_bml_post25x.cf
-rw-r--r--  1 root root    15311 2005-06-01 20:00 72_sare_redirect_post3.0.0.cf
-rw-r--r--  1 root root    10147 2005-06-01 20:00 99_sare_fraud_post25x.cf
-rw-r--r--  1 root root    14284 2004-04-28 13:22 LoserLamer.cf
-rw-r--r--  1 root root 13208253 2006-02-21 13:28 blacklist.cf
-rw-r--r--  1 root root  3800183 2006-02-21 13:36 blacklist-uri.cf
-rw-r--r--  1 root root   110046 2005-12-15 02:00 bogus-virus-warnings.cf
-rw-r--r--  1 root root      935 2005-06-29 20:05 init.pre
-rw-r--r--  1 root root     1873 2006-02-21 11:40 local.cf
-rw-r--r--  1 root root    58203 2005-06-01 20:00 mangled.cf
-rw-r--r--  1 root root     4883 2004-05-25 12:03 random.cf
-rw-r--r--  1 root root    56238 2005-06-01 20:00 tripwire.cf



after moving all the rules (excepted the local.cf) it gives :

Code: Select all
top - 13:36:44 up  4:35,  2 users,  load average: 1.17, 1.78, 1.81
Tasks:  70 total,   2 running,  68 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0% us,  0.7% sy,  0.0% ni, 97.0% id,  1.3% wa,  0.0% hi,  0.0% si
Mem:    516408k total,   119760k used,   396648k free,     4732k buffers
Swap:   594364k total,    28320k used,   566044k free,    38044k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9874 amavis    17   0 50064  43m  12m S  0.0  8.6   0:00.02 amavisd-new
9875 amavis    18   0 50064  43m  12m S  0.0  8.6   0:00.01 amavisd-new
9871 amavis    16   0 49424  43m  11m S  0.0  8.6   0:01.90 amavisd-new
4695 mysql     16   0  115m 1888 8920 S  0.0  0.4   0:00.80 mysqld



it seems like to be ok
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Wed Feb 22, 2006 2:48 pm

You have to put init.pre back, then restart amavisd-new again.

These two will kill your system (as you noticed):
-rw-r--r-- 1 root root 13208253 2006-02-21 13:28 blacklist.cf
-rw-r--r-- 1 root root 3800183 2006-02-21 13:36 blacklist-uri.cf
They are not necessary either, spamassassin uses other means to get the same information. NEVER USE THESE, or bigevel.cf.

I would spend some good quality time deciding which rule sets to keep. Only use the ones that you know are going to solve a particular problem.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Wed Feb 22, 2006 3:08 pm

an other problem coming from the amavisd-new restarting (yeepee) :
all the mails waiting in the queue (listed by mailq) may not be deliverable :

Code: Select all
1D45F56AED     4060 Wed Feb 22 13:59:12  hiettgirola@gbrooks.com
(delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]: Connection refused)
                                         steve.messner@domain1.com


I've tried "sendmail -q -v" in vain.
In mail.log, I can read what follows for all mails waiting, as soon as I try "sendmail -q -v" :

Code: Select all
Feb 22 14:02:39 web02-gti postfix/qmgr[1840]: 1927756AFD: to=<murriel@domain1.com>, relay=none, delay=1268, status=deferred (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]: Connection refused)


qshape is empty, but mailq grows up ...
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Wed Feb 22, 2006 3:27 pm

I use 'postfix flush' to flush the queue.
You want to use 'qshape deferred' to see deferred mail.
Feb 22 14:02:39 web02-gti postfix/qmgr[1840]: 1927756AFD: to=<murriel@domain1.com>, relay=none, delay=1268, status=deferred (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]: Connection refused)
This error will occur if amavisd-new is not running. Make sure it is.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Wed Feb 22, 2006 3:34 pm

amazing !
how do you do ? are you into my computer ? :D
amavisd-new wasn't running and I don't know why and how ...
last time I've touched amavisd-new process it was to reload it.
so weird ...
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Wed Feb 22, 2006 3:36 pm

Once amavisd-new is running, monitor with:
tail -f /var/log/mail.log
and see how things are going.
If things start flowing again, then you can 'postfix flush', but only do it once(if at all), because it actually can make matters worse if everything is not working properly. It might take a few minutes before the mail starts to get re-queued.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Wed Feb 22, 2006 3:40 pm

It works perfectly
The mail queue is emptying normally.
pfiouuu ... a big useless fear :oops:
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Wed Feb 22, 2006 3:53 pm

Mr88Talent IS the man? N'est-il pas? :D:D:D

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Wed Feb 22, 2006 4:02 pm

Yes ! He is !!!
Thank you mr88talent very much tone of lot (I don't know if I can say that but I say it nevertheless :wink: )
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby vmaillot » Mon Feb 27, 2006 11:16 am

Ok, that might be written in an other post, but as it's my story, I would continue it here :D

One week after the beginning of the wonderfull story of my spambox, I just have to say : waou !
It's so powerfull ! I would like to thanks again mr88talent for his great help :)

While I'm here, I just ask a tiny question : Why my logcheck daily summary says that the spambox delivered more email than received ?
(precision : it's just an ingoing filter)

Code: Select all
   5328   received
   5383   delivered
      0   forwarded
      0   deferred
   7453   bounced
    185   rejected (3%)
      0   reject warnings
      0   held
      0   discarded (0%)
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Mon Feb 27, 2006 12:27 pm

I have not spent any time trying to figure that out, but mine does the same thing. Maybe something to do with multiple-recipient mail.

I am concerned with:

7453 bounced
185 rejected (3%)

Are you finding many of these bounced messages in the deferred queue? Are they mail addressed to invalid users that your down-stream server rejected? If so, you need to either populate your relay_recipients map with every single valid email address (the preferred method) or possibly look into using reject_unverified_recipient as an alternative. If you use reject_unverified_recipient you might want to use a static cache like:
address_verify_map = btree:/etc/postfix/verify

See:
http://www.postfix.org/ADDRESS_VERIFICATION_README.html

You should be rejecting a lot more mail, and bouncing a lot less mail.
If you use reject_unverified_recipient, make sure you understand how how it functions. You don't want to tie up your server with bounces that can never be delivered because the sender is bogus. It's a waste.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby Netopia » Mon Feb 27, 2006 12:31 pm

I don't get as much email as you do, so I'm sending my weekly numbers. The questions that I have are:

Are you using pfloggsum.pl and have you implemented the changes that I posted about (url=http://freespamfilter.org/forum/viewtopic.php?t=208]Here[/url]

and why do you have so many bounced and I have so many rejected? It looks like our machines are getting rid of spam in two different ways, but I don't know enough about how the various systems work to know why.

Code: Select all
10150   received
   9897   delivered
      0   forwarded
    130   deferred  (699  deferrals)
     47   bounced
  15842   rejected (76%)
      0   reject warnings
      0   held
      0   discarded (0%)


Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby Netopia » Mon Feb 27, 2006 12:32 pm

DOH!

Mr88 answered while I got called away as I was starting to write my last post!

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby mr88talent » Mon Feb 27, 2006 12:33 pm

1498 received
1739 delivered
0 forwarded
0 deferred
0 bounced
5324 rejected (75%)
0 reject warnings
0 held
0 discarded (0%)
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby Netopia » Mon Feb 27, 2006 12:44 pm

Is the difference you guys have with more delivered than received that the server is generating emails so they are delivered to the end user but never received from outside?

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Mon Feb 27, 2006 1:07 pm

Joe you may be right.
- 1 mail per hour to send the logcheck report.
delivered = +1 / hour = +24 / day

- 1 mail per day for mailq and 1 for qshape
delivered = +2 / day

- 1 mail for each rules/antivirus update.
delivered = + ... / day (it depends)

So, I've got 26 mails minimum added to the received.


mr88talent =>
you wrote 5324 rejected (75%) , is that with your "address_verify_map = btree:/etc/postfix/verify" suggestion ?
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Mon Feb 27, 2006 1:36 pm

I think it counts the number of times the string "client=" is found. If a message goes into amavisd-new that has two recipients (for example) amavisd-new will connect to Postfix twice after the mail is processed. Recipient expansion occurs within amavisd-new. One message in, two messages out.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Mon Feb 27, 2006 1:50 pm

"One message in, two messages out."
I agree, because in my log, there are many multi-fake-recipients spams.
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Mon Feb 27, 2006 2:46 pm

mr88talent =>
you wrote 5324 rejected (75%) , is that with your "address_verify_map = btree:/etc/postfix/verify" suggestion ?


The address_verify_map is an optional file based database of valid and invalid recipients that Postfix creates if the reject_unverified_recipient restriction is used. Postfix normally stores this information in memory if a restriction like reject_unverified_recipient is used. The address_verify_map can instead be used to store the data.

I use relay_recipients to reject mail to invalid recipients. You would only use reject_unverified_recipient under four circumstances.

1) There is no other way possible to determine who valid recipients are. This means it would be impossible for you to list all the email address you relay mail for in the relay_recipients map. If this is TRULY the case, you would get rid of:

relay_recipient_maps = hash:/etc/postfix/relay_recipients

Notice in my doc I say "You MUST remove the entries above at some point in the near future and replace them with every single one of your valid recipients' email addresses."

2) It actually has to work. If you read the documentation:
http://www.postfix.org/ADDRESS_VERIFICATION_README.html
the connecting client is put on hold for a few seconds while Postfix checks the downstream server to see if the recipient is valid. If the downstream server returns a 55x or 45x code, Postfix will reject the message. If that server does not immediately reject the request, you cannot use reject_unverified_recipient.

3) You must have sufficient memory or disk space to hold the database. Probably would not grow to more than 100MB, but I would guess it depends on how many domains are hosted.

4) You must be willing to take some (probably small) additional risk. If the database becomes corrupt, or you run out of disk space or memory, "the world comes to an end". You would have to delete the database (if file based) and stop and start Postfix manually.
Last edited by mr88talent on Mon Feb 12, 2007 12:22 am, edited 9 times in total.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby mr88talent » Mon Feb 27, 2006 2:57 pm

Notice in my doc I say "You MUST remove the entries above at some point in the near future and replace them with every single one of your valid recipients' email addresses."
To avoid confusion, this does not mean remove:

relay_recipient_maps = hash:/etc/postfix/relay_recipients

It means edit /etc/postfix/relay_recipients and replace your domains:

@example.com 1
@example2.com 1
@example3.com 1

with valid email addresses:

user1@example.com 1
user2@example.com 1
user3@example.com 1

The domains are just placeholders that allow you to deliver to those domains while you do the work of obtaining every single one of your valid email addresses.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Thu Mar 09, 2006 12:35 pm

An other reply to thanks mr88talent again !
100 % of the spam is marked as spam
0 % of ham marked as spam

amazing 8)

If it interess someone, I've got a vbs script running under windows giving in one text file all the email adresses / secondary email adresses / distribution list email adresses of an exchange mail system.
It is really usefull to populate /etc/postfix/relay_recipients.
Just ask me by private mail :wink:

humm ... just a tiny little question :
is there any way to send all spam mails to 1 special email adress ?
$spam_quarantine_to ? :roll:
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby vmaillot » Thu Mar 09, 2006 4:41 pm

mmmm ? :?

It's like if $spam_quarantine_to was not working at all ...
I've read that this is the parameter to change to redirect all spams to one special email adress, but it's running for 2 hours without any success.

Does anyone have an idea ?
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Thu Mar 09, 2006 4:49 pm

I didn't answer before because I usually get the wrong answer and someone else comes back with the correct one... but since no one is answering, I'll give it a shot:


Code: Select all
$sa_kill_level_deflt = 7.5; # triggers spam evasive actions


Do you have that line (with whatever value you want) uncommented? I believe that as long as you don't have this enabled, nothing will ever be "killed" but will instead be sent to the end user. I believe that with this enabled, the email is "killed" as far as the end user is concerend, but then the program does with the mail as you've directed it.

Code: Select all
$sa_quarantine_cutoff_level = 20; # spam level beyond which quarantine is off


This is the level at which SPAM is just deleted and no one sees it.

Unless someone else answers, I'd at least see what you have these set to in amavisd.conf

Bon chance,

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Thu Mar 09, 2006 4:59 pm

Wow, you are so courageous ! :lol:
By the way, thanks for your answer !

I've watched the parameters you suggested, and here is what I have :

$sa_tag_level_deflt = -9999.9; (to add spam info to all mails)
$sa_tag2_level_deflt = 3.5; (my configuration)
$sa_kill_level_deflt = 9999.9; (to only tag spam)
$sa_dsn_cutoff_level = 9; (i think it's the default value)

And this one is commented :
#$sa_quarantine_cutoff_level = 20;

Everything is done to not kill the spam (it's important).
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Thu Mar 09, 2006 5:14 pm

Just got this right out of Mr88's instructions:

Code: Select all
Change
$sa_kill_level_deflt = $sa_tag2_level_deflt;
to
$sa_kill_level_deflt = 8.0; #
I just inserted 8.0; then commented out the rest of the line.
On our system, this will trigger the spam to be quarantined if it scores 8.0 or higher.
If you plan on deleting the spam, set this at 12.0 or higher.
If you only want spam tagged and sent to the recipients (not quarantined at all), set this at 9999.9
(this would be for ISPs and large companies that configure the MUAs to further process spam).


If you only want spam tagged and sent to the recipients (not quarantined at all), set this at 9999.9


There's your problem!

So, I would change your setting to something like:

Code: Select all
$sa_kill_level_deflt = 20.0;


That's a reasonably high score and I doubt you'll get any false positives... and even if you do, they will only be sent to your quarantine email address, so you can catch them. I have mine set to "kill" (quarantine) at just 7.5 and completely delete at 20. In the couple weeks I've had the filter running, I've had 6,935 emails that scored between 7.5 and 20 get quarantines (and looked at all of them)... no false positives so far!

Joe
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Thu Mar 09, 2006 5:30 pm

well done ;)

I apply this configuration and I'll see the spam mail box.
We will see if it's ok.
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Thu Mar 09, 2006 5:34 pm

Hoorah! I've gone from a complete and utter neophyte noobie to actually being able to help some one!

:D :D :D :D :D :D :D :D :D :D :D

Joe
the happy
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby vmaillot » Thu Mar 09, 2006 5:49 pm

Joe ! you have to be more happy : it works !!!!! :) :) :)

Excellent job ! :D
Thanks a lot, you're quite so excellent as mr88talent :lol:
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby Netopia » Thu Mar 09, 2006 6:09 pm

vmaillot wrote:Joe ! you have to be more happy : it works !!!!! :) :) :)

Excellent job ! :D
Thanks a lot, you're quite so excellent as mr88talent :lol:


I might have done a good job, but as excellent as Mr88? Ce n'est pas vrai!

:D

Joe
I love admins who remove 30 second delays for ID10Ts like me who make lots of speeling eerors.

Thanks Dave!
User avatar
Netopia
Moderator
 
Posts: 412
Joined: Mon Oct 10, 2005 5:05 pm
Location: Maryland, USA

Postby mr88talent » Thu Mar 09, 2006 8:19 pm

It helps keep things tidy if a new thread is started when a new subject comes up.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City

Postby vmaillot » Fri Mar 10, 2006 10:17 am

oosp sorry :roll:
But I thought as it was my spambox story I could continue on this post :oops:
I promess I'll do a new topic next time.
Thanks !
Vincent
User avatar
vmaillot
 
Posts: 37
Joined: Mon Feb 20, 2006 4:15 pm
Location: Montreal, Qc - Canada

Postby mr88talent » Fri Mar 10, 2006 11:03 am

No problem :) It is easier to find solutions to problems when the subject reflects the issue at hand.
User avatar
mr88talent
Moderator
 
Posts: 1676
Joined: Tue Mar 08, 2005 4:19 pm
Location: Salt Lake City


Return to Debian

Who is online

Users browsing this forum: No registered users and 0 guests