.\" -*- mode: troff; coding: utf-8 -*-
.\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
.ie n \{\
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
.    if \nF \{\
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{\
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\"
.\" Required to disable full justification in groff 1.23.0.
.if n .ds AD l
.\" ========================================================================
.\"
.IX Title "POSTFIX2DLF.IN 1"
.TH POSTFIX2DLF.IN 1 2025-09-28 "Lire 2.1.1" "LogReport's Lire Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH NAME
postfix2dlf \- convert postfix logfiles to dlf format
.SH SYNOPSIS
.IX Header "SYNOPSIS"
\&\fBpostfix2dlf\fR
.SH DESCRIPTION
.IX Header "DESCRIPTION"
postfix2dlf converts a postfix logfile to a Lire email Distilled Log Format
file.  It calls \fBpostfix2dlf_pre\fR\|(1) and \fBpostfix2dlf_main\fR\|(1) to do the real job.
It expects the logfile on stdin, and prints the DLF to stdout.  Diagnostics
are printed to stderr.  This script is called by \fBlr_log2xml\fR\|(1).
.SH RATIONALE
.IX Header "RATIONALE"
The postfix2dlf architecture is quite different from other email dlf
convertors.  This is because when parsing a postfix logfile, there is \fIno\fR
sign indicating wether a message has been delivered to all addressees.  (In a
sendmail log, there is a rcpts= field, which can be used for this purpose.)
.PP
There are several workarounds to deal with such a log.  1: keep track of all
queueid\*(Aqs along with their from\- info during the entire logfile processing.
This is too memory hungry.  2: do some 10\-seconds heuristic: assume a message
will never be longer in the queue than e.g. 10 seconds.  Or, alternatively,
assume no more than \f(CW$LR_POSTFIX_MAX_QUEUE_SIZE\fR messages will be in the queue
concurrently.  This is too errorprone.  3: Use \fBsort\fR\|(1) to sort the loglines on
queueid.  This is what we used to do.  However, we don\*(Aqt know beforehand which
field will store the queueid (logfiles processed by Sun Solaris syslog will
have their queueid on another position, e.g.)  Calculating this position would
mean parsing the log twice.
.PP
We\*(Aqve choosen a variation of the third alternative: we store the information
which we would\*(Aqve gotten by \fBsort\fR\|(1) in a db file.  While building this db file,
, which holds a map from queueids to nof\-lines\-with\-this\-id, we do as much
preprocessing as we can.  The preprocessed log is printed to a tmpfile.  The
structure of the preprocessed log is the same as the raw log: one raw logline
gives one preprocessed line.  The preprocessed log is converted to dlf by
\&\fBlr_postfix2dlf_main\fR\|(1), using the information in the db file.
.PP
A nice sideeffect of this way of processing is: the log is processed in the
original time\-sorted order.
.SH EXAMPLE
.IX Header "EXAMPLE"
A logfile
.PP
.Vb 10
\& Dec 1 04:02:56 internetsrv postfix/pickup[20919]: 
\&  693A3578E: uid=0 from=<root>
\& Dec 1 04:02:56 internetsrv postfix/cleanup[20921]: 
\&  693A3578E: message\-id=<john.doe.1@example.com>
\& Dec 1 04:02:57 internetsrv postfix/qmgr[20164]: 693A3578E: 
\&  from=<john.doe.2@example.com>, size=617 (queue active)
\& Dec 1 04:02:57 internetsrv postfix/cleanup[20921]: 
\&  E325C578D: message\-id=<john.doe.1@example.com>
\& Dec 1 04:02:58 internetsrv postfix/local[20924]: 
\&  693A3578E: to=<john.doe.2@example.com>, relay=local, 
\&  delay=3, status=sent (forwarded as E325C578D)
\& Dec 1 04:02:58 internetsrv postfix/qmgr[20164]: E325C578D: 
\&  from=<john.doe.2@example.com>, size=769 (queue active)
\& Dec 1 04:02:59 internetsrv postfix/smtp[20925]: E325C578D: 
\&  to=<john.doe.3@example.com>, 
\&  relay=1.example.com.vp.pt[10.0.0.1], delay=2, status=sent 
\&  (250 Requested mail action Ok.)
\& Dec 1 06:58:22 internetsrv postfix/smtpd[21142]: connect 
\&  from 2.example.com.fi[10.0.0.2]
\& Dec 1 06:58:23 internetsrv postfix/smtpd[21142]: 
\&  42BFE578D: client=2.example.com.fi[10.0.0.2]
\& Dec 1 06:58:24 internetsrv postfix/cleanup[21143]: 
\&  42BFE578D: message\-id=<john.doe.4@example.com>
\& Dec 1 06:58:24 internetsrv postfix/qmgr[20164]: 42BFE578D: 
\&  from=<john.doe.5@example.com>, size=2473 (queue active)
\& Dec 1 06:58:26 internetsrv postfix/smtp[21145]: 42BFE578D: 
\&  to=<john.doe.6@example.com>, 
\&  relay=1.example.com.vp.pt[10.0.0.1], delay=3, status=sent 
\&  (250 Requested mail action Ok.)
\& Dec 1 06:59:22 internetsrv postfix/smtpd[21142]: 
\&  disconnect from 2.example.com.fi[10.0.0.2]
\& Dec 1 07:08:28 internetsrv postfix/smtpd[21160]: connect 
\&  from 2.example.com.fi[10.0.0.2]
\& Dec 1 07:08:28 internetsrv postfix/smtpd[21160]: 
\&  C7B39578D: client=2.example.com.fi[10.0.0.2]
\& Dec 1 07:08:29 internetsrv postfix/cleanup[21161]: 
\&  C7B39578D: message\-id=<john.doe.7@example.com>
\& Dec 1 07:08:29 internetsrv postfix/qmgr[20164]: C7B39578D: 
\&  from=<john.doe.8@example.com>, size=2173 (queue active)
\& Dec 1 07:08:32 internetsrv postfix/smtp[21163]: C7B39578D: 
\&  to=<john.doe.9@example.com>, 
\&  relay=3.example.com.vp.pt[10.0.0.3], delay=4, status=sent 
\&  (250 Requested mail action Ok.)
\& Dec 1 07:08:33 internetsrv postfix/smtpd[21160]: 
\&  disconnect from 2.example.com.fi[10.0.0.2]
\& Dec 1 07:18:42 internetsrv postfix/smtpd[21166]: connect 
\&  from 2.example.com.fi[10.0.0.2]
.Ve
.PP
will get converted to
.PP
.Vb 10
\& 1007175776 internetsrv 693A3578E <john.doe.1@example.com>
\&  john.doe.2 example.com localhost 127.0.0.1 617 3 0
\&  john.doe.2 example.com localhost 127.0.0.1 sent
\&  (forwarded_as_e325c578d)
\& 1007175779 internetsrv E325C578D <john.doe.1@example.com>
\&  john.doe.2 example.com localhost 127.0.0.1 769 2 0
\&  john.doe.3 example.com 1.example.com.vp.pt 10.0.0.1 sent
\&  (250_requested_mail_action_ok.)
\& 1007186303 internetsrv 42BFE578D <john.doe.4@example.com>
\&  john.doe.5 example.com 2.example.com.fi 10.0.0.2 2473 3 0
\&  john.doe.6 example.com 1.example.com.vp.pt 10.0.0.1 sent
\&  (250_requested_mail_action_ok.)
\& 1007186908 internetsrv C7B39578D <john.doe.7@example.com>
\&  john.doe.8 example.com 2.example.com.fi 10.0.0.2 2173 4 0
\&  john.doe.9 example.com 3.example.com.vp.pt 10.0.0.3 sent
\&  (250_requested_mail_action_ok.)
.Ve
.PP
postfix2dlf will be rarely used on its own, but is more likely called by
lr_log2report:
.PP
.Vb 1
\& $ lr_log2report postfix < /var/log/mail.log > report
.Ve
.PP
\&.  If you\*(Aqd really like to run this script standalone (e.g. for debugging)
run it as
.PP
.Vb 1
\& $ LR_SERVICE=email LR_ID=\`date +%s\` /path/to/lire/convertors/postfix2dlf < /var/log/mail.log > mail.dlf
.Ve
.PP
\&.  Be sure to have /path/to/libexec/lire and /path/to/lire/convertors in your
PATH, and be sure to have TMPDIR, LR_DBFILE and LR_DBDIR set.  You could
manually source /path/to/etc/lire/profile_lean and /path/to/etc/lire/defaults
to achieve this.
.SH LOGFORMAT
.IX Header "LOGFORMAT"
Postfix logs look like this:
.SS "from local to remote"
.IX Subsection "from local to remote"
.Vb 9
\& postfix/pickup[81586]: 094BE204: uid=1001 from=<edwin>
\& postfix/cleanup[81683]: 094BE204: 
\&  message\-id=<20000531080729.L39824@cgmd76206.c.nl>
\& postfix/qmgr[13460]: 094BE204: 
\&  from=<edwin@cgmd76206.c.nl>, size=1717 (queue active)
\& postfix/smtp[81685]: 094BE204: to=<r.moeskops@c.nl>, 
\&  relay=smtp.c.nl[212.83.68.146], delay=4, status=sent (250 
\&  Message received: 
\&  20000531060722.ZCOV13476.relay02@cgmd76206.c.nl)
.Ve
.SS "from local to local"
.IX Subsection "from local to local"
.Vb 8
\& postfix/pickup[81849]: 473B9204: uid=1001 from=<edwin>
\& postfix/cleanup[81916]: 473B9204: 
\&  message\-id=<200005310901.LAA56567@kludge.mpn.cp.p.com>
\& postfix/qmgr[13460]: 473B9204: 
\&  from=<edwin@cgmd76206.c.nl>, size=1997 (queue active)
\& postfix/local[81918]: 473B9204: to=<edwin@cgmd76206.c.nl>, 
\&  relay=local, delay=0, status=sent 
\&  ("|exec /usr/local/bin/procmail \-t")
.Ve
.SS "from remote to local"
.IX Subsection "from remote to local"
.Vb 9
\& postfix/smtpd[82056]: A17131C5: 
\&  client=gw\-nl1.o\-it.com[193.79.128.34]
\& postfix/cleanup[82057]: A17131C5: 
\&  message\-id=<023201bfcad1$16365ba0$775910ac@ehvbos.nl.oit.com>
\& postfix/qmgr[13460]: A17131C5: 
\&  from=<Jan.Stap@nl.o\-it.com>, size=1692 (queue active)
\& postfix/local[82059]: A17131C5: 
\&  to=<majordomo\-org@cgmd76206.c.nl>, relay=local, delay=1, 
\&  status=sent ("|/usr/local/majordomo/wrapper majordomo")
.Ve
.SS "from remote to remote"
.IX Subsection "from remote to remote"
.Vb 10
\& postfix/smtpd[58567]: connect from 
\&  kweetal.t.nl[131.155.2.7]
\& postfix/smtpd[58567]: 9A16E229: 
\&  client=kweetal.t.nl[131.155.2.7]
\& postfix/cleanup[58570]: 9A16E229: 
\&  message\-id=<200006041246.OAA23888@bw2.baub.bwk.t.nl>
\& postfix/qmgr[236]: 9A16E229: 
\&  from=<edwin@bw2.baub.bwk.t.nl>, size=774 (queue active)
\& postfix/smtpd[58567]: disconnect from 
\&  kweetal.t.nl[131.155.2.7]
\& postfix/smtp[58574]: 9A16E229: to=<joostvb@x.nl>, 
\&  relay=mx3.x.nl[194.109.6.48], delay=15, status=sent (250 
\&  OAA23290 Message accepted for delivery)
.Ve
.SH PERFORMANCE
.IX Header "PERFORMANCE"
We ran the postfix2dlf suite on a 24MB postfix logfile.  Results were:
.PP
.Vb 3
\& postfix2dlf_pre info read 195257 lines; output 177027 DLF lines; 3 errors
\& postfix2dlf_pre info memory stats: vsize=5900K rss=4508K majflt=430
\& postfix2dlf_pre info elapsed time in seconds real=626 user=449.16 system=8.7
\&
\& postfix2dlf_main info read 177027 lines; output 61671 DLF lines; 0 errors
\& postfix2dlf_main info memory stats: vsize=5976K rss=4656K majflt=427
\& postfix2dlf_main info elapsed time in seconds real=245 user=152.05 system=4.15
.Ve
.PP
\&.
.SH EXAMPLES
.IX Header "EXAMPLES"
postfix2dlf will be rarely used on its own, but is more likely called by
lr_log2report:
.PP
.Vb 1
\& $ lr_run lr_log2report postfix < /var/log/maillog
.Ve
.SH BUGS
.IX Header "BUGS"
This script needs a lot of space in TMPDIR: it creates a tmpfile which is about
the same in size as the raw log it\*(Aqs being fed.
.PP
Occasionally, postfix reuses its queueids very fast.  We can\*(Aqt cope with this.
.SH THANKS
.IX Header "THANKS"
Brad Knowles, for supplying patches.  Emanuele "luca" for pointing out
the lmtp delivery.
.SH "SEE ALSO"
.IX Header "SEE ALSO"
\&\fBpostfix2dlf_main\fR\|(1), \fBpostfix2dlf_pre\fR\|(1) and the other email dlf convertors:
\&\fBargomail2dlf\fR\|(1), \fBexim2dlf\fR\|(1), \fBnms2dlf\fR\|(1), \fBqmail2dlf\fR\|(1), \fBsendmail2dlf\fR\|(1); the
caller \fBlr_log2xml\fR\|(1).
.SH VERSION
.IX Header "VERSION"
\&\f(CW$Id:\fR postfix2dlf.in,v 1.40 2006/07/23 13:16:34 vanbaal Exp $
.SH COPYRIGHT
.IX Header "COPYRIGHT"
Copyright (C) 2000, 2001, 2002 Stichting LogReport Foundation LogReport@LogReport.org
.PP
This program is part of Lire.
.PP
Lire is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
.PP
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
.PP
You should have received a copy of the GNU General Public License
along with this program (see COPYING); if not, check with
http://www.gnu.org/copyleft/gpl.html.
.SH AUTHOR
.IX Header "AUTHOR"
Joost van Baal, embrionic version by Edwin Groothuis.