Subject: Easy to parse mail messages?

From: "J. Daniel Smith" <[email-masked]>
Date: Wed, 1 Feb 95 09:12:10 -0500

I'm in the process of setting up a system to process incoming mail
messages automatically (obviously using procmail).  Most of the
interesting information I want to put in the body of the message
itself, and NOT in the message headers.  This is to provide maximum
flexibility for a wide variety of MUAs and mail gateways.

I have pretty much total control over the format of the message body
text (it will be program generated), so my goal is to generate a
format that is easiest to parse using procmail/formail.

What I've come up with so far, is to make the body contain a bunch of
X- fields, as in
-----
X-Company: Acme, Inc.
X-Name: Joe Blow
X-Product: Gizmo
X-Comment: Your new Gizmo gadget is really great!  I would like to
 order a gazillion more, as I think Gizmo gadgets make great gifts.
X-Message-Version: 1.0
-----

Now I can do something along the lines of
-----
# Save only X-* lines in the message body; some mailers put all kinds
# of screwy things in the message that mess up formail if they are part
# of the header.
:0bf
| grep -i '^X-[a-z][a-z]*:'
# remove blank lines, this makes the X- headers in the body part of
# the mail header so "formail" can be used to extract infomation
:0f
| sed '/^$/ d'
-----
The only problem is that wrapped lines (X-Comment:) won't survive the
"grep".  Something similar to the above steps are necessary in case
the above message is saved to a file and pasted into another mailer;
things like automatic signatures and the like may get added.

I think I'm headed in the right direction here, but I'm wondering if
anyone else has some comments or further suggestions.

Thanks,
   Dan
----------------------- message is author's opinion only ----------------------
J. Daniel Smith <[email-masked]>                   http://www.bristol.com/~dan
Bristol Technology Inc.	                        +1 203 438 6969, 438-5013 (FAX)
Ridgefield, Connecticut (USA)                           {info,[email-masked]
                                --------
 Von guten Maechten wunderbar geboren, erwarten wir getrost, was kommen mag.
 Gott ist mit uns am Abend und am Morgen, und ganz gewiss an jedem neuen Tag.
	- Dietrich Bonhoeffer

---

From: [email-masked] (David W. Tamkin)
Date: Wed, 1 Feb 1995 10:33:32 -0600 (CST)

Dan Smith wrote,

| The only problem is that wrapped lines (X-Comment:) won't survive the
| "grep".  Something similar to the above steps are necessary in case
| the above message is saved to a file and pasted into another mailer;
| things like automatic signatures and the like may get added.

Let's see if some of the easier ways work first:

1. Will those X-headers be at the top of the body?  If so,

 :0fh
 | formail -X ""

  will eliminate the blank line at the end of the real head, and if the first
  lines of the body are legal headers, they'll look like part of the head
  from then on.  Even more easily, if they're at the top of the body,

 :0bf
 | formail -f whatever other options you please

  will parse them.

2. Will there be other indented lines in the body?  If not,

  :0bf # brackets enclose X, space, tab
  | grep -i ^[X 	]

3. If neither #1 nor #2 applies, you can always use one of my nasty old
   sed scripts to reduce the body to X- headers and indented lines that
   follow them, something like this:

  savemetas="$SHELLMETAS"
  SHELLMETAS # not that kind of bracket, asterisk, or semicolon
  :0bf
  | sed -e :b -e '/^X-[!-9;-~]*:/ !d; :a' -e 'n; /^[ 	]/ta' -e bb
  SHELLMETAS="$savemetas"

David W. Tamkin  Box 3284  Skokie, Illinois  60076-6284
[email-masked]    MCI Mail: 426-1818   +1 312 714 5610

---

From: [email-masked] (Stephen R. van den Berg)
Date: Wed, 1 Feb 1995 17:42:48 +0100

>things like automatic signatures and the like may get added.

The better solution would probably be something like:

	# Delete everything in the body up to the first X-field.

	:0 fbw
	| sed -e '/^X-[-0-9a-zA-Z]*:/,$ !d'

	# Delete the first empty line in the mail, tack start of body onto
	# header (yes, get out the sed manual, this one was not trivial to
	# construct :-).

	:0 fw
	| sed -e '1,/^$/ !b' -e '/^$/ d'

-- 
Sincerely,                                  [email-masked]
           Stephen R. van den Berg (AKA BuGless).

He did a quarter of the work in *half* the time!