Author: Marian Gunkel
Contact: gunkel@student.hu-berlin.de
Date: 25th January 2002
Last change: 26th January 2002

Synchronisation?
It doesn't matter what medium someone uses: a posting via a mailing list will show up in a web based forum, too. A posting in a web based forum will be sent to the mailing list as well. Discussion threads are possible.

What and Why

This document describes how to setup the synchronisation of a web based discussion forum (Phorum version 3.2.11) and an ordinary majordomo mailinglist (ML).
I mainly deal with problems that can and will occur in the process as well as solutions.

For a long time I have been obsessed with an all-round solution for computer-based communications. Users need to decide themselves what medium they are using to communicate. According to their work and life style, users want to choose if they are using email or web based discussion groups. With the solutions described, it is easier to communicate. However, I stick to asynchronous media for the time being (one reason is: I hate chats :-)).

Currently, the solution works in a test environment with only little bugs. In a production environment, however, more bugs will show up. I'll do my best to debug the system and report all changes here and at the Phorum homepage.

I assume you are familiar with both the phorum software (available at http://www.phorum.org) and the majordomo mailing list manager (available at http://www.greatcircle.com/majordomo/). While you should have installed phorum and have it running, you can use an ordinary majordomo mailinglist (you need to be able to subscribe email addresses to it).

Chapters:

1 - Sending Phorum-postings via email to a mailing list
2 - Sending mail from a mailing list to Phorum
3 - Ongoing Problems
4 - The files (cm.pl, syncmail.php)
5 - Thanks to Phorum, Ralph Hoehn, Jan-Klaas Kollhof

1 - Sending Phorum-postings via email to a mailing list

Phorum has already built in the possibility to send postings to an email address. This can be the address of a mailing list. You can also specify the "return address" which will show up as REPLY-TO: header line in an email. And you can specify the ML tag (which is usually added to the subject by the ML software, i.e. [foldingboats]).
The ML tag will be stripped off the subject when posting a ML email.

Problem No. 1:

The FROM-field in an email sent by Phorum contains the email-address of the poster (who is not likely to be subscribed to the ML). Majordomo is not accepting emails by non-subscribers. Those emails will bounce (be rejected and sent to the list-owner).

Solution - overview:

Phorum sends all posts as email to a "neutral email account" (dummy_address@gmx.net). It also sets the FROM: header to dummy_address@gmx.net.
The "neutral email account" dummy_address@gmx.net is filtering all emails and forwards the right emails to majordomo (dummy_address@gmx.net is subscribed to majordomo).
In my case, I am using gmx.de, a German freemail provider. You need header-filtering and instant forwarding to other email-addresses.

Solution - what to do:

1. Phorum: change file include/post.php
add one line in the range of line 380:
$ebody = '';
if ($id) {
$ebody.="This message was sent from: $ForumName.\n";
$ebody.="Original author: $plain_author <$from_email>\n"; // added so people know who's the author
$ebody.="<$forum_url/$read_page.$ext?f=$num&i=$id&t=$thread> \n";
$ebody.="----------------------------------------------------------------\n\n";

In the range of line 390, change the line
"\nFrom: $plain_author <$from_email>" .
to
"\nFrom: $plain_author <dummy_address@gmx.net>" .

2. Phorum admin area - Forum Properties
mailing list address: dummy_address@gmx.net
return address: dummy_address@gmx.net
[Problem: if a poster has checked email replies to his/her postings and answers those emails, they will be sent to dummy_address@gmx.net and then forwarded to the ML but the email will bounce - needs to be solved]
mailing list tag: foldingboats [just an example; enter your mailing list tag if you have one]

3. setup dummy_address@gmx.net (not this address, of course ;-))
- set up email account
- install filters:

• delete all emails with a typical majordomo-header line
  (in my case, I delete all emails with "Sender: owner-[name of ML]@domain.com" in the header)
• delete all emails with "TO: dummy_address@gmx.net" AND "TO:ML@domain.com"
  (this is to avoid double postings when someone hits "reply all" in the email program)
• forward all emails with *no* ML-header to the ML
  

4. set up phorum@domain.com [I'll talk about this one later. It is the main adress for receiving emails that are to be posted into phorum]
5. subscribe phorum@domain.com and dummy_address@gmx.net to the ML.
As a list-owner, you should have the necessary password. Send an email to majordomo@domain.com with the following two lines:
approve [password] subscribe [ML name] phorum@domain.com
approve [password] subscribe [ML name] dummy_address@gmx.net

Now we can send all phorum postings to a ML (and have installed some features that are needed in the second step: sending emails to phorum).

2 - Sending Phorum-postings via email to a mailing list

This is the trickier part. Something where most web-based discussion forums just give up. Not phorum. A script called phorummail.php (in the scripts folder) is doing all the work if you got a well designed webserver and mailserver.
Problem: mine isn't well designed. So I "invented" a procedure to make it work.

Problem No. 1:

- My .procmailrc is not allowed to execute a script.
Phorummail needs to have an email "piped" (sent as standard input) to it, together with some info where the phorum is, where phorummail is and what forum the email should go to.
I work with procmail, a UNIX software that processes and filters mail. The filter information is stored in a special file called .procmailrc [note the leading dot]. Unfortunately, my procmail is not allowed to execute a script (if you "pipe" an email to a script, you are technically executing that script).

Solution:

So, I decided to put emails for a certain phorum into a special folder. As I mentioned, phorum@domain.com is subscribed to the ML and all emails are being sent to it.

Here is my .procmailrc recipe:

LOGFILE =$HOME/Procmail/procmail.log
LOGABSTRACT = "all"
VERBOSE = "on"

:0
* ^TO_phorum@domain.com
* ^Subject:.*foldingboats
www/phorum/mail/forum1/

:0
* ^TO_phorum2@domain.com
* ^Subject:.*foldingboatbuilders
www/phorum/mail/forum2/

This will log everything in the file Procmail/procmail.log [you can delete those first three lines after everything runs smoothly]. It will store emails to phorum@domain.com with the word foldingboats in the folder www/phorum/mail/forum1/ [note the relative path]. Inside the folder forum1, the first email will make three folders new, cur and temp. In the folder new you will find the emails.
How do we avoid mail loops (Phorum-posting -> dummy_address@gmx.net -> ML -> phorum@domain.com -> Phorum-posting -> dummy_address@gmx.net -> ML -> phorum@domain.com -> Phorum-posting .....)?
Emails with phorum-headers in them [i.e. sent from phorum to dummy_address@gmx.net] will be both deleted via .procmailrc filters [I have not written them yet] and via phorummail/syncmail. All other ML-messages are forwarded to the folder mail/forum1.

Problem No. 2:

How do we get the emails into phorummail?

Solution:

A script called syncmail.php is doing all the work phorummail does as well as some extra things. It is basically phorummail with a couple of extensions. Syncmail.php is included in the files phorum/list.php and phorum/index.php.
list.php, lines 26 and 27:
26 //sync mails
27 $s=file("http://www.domain.com/phorum/scripts/syncmail.php?f=$num");

index.php, lines 20-22:
20 //sync all the mails by calling the syncmail.php
21 $s=file("http://www.domain.com/phorum/scripts/syncmail.php?f=1");
22 $s=file("http://www.domain.com/phorum/scripts/syncmail.php?f=2");

So, if someone is accessing the phorum via a browser, the following steps are executed:

• syncmail will execute a perl-script cm.pl that chmods the new mail messages in the folder mail/forum1 to 644.
• it will write each email into a new temporary file to avoid messing up with new incoming emails
• it will do the same as phorummail does with each email (checking emails, stripping certain things and posting them into the DB.
• above the normal phorummail functions, syncmail is also stripping off the ML footer as well as decoding MIME-encoded email subjects (which would otherwise look messy on the phorum).
• it will delete the temporary email.

This works better the more hits your phorum gets (the more times index.php and/or list.php are called). As soon as someone hits the phorum, all new emails will be included into the DB and show up as new postings.

3 - Ongoing Problems

Currently, there is one scenario where I have found no solution:

- a poster has asked for email-notification and is replying to an answer via email. Should this go into the ML (via REPLY-TO: gmx.de) and then into the Phorum or should it go directly to phorum@mydomain.com, posted into the DB and then sent to the ML?
-> At the moment, he/she will reply to dummy_address@gmx.net (which is both FROM and REPLY-TO). This will forward the email to the ML but the ML will bounce this message (since there is FROM: poster_email_address).

I am also still finetuning the filters in gmx.de and the .procmailrc file. I would like to do all the filtering via these instances in order to avoid phorum-confusion with dublicate messages and the like.

Postings via email are sometimes a bit messy, I need to work on stripping off the MIME-parts of those emails.

4 - The Scripts:

-- cm.pl:

#!/usr/bin/perl
$mailPath = "/home/domain/www/phorum/mail/forum";
@phorumNums = ('1','2');

print"Content-type: text/html\n\n";

foreach $n (@phorumNums)
{
#reading in the current filenames
opendir(dir, "$mailPath$n/new");
while($fle=readdir(dir))
{
if (!(-d $fle))
{
chmod 0644, "$mailPath$n/new/$fle";
}
}
closedir(dir);
}
print "cm OK";

//cm.pl ends here .......................

-- syncmail.php:

<?PHP
//make sure script won't be stopped
ignore_user_abort();

// some vars
$PhorumMail=true; // Do not touch.
$phorum_path="/home/domain/www/phorum"; // this the path to the main Phorum dir.
$admin_email="phorum@domain.com"; // This should match the default email for Phorum.

$plChmodPath="http://www.domain.com/phorum/scripts/cm.pl"; //perl script to chmod the mails
$s=file($plChmodPath);

//$verbose=false;
$num="" + join("", $argv); //forum-number

@chdir($phorum_path);
// check for all we need.
include "common.php";
include "include/post.php";
$phorummailcode = strtolower($PhorumMailCode);
$email=$DefaultEmail;

$postMailPath="$phorum_path/mail/forum$num/new"; //path to the folder mails are stored in

$tmpPath="$postMailPath/" . microtime();
mkdir($tmpPath, 0777);
$handle=opendir($postMailPath);
do
{
$fName=readdir($handle);
if (is_file("$postMailPath/$fName"))
{
//move all files into the new directory
rename("$postMailPath/$fName", "$tmpPath/$fName");
}
}while($fName != "");
closedir($handle);

//parse tmp-folder and post the mails
//when finished delete the folder
$handle=opendir($tmpPath);
do
{
$fName=readdir($handle);
//read and post all files but the clear-file
if (is_file("$tmpPath/$fName"))
{
//save name in .clear file for deleting it later
// Get input
$stdin=file("$tmpPath/$fName");
$message=implode("", $stdin);
if($message=="" || empty($num))
{
if(empty($num))
{
$error ="PhorumMail could not run for the following reason(s):\n\n";
if(empty($num))
$error.=" The forum id was not specified.\n";

$error.="\n";
$error.="An example of a correct PhorumMail command line is:\n\n";
$error.=" /usr/local/bin/phorummail forum=5 path=/usr/home/www/phorum\n\n";
$error.="A copy of the message is included below.\n\n";
$error.="====================================================================\n\n";
$error.="$message";
mail($email, "PhorumMail failure", $error, "From: PhorumMail <$email>\nReturn-Path: Phorummail <$email>");
}
}else
{
// read in headers
reset($stdin);
// Would be nice to use array_shift() here, but that's PHP4 only.
while (list($linenum,$line) = each($stdin))
{
$line = trim($line);
$parts=explode(": ", $line);
$type=$parts[0];
unset($parts[0]);
$value=trim(implode(": ", $parts));
// Use strtolower to avoid case problems
$eHeaders[strtolower($type)]=$value;
unset($stdin[$linenum]);
if (empty($line))
{
break;
}
}

// read in the body

## We do this in the post function now.
$body=str_replace("\r\n", "\n", trim(implode("", $stdin)));

//strip off subscriptions lines
$body=str_replace("#########################################################", "", $body);
$body=str_replace("Foldingboats Mailing List - All postings copyright the author and not to be\n", "", $body);
$body=str_replace("reproduced outside Foldingboats or Foldingboats archives without author's permission\n", "", $body);
$body=str_replace("Submissions: foldingboats@pouchboats.com\n", "", $body);
$body=str_replace("Subscriptions: majordomo@pouchboats.com\n", "", $body);

print "<pre>\n"; print_r(get_defined_vars()); print "</pre>\n";

$body.="------------------------------------------\n";
$body.="Posted to Phorum via PhorumMail\n";


if (empty($eHeaders["date"]))
{
$datestamp = date("Y-m-d H:i:s");
} else
{
$datestamp = date("Y-m-d H:i:s");
// $datestamp = date("Y-m-d H:i:s", strtotime($eHeaders["date"], time()));
// changed because posted emails had a silly date (like 5 days ago)
}

if(@ereg("([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)", $eHeaders["received"], $regs))
{
$ip=$regs[0];
} else
{
$ip="PhorumMail";
}
$host = @gethostbyaddr($ip);

if (empty($eHeaders["from"]))
{
$author='';
$email='';
} else
{
if(ereg('"([^"]+)" <([^>]+)>', $eHeaders["from"], $regs))
{
$author=$regs[1];
$email=$regs[2];
} else if(ereg('([^<]+)<([^>]+)>', $eHeaders["from"], $regs))
{
$author=trim($regs[1]);
$email=$regs[2];
} else if(substr($eHeaders["from"],0,1)=="<")
{
$author=substr($eHeaders["from"], 1, -1);
$email=$author;
} else
{
$author=$eHeaders["from"];
$email=$eHeaders["from"];
}
}

$thread = 0;
$parent = 0;


if (empty($eHeaders["subject"]))
{
$subject = '';
} else
{
$subject = trim($eHeaders["subject"]);
// Strip out [$ForumEmailTag] if it exists.
// Use eregi, because some software seems to tamper with the case of the tag!
if (eregi("^([^[]*)\\[$ForumEmailTag\\](.*)$", $subject, $regs))
{
$subject = trim($regs[1]) . ' ' . trim($regs[2]);
$subject = trim($subject);
}
// Look for "[forum:thread:parent]" at the end of the subject
// We aren't actually using these anymore, but their presence would
// screw up the threading.
if (ereg('^(.+) +\[([0-9]+):([0-9]+):([0-9]+)\]$', $subject, $regs))
{
$subject = $regs[1];
// $forum=$regs[2];
// $thread=$regs[3];
// $parent=$regs[4];
}
}

// Decode characters like German Umlaute in subject
$subject = imap_mime_header_decode($subject);
$subject = $subject[0]->text;


$forum = $num;
if (!empty($eHeaders["x-phorum-$phorummailcode-forum"])&& (strcmp($eHeaders["x-phorum-$phorummailcode-forum"],$ForumName) == 0))
{
if (!empty($eHeaders["x-phorum-$phorummailcode-thread"]))
{
$thread = $eHeaders["x-phorum-$phorummailcode-thread"];
}
if (!empty($eHeaders["x-phorum-$phorummailcode-parent"]))
{
$parent = $eHeaders["x-phorum-$phorummailcode-parent"];
}
}

$action="post";
$toaddress = empty($eHeaders["to"]) ? '' : $eHeaders["to"];

$msgid = empty($eHeaders["message-id"]) ? '' : $eHeaders["message-id"];

// The email this is a reply to should be in In-Reply-To, but some
// mailers seem to use References instead.
// Both fields can have multiple message ids in them, so just grab the first.
$inreplyto = '';
if (@ereg('^(<[^>]+>)', $eHeaders["in-reply-to"], $regs))
{
$inreplyto = $regs[1];
} else if (@ereg('^(<[^>]+>)', $eHeaders["references"], $regs))
{
$inreplyto = $regs[1];
}

$IsError = check_data($host, $author, $subject, $body, $email);
if (!empty($IsError))
{
violation();
}

$author=trim($author);
$subject=trim($subject);
$email=trim($email);
$body=chop($body);
list($author, $subject, $email, $body) = censor($author, $subject, $email, $body);

$author = addslashes($author);
$email = addslashes($email);
$subject = addslashes($subject);
$body = addslashes($body);

$plain_author=stripslashes($author);
$plain_subject=stripslashes(ereg_replace("<[^>]+>", "", $subject));
$plain_body=stripslashes(ereg_replace("<[^>]+>", "", $body));

$author = htmlspecialchars($author);
$email = htmlspecialchars($email);
$subject = htmlspecialchars($subject);

$org_attachment = '';

if(($email==$ForumModPass && $ForumModPass!="") || ($email==$Password && $Password!=""))
{
$ForumModeration='';
$email=$ForumModEmail;
$author = "<b>$author</b>";
$subject = "<b>$subject</b>";
$body="$body";
$host="<b>$ForumStaffHost</b>";
}
else
{
$body=eregi_replace("</*HTML>", "", $body);
if($ForumAllowHTML=="Y")
{
$body="$body";
}
}

// If there are no Phorum headers, put it in the database.
if(empty($eHeaders["x-phorum-$phorummailcode-version"]))
{
post_to_database();
}

// Notify people who wanted replies sent to them
post_to_email();
}
//delete the posted file
unlink("$tmpPath/$fName");
}
}while($fName != "");
closedir($handle);
//delete the tmp-folder
rmdir($tmpPath);
?>

5 - thanks to Phorum, Ralph Hoehn, Jan-Klaas Kollhof

I want to thank the Phorum development team who have been designing a marvellous piece of software. Ralph Hoehn of www.PouchBoats.com gave the necessary funds and structure to develop the synchronisation. Jan-Klaas Kollhof has been planning and coding syncmail.php and cm.pl as my PHP and Perl knowledge is veeery small ;-).