Exporting a mailman archive to MySQL

I live!

Here’s a Python 3 script to export a mailman archive into a useful format in MySQL.

This is a big step in migrating from mailman to A.N.Other web-based forum (Buddypress, Drupal, whatever).

Postie and wp-mailman

So, Postie is looking very handy. It’s got POP3 and IMAP mail handling, as well as MIME attachments. My plugin will essentially be a merge of that with WP-Mailman, and some extra code.

My plugin creates an extra database table, to cross-reference BB post IDs with email message ids. That way, the plugin can track which message in response to which.

Postie seems to do this tracking of threads by using the email subject, which I’ll use as a backup plan. Primary line of attack is to process in-reply-to and references email headers for message ids that appear in the cross-reference table.

Postie also does nice filtering. It should be possible to automatically pick up the mailing-list footer via python, and add a postie-filter to strip out that footer.

how to do it?

So, here’s the question.

Given a mailman mailing list, and a corresponding buddypress forum**, what are the options for linking them? And what’s best?

There will be 2 groups of posters: those who post via email, and those who post via forum***. There will be 2 groups of readers: those who read via email, and those who read via forum***.

Posting via email: the options

P1: post to a single email box

P2: post to mailman (as mailing-list users do currently)

Ways that wordpress could collect post:

M1: by reading a POP3 email box

M2?: is it possible to put a trigger in mailman, so every time a message will be distributed to subscribers, it calls a web page? That way, it could pass the message as POSTed variables.

Ways that things go from wordpress forum to the mailing list:

W1: by hooking groups_new_forum_topic  every time a new topic appears on the forum, a new thread is injected into mailman; and by hooking groups_new_forum_topic_post, every time a new post is added to a topic, a new post is injected into mailman, with in-reply-to headers to match the subject. A custom-header would be added to record that this mailman message originated in wordpress

W2: hook the creation of all posts, and submit them to the mailing list. Only show a post on the forum, once it appears on the mailing list. That way, all post-moderation can be done via the mailman interface

Alternative routes to combine these

1. P1+M1+W1: users post to a single email box, never to mailman directly.  The email address currently used by mailman is reassigned to a pop3 box, and a new, obscured email address is created for the mailing list: this obscured address will be known only to the list admin, and to wordpress. A cron task runs on wordpress frequently, checking on that email box. It adds messages to the forum.  So for a user posting by email, the route is user -> pop3 box -> wordpress -> forum -> mailman. All moderation would be done in wordpress.
2. P2+M1+W1: users post to the mailman list. The wordpress cron task checks the pop3 of an email address subscribed to the mailman list.  Any messages that didn’t originate in wordpress (check custom headers to find this), get inserted into the forums. So for a user posting by email, the route is user -> mailman -> wordpress -> forum . Moderation would have to be split across wordpress and mailman. This is bad.
3. P2+M1+W2: users post to the mailman list. All forum posts are intercepted, and held from appaering on the forum until they go via the mailing list. So, as above, for a user posting by email, the route is user -> mailman -> wordpress -> forum. And for a user posting by web, the route is user -> wordpress -> mailman -> forum . For list administrators & moderators used to the mailman moderation interface, this will offer the path of least resistance.

I think those are the options, anyway.

**(let’s keep it simple with just one of each, for now)

*** we’ll get to posting and reading by IMAP, SMS, twitter or carrier-pigeon later. One thing at a time, drak it.

processing python values in php

Trying out pastebin:

Here I process the values returned from my python script. This was inspired by the work of Christian Aust.

In the beginning

In the beginning, was Christian Aust’s mailman plugin for wordpress

And that does a simple job, well. Users can subscribe and unsubscribe to one or several mailman lists.  The admin chooses the lists, from the set of public mailman mailing lists operating on the same server. All interactions are done via python.

So, I’ve started with that, and am working at extending it:

1. DONE: users can now switch between subscribed, unsubscribed, subscribed but receiving no messages; or subscribed to the digest
2. IN PROGRESS: I’m working at providing an alternative web connection to the mailman options screens, in case local python access isn’t possible.  One of my target projects uses cPanel, which can be fussy about interfering with command-line access to mailman
3. AMBITION: to post all messages received on a mailman mailing list, onto a specific forum
4. AMBITION: to ensure that those messages end up on the right topic!
5. AMBITION: to forward any messages posted onto the forum directly, get forwarded onto the mailman list too
6. AMBITION: to prevent loops – I don’t want any messages getting repeated infinitely, bouncing between mailman and forum
7. AMBITION: to provide an easy way to merge buddypress membership with mailman membership

That’s probably enough for starters, isn’t it? I’ll start posting code up, next