Page History

...

...

...

...

* Provide means to synchronize Magnolia-based forums with other means of communication, such as mailing-lists. See MGNLFORUM-130@jira.

Observation (exporter) and scheduled task (importer)

...

By polling the mailing list archive (or an account that is subscribed to the list), we should be able to add mgnl:message nodes to the forum.

Other suggestion: setup a mail server, get mails delivered to it and have a listener process those mails. (probably could be done with maillets)

When sending a message to the mailing-list, we should add Mail clients do not (necessarily) keep custom headers, so that we can recognize threads.
Mail clients, for example, should keep those headers when replying (to be validated), and thus match mail-threads with forum-threads.
... well – turns out mail clients do not keep such headers.

When fetching new messages from the list, we check if our custom headers are there. If they are, we add the message to an existing thread. If not, we create a new thread.

New mgnl:message nodes should only be sent to the mailing-list if they have not been created by the importer(mail-poller), i.e if they do not originate from the mailing-list. To allow so, the importer will probably need to set a custom property on mgnl:message nodes.

When sending an email to the list, we should mark the corresponding mgnl:message with the corresponding mail's Message-Id.

When receiving an email, the In-Reply-To will hopefully contain a reference to a Message-Id corresponding to an mgnl:message we know. If not, we can assume this is a new thread.

Open issues

• how to match an email sender with a possibly known forum user ?

Proposed mail headers

we can't rely on using custom headers to mark forum threads etc.

When sending a message to the mailing-list, we keep track of the mail's Message-Id, so we can match responses to the thread.

When fetching new messages from the list, we check if there is a forum:message with the same Message-Id (in which case it means the message was already exported); if this email has an In-Reply-To field with a known Message-Id, we know which thread to attach this message to,

User matching

We need to be able to find users by email. This is currently not possible via the UserManager API.

Proposed custom mail headers

Email clients dont propagate such headers (when responding), but we could still want to send email with custom headers: this is not valid anymore as clients dont propagate such headers.

• X-Magnolia-Forum-ThreadID: the UUID of the thread
• X-Magnolia-Forum-ServerID: a unique identifier to make sure we're pushing data to/from the correct Magnolia instance ?
  • how to generate such a server id? (Jira has such an id for example, when registering a license)

...

• One forum <-> one list
  • nntp host, credentials etc
  • our mailing lists nntp-group-names are in the form of com.magnolia-cms.test1-list
• Ignore Possibility certain email senders (typically Jira or Hudson)
• filter email signatures and quotes (some regex work should help, but what do we do with inlined quotes?)

Libs

I've had some reasonable success at using Kerio's NNTP with commons-net: http://commons.apache.org/net/^{Image Removed}
Still looking for alternatives as commons-net's api isn't superb, and has some scary longstanding issues: https://issues.apache.org/jira/browse/NET-298^{Image Removed}https://issues.apache.org/jira/browse/NET-276^{Image Removed}

-> will use email instead, as Kerio's NNTP implementation does not support the NEWNEWS command.

Open questions

• Implement an observer/listener pattern on forum instead of relying on jcr ?
  • would match the idea of having ForumManager return Forum, Thread and Message bean instances instead of exposing jcr nodes.
  • jcr observation can filter per node type but it filters on the parent node type, ie we need to filter for mgnl:thread and mgnl:message (the latter, because messages can be nested), and thus we can't use ForumManager.getMessage() directly from the EventListener. The EventListener would have to 1) fetch the node and check its type 2) either bypass ForumManager altogether or let ForumManager re-fetch the node
  • we still need deferred and buffered execution of events if possible, so I need to look into reusing info.magnolia.cms.util.DelayedExecutor or the java.util.concurrent package.
    
• Scheduler: avoid overlapping jobs: MGNLSCH-18@jira
• Can we post directly to nntp with Kerio ? yes we can
• filter forum markup ? (probably not, especially if we use markdown)
• filter email signatures and quotes (some regex work should help, but what do we do with inlined quotes?)
  • This has not been implemented yet, but is still an open option for the future.

Links

• NNTP RFC: http://tools.ietf.org/html/rfc977^{Image Removed}
• Arpa text messages: http://tools.ietf.org/html/rfc822^{Image Removed}
• More human-friendly description of the protocol: http://www.faqs.org/docs/linux_network/x-087-2-nntp.protocol.html^{Image Removed}
• http://www.tcpipguide.com/free/t_UsenetMessageFormatandSpecialHeaders-2.htm^{Image Removed}