Analyze an MBOX Email Archive

Extract timeline and sender information from an MBOX email archive file

The conversation graph of a mailbox consists of vertices representing each email and a directed edge from each vertex to its direct replies. Such a representation allows finding and analyzing the different conversation threads present in the mailbox. The example uses a mailing list archive taken from here and stored as an "MBOX" file. First we will form the conversation graph for the collection of messages.

Download the MBOX file:

In[1]:=

file=URLDownload["https://lists.apache.org/api/mbox.lua?list=user@forrest.apache.org&date=2004-12"];

Import particular message elements for all email messages:

In[2]:=

fields={"MessageID","ReplyToMessageID","OriginatingDate","Subject","BodyPreview","NewBodyContent","FromAddress"};rawData=Import[file,{"MBOX",fields}];

Create an association with message IDs as the keys and associations containing the various files as the values:

In[3]:=

assocs=AssociationThread[fields,#]&/@Transpose[rawData];mboxData=Association[(#MessageID#)&/@assocs];Length[mboxData]

Out[3]=

130

Extract just the message IDs:

In[4]:=

messageVertices=Keys[mboxData];

Select all messages that are replies to another message:

In[5]:=

replies=Select[mboxData,#ReplyToMessageID=!=None&&MemberQ[messageVertices,#ReplyToMessageID]&];

Create edges from each message to each of its replies:

In[6]:=

messageEdges=Values[Map[DirectedEdge[#ReplyToMessageID,#MessageID]&,replies]];

Create a graph from the vertices and edges, using the new body content as the tooltip for each vertex:

In[7]:=

messageGraph=Graph[messageVertices,messageEdges,{GraphLayout"LayeredDigraphEmbedding",ImageSizeMedium,VertexLabelsMap[#Placed[HoldForm[Evaluate[mboxData[#,"NewBodyContent"]]],Tooltip]&,messageVertices]}]

Out[7]=

Next, let's separate the individual conversations to make a timeline plot of when each conversation begins and ends.

Each connected component of the graph is one conversation thread. Separate individual message threads:

In[8]:=

messageThreads=WeaklyConnectedGraphComponents[messageGraph];Length[messageThreads]

Out[8]=

Compute the timeline of each conversation thread to find the earliest and latest originating dates:

In[9]:=

intervals=Interval[MinMax@Values[mboxData〚VertexList[#]〛]〚All,"OriginatingDate"〛]&/@messageThreads;

Create custom tooltips for each thread with the subject, start time and end time:

In[10]:=

intervalLabeldata=Association[MapIndexed[First[#2]mboxData[VertexList[#]〚1〛,"Subject"]&,messageThreads]];intervalLabelFunction[{start_,end_},pos_,label_]:=Column[{Row[{Style["Subject: ",Bold],intervalLabeldata[First[pos]]}],Row[{Style["Start: ",Bold],DateObject[start]}],Row[{Style["End: ",Bold],DateObject[end]}]}]

Create a

TimelinePlot

with the enhanced label:

In[11]:=

TimelinePlot[intervals,LabelingFunctionintervalLabelFunction]

Out[11]=

The threads can also be analyzed individually. The following selects the first thread containing exactly three messages:

In[12]:=

selection=SelectFirst[messageThreads,VertexCount[#]3&];

Visualize a timeline of the messages in this thread using new body content as labels:

In[13]:=

data=Labeled[mboxData[#,"OriginatingDate"],Tooltip[#,mboxData[#,"NewBodyContent"]]]&/@VertexList[selection];TimelinePlot[data]

Out[13]=

Make a table of this conversation, showing each message's date and new body content:

In[14]:=

TextGrid[Prepend[{mboxData[#,"OriginatingDate"],StringRiffle@mboxData[#,"NewBodyContent"]}&/@VertexList[selection],{"Originating Date","New Body Content"}],FrameAll]

Out[14]=

Originating Date	New Body Content
Wed 8 Dec 2004 15:10:19GMT+1	Hello,Section numbering is enabled for document rendered in PDF, but not forHTML (tigris skin).I would like to get auto-numbering for section headers:1. Section11.1 Section1.11.2 Section1.22. Section2I am unable to find where to enable something like this.Any ideas?Thanks for your help.Olivier.
Wed 8 Dec 2004 14:32:24GMT	Jacques, Olivier (OCBU-Test Infra) wrote: It is not available in the HTML stylesheets, it would be good to seethis as a patch.You can see how it is done in PDF by looking in the file document2fo.xslin the common skin, seehttp://svn.apache.org/viewcvs.cgi/checkout/forrest/trunk/main/webapp/skins/common/xslt/fo/document2fo.xslIf you search for numbersections in that file you'll see the relevantparts.To add this to the HTML versions of pages you need to editdocument2html.xsl - seehttp://svn.apache.org/viewcvs.cgi/forrest/trunk/main/webapp/skins/common/xslt/html/document2html.xsl?rev=109943&view=autoAn appropriate configuration element should also be added to theskinconf.xml file.Ross
Wed 8 Dec 2004 15:49:13GMT+1	Maybe this helps, too:Element:before { content:Value; }content:counter(Name); counter-increment(Name);Seems to be CSS 2.0, I never tried it out.see, e.g.http://en.selfhtml.org/css/eigenschaften/pseudoformate.htm#nummerierungthe example there does not display in Firefox and MSIEit does in OperaCheers,JohannesRoss Gardler wrote: --User Interface Design GmbH * Teinacher Str. 38 * D-71634LudwigsburgFon +49 (0)7141 377 000 * Fax +49 (0)7141 377 00-99Gesch�ftsstelle: User Interface Design GmbH Lehrer-G�tz-Weg 11 D-81825 M�nchenwww.uidesign.deBuch User Interface Tuning von Joachim Machate & MichaelBurmesterwww.user-interface-tuning.deAttraktivit�t von interaktiven Produkten messen mitwww.attrakdiff.de

A sender graph of a mailbox consists of vertices denoting a single sender and edges denoting that two senders participate in the same conversation thread. By weighting the edges by the number of threads in which the senders both appear, the graph illustrates senders that tend to participate in the same conversations.

Extract all senders from the "MBOX" to get the vertices of the sender graph:

In[15]:=

senderVertices=Values[DeleteDuplicates[mboxData〚All,"FromAddress"〛]]

Out[15]=

{crossley@apache.org,mailings@matthias-wessendorf.de,thorsten@apache.org,arikkfir@users.sourceforge.net,nicolaken@apache.org,rgardler@apache.org,nbrajkovic@qnamic.com,glen@iteclectic.com,smoore@ebmail.gdeb.com,johannes.schaefer@uidesign.de,stevanbt@yahoo.co.uk,lindarose@cox.net,ian.springer@hp.com,jmf@estg.ipvc.pt,olivier.jacques@hp.com,charles@dspdesign.com,dave@brondsema.net,rick@apache.org,praveen@hcltech.co.jp,Stefan.Baramov@trx.com,uv@upaya.co.uk,pauls@blubb.at,funkymonk_02@yahoo.com.au,brucelee_1969@yahoo.com.au,sendtoandy@gmail.com,robert.mcintosh@zurichna.com}

Separate message threads and convert each one to a list of message IDs:

Convert each message ID to a sender address, deleting the duplicates in each thread:

Find sender pairs in each thread, returning them as a flat list of edges:

Create the graph from the computed vertices and and distinct edges:

Define a function that labels each vertex, making the size increase with the number of other users that participate in threads with the given user:

Make the thickness of an edge proportional to the number of threads in which it appears:

Add these properties to the previous graph to get the completed sender graph:

External Links

Wolfram Reference - MBOX

Publisher Information

Contributed by: Wolfram Staff

Wolfram Language Example Repository

Analyze an MBOX Email Archive

External Links

See Also

Publisher Information

Analyze an MBOX Email Archive

External Links

See Also

Related Symbols

Publisher Information