FaizonZaman/LexicalCases

(1.2.0) current version: 2.0.3 »

Extract lexical patterns from text

Contributed By: Faizon Zaman

Search text for lexical patterns. While similar to StringCases, TextCases etc., content types can be used freely in a StringExpression (as opposed to expressing complex lexical patterns in a Containing wrapper). Files, search index objects, and Wikipedia queries are supported. Results are returned in a summary object which supports several subvalues. Consult the documentation for usage.

Installation Instructions

To install this paclet in your Wolfram Language environment, evaluate this code:
PacletInstall["FaizonZaman/LexicalCases"]

Details

I developed this functionality for a particular work task (Lexical Programmer @ Wolfram|Alpha). I needed to find adjectives that precede certain phrases. I started by gathering article text from Wikipedia of relevant domains, then searched for the desired lexical sequences. The initial approach used TextCases and TextContents with the Containing wrapper, (Containing["AdjectivePhrase",Verbatim["music"]] for example), but it was slow. So I designed some lexical tokens I could use in a StringExpression that could be used with StringCases.

TextCases is used internally to extract examples of a content type from the source text. These examples then replace the TextType in the StringExpression. You can see the lexical pattern generated from a piece of text with ExpandPattern. Note that this means, for example, if you have TextType["Adjective"] in your lexical pattern, the token will be replaced by all text snippets identified as adjectives in the source text.

I’d also like to say thanks to Swastik Banerjee for helping me improve SearchIndexObject support!