FaizonZaman/ LexicalCases

Extract lexical patterns from text

Contributed by: Faizon Zaman

Installation Instructions

To install this paclet in your Wolfram Language environment, evaluate this code:
PacletInstall["FaizonZaman/LexicalCases"]

To load the code after installation, evaluate this code:
Needs["FaizonZaman`LexicalCases`"]

Details

This functionality came about when I needed to find adjectives before certain phrases. I initially gathered article text from Wikipedia, then used TextCases and TextContents with the Containing wrapper, (Containing["AdjectivePhrase",Verbatim["music"]] for example), but it turned out to be slow. So I implemented some lexical pattern objects for use in StringExpression's. This way I can search for the string pattern with StringCases.

TextCases is used internally to extract examples of a content type from the source text. These examples then replace the TextType in the StringExpression. You can see the lexical pattern generated from a piece of text with ExpandPattern. Note that this means, for example, if you have TextType["Adjective"] in your lexical pattern, the token will be replaced by all text snippets identified as adjectives in the source text.

I'd also like to say thanks to Swastik Banerjee for helping me improve SearchIndexObject support!