3.0 Module 3: A Deep Dive into the OpenNLP API
3.1 Introduction: Understanding the Core Classes and Methods
This module serves as a technical reference to the core Application Programming Interface (API) of Apache OpenNLP. Before we can implement the NLP tasks discussed in later modules, it is essential to understand the primary classes and methods that provide this functionality. This section will catalog the key API components for each major NLP task, providing a clear map of the toolkit’s structure. We will cover the specific packages, model classes, and operational classes for sentence detection, tokenization, named entity recognition, part-of-speech tagging, parsing, and chunking.
3.2 API for Sentence Detection
The API components for sentence detection are located in the opennlp.tools.sentdetect package.
- SentenceModel Class: This class is responsible for loading and representing the pre-trained statistical model used for sentence detection (e.g., en-sent.bin). Its constructor requires an InputStream object that points to the model file, which allows the model data to be read into memory.
- SentenceDetectorME Class: This is the primary operational class for sentence segmentation. It utilizes a maximum entropy statistical model to accurately identify sentence boundaries within a raw text string. It analyzes characters like periods and question marks in context to disambiguate their meaning.
| Method | Description |
| sentDetect() | This method takes a raw text String as input and splits it into individual sentences. It returns a String array where each element is a detected sentence. |
| sentPosDetect() | This method also processes a raw text String, but instead of returning the sentence text, it returns the character positions (start and end indices) of each sentence. The result is an array of Span objects. |
| getSentenceProbabilities() | After a call to sentDetect(), this method can be invoked to retrieve the confidence scores for each sentence boundary decision. It returns an array of double values corresponding to the probabilities of the detected sentences. |
3.3 API for Tokenization
The components for tokenization are found within the opennlp.tools.tokenizer package.
- TokenizerModel Class: Similar to its sentence detection counterpart, this class represents the pre-trained tokenization model (e.g., en-token.bin) and is instantiated using an InputStream of the model file.
- Tokenizer Classes: OpenNLP offers several classes for tokenization, each employing a different strategy:
- SimpleTokenizer: A rule-based tokenizer that splits text based on character classes (e.g., separating letters from punctuation).
- WhitespaceTokenizer: A simpler rule-based tokenizer that uses only whitespaces (spaces, tabs, newlines) as delimiters.
- TokenizerME: The most sophisticated of the three, this class uses a maximum entropy model to perform tokenization, allowing it to make more nuanced decisions based on learned patterns.
| Method | Description |
| tokenize() | This method takes a raw text String as input and breaks it down into individual tokens. It returns the result as a String array. |
| tokenizePos() | This method identifies the character positions or spans of each token within the original string, returning the result as an array of Span objects. |
Note: The TokenizerME class includes an additional method, getTokenProbabilities(), which returns the confidence probabilities associated with the tokenization decisions made during the most recent call to tokenize().
3.4 API for Named Entity Recognition
The API for Named Entity Recognition (NER) is located in the opennlp.tools.namefind package.
- TokenNameFinderModel Class: This class represents a pre-trained NER model. OpenNLP provides several of these, each trained for a specific entity type (e.g., en-ner-person.bin, en-ner-location.bin). It is instantiated with an InputStream of the chosen model file.
- NameFinderME Class: This is the core class for NER. It uses a maximum entropy model to scan a sequence of tokens and identify spans that correspond to named entities of the type defined by the loaded model.
| Method | Description |
| find() | This method takes an array of String tokens as input and detects the named entities within that sequence. It returns an array of Span objects, where each span indicates the indices of the tokens that form a named entity. |
| probs() | After a find() operation, this method can be called to retrieve the probabilities associated with the sequence of entity tags that were just assigned. |
3.5 API for Part-of-Speech (POS) Tagging
The components for POS tagging reside in the opennlp.tools.postag package.
- POSModel Class: This class loads and represents the pre-trained model for POS tagging (e.g., en-pos-maxent.bin) from an InputStream.
- POSTaggerME Class: This class is responsible for assigning a part-of-speech tag (e.g., noun, verb, adjective) to each token in a sentence. It uses a maximum entropy model to predict the most likely tag for each word given its context.
| Method | Description |
| tag() | This method accepts an array of String tokens as input and returns a corresponding array of String POS tags. |
| probs() | This method retrieves the confidence probabilities for each tag assigned during the most recent call to the tag() method. |
3.6 API for Syntactic Parsing
The API components for parsing are distributed across the opennlp.tools.parser and opennlp.tools.cmdline.parser packages.
- ParserModel Class (opennlp.tools.parser): Represents the pre-trained model used for syntactic parsing (e.g., en-parser-chunking.bin), loaded via an InputStream.
- ParserFactory Class (opennlp.tools.parser): This factory class is used to instantiate a Parser object from a loaded ParserModel. Its static create() method is the primary entry point for creating a parser instance.
- ParserTool Class (opennlp.tools.cmdline.parser): This utility class provides a high-level method for parsing text.
Its key method is parseLine(), a static method used to parse an entire line of text. It accepts three parameters: the String to be parsed, the Parser object to use, and an integer specifying the number of top-scoring parses to return.
3.7 API for Chunking
The components for chunking, or shallow parsing, are located in the opennlp.tools.chunker package.
- ChunkerModel Class: This class represents the pre-trained model for chunking (e.g., en-chunker.bin) and is instantiated from an InputStream.
- ChunkerME Class: This class uses a maximum entropy model to segment a sentence into syntactically correlated parts of words, such as Noun Phrases and Verb Phrases.
| Method | Description |
| chunk() | This method generates chunks for a given sequence. It requires two input arrays: an array of tokens and a corresponding array of their POS tags. |
| probs() | This method returns the probabilities associated with the sequence of chunk tags assigned during the most recent chunk() operation. |
With this overview of the core API, we are now ready to move on to practical implementation. Our first task will be to use these components to perform Sentence Detection.