Package opennlp.tools.tokenize
Class DefaultTokenContextGenerator
java.lang.Object
opennlp.tools.tokenize.DefaultTokenContextGenerator
- All Implemented Interfaces:
TokenContextGenerator
A default
TokenContextGenerator which produces events for maxent decisions
for tokenization.-
Constructor Summary
ConstructorsConstructorDescriptionInitializes a plainDefaultTokenContextGeneratorinstance.DefaultTokenContextGenerator(Set<String> inducedAbbreviations) Initializes a customizedDefaultTokenContextGeneratorinstance via a set ofinducedAbbreviations. -
Method Summary
-
Constructor Details
-
DefaultTokenContextGenerator
public DefaultTokenContextGenerator()Initializes a plainDefaultTokenContextGeneratorinstance. -
DefaultTokenContextGenerator
Initializes a customizedDefaultTokenContextGeneratorinstance via a set ofinducedAbbreviations.- Parameters:
inducedAbbreviations- The induced abbreviations to be used for this instance.
-
-
Method Details
-
getContext
- Specified by:
getContextin interfaceTokenContextGenerator- Parameters:
sentence- The string that represents a sentence.index- The index to consider splitting tokens.- Returns:
- An array of features for a
sentenceat the specifiedindex.
-