Package | Description |
---|---|
opennlp.tools.cmdline.tokenizer | |
opennlp.tools.tokenize |
Contains classes related to finding token or words in a string.
|
opennlp.uima.tokenize |
Package related to finding tokens or word segments.
|
Modifier and Type | Method and Description |
---|---|
protected TokenizerModel |
TokenizerModelLoader.loadModel(InputStream modelIn) |
Modifier and Type | Method and Description |
---|---|
static TokenizerModel |
TokenizerME.train(ObjectStream<TokenSample> samples,
TokenizerFactory factory,
TrainingParameters mlParams)
Trains a model for the
TokenizerME . |
static TokenizerModel |
TokenizerME.train(String languageCode,
ObjectStream<TokenSample> samples,
boolean useAlphaNumericOptimization)
Deprecated.
Use
#train(String, ObjectStream, TokenizerFactory, TrainingParameters)
and pass in a TokenizerFactory |
static TokenizerModel |
TokenizerME.train(String languageCode,
ObjectStream<TokenSample> samples,
boolean useAlphaNumericOptimization,
int cutoff,
int iterations)
Deprecated.
Use
#train(String, ObjectStream, TokenizerFactory, TrainingParameters)
and pass in a TokenizerFactory |
static TokenizerModel |
TokenizerME.train(String languageCode,
ObjectStream<TokenSample> samples,
boolean useAlphaNumericOptimization,
TrainingParameters mlParams)
Deprecated.
Use
#train(String, ObjectStream, TokenizerFactory, TrainingParameters)
and pass in a TokenizerFactory |
static TokenizerModel |
TokenizerME.train(String languageCode,
ObjectStream<TokenSample> samples,
Dictionary abbreviations,
boolean useAlphaNumericOptimization,
TrainingParameters mlParams)
Deprecated.
Use
#train(String, ObjectStream, TokenizerFactory, TrainingParameters)
and pass in a TokenizerFactory |
Constructor and Description |
---|
TokenizerME(TokenizerModel model) |
TokenizerME(TokenizerModel model,
Factory factory)
Deprecated.
use
TokenizerFactory to extend the Tokenizer
functionality |
Modifier and Type | Method and Description |
---|---|
TokenizerModel |
TokenizerModelResourceImpl.getModel() |
TokenizerModel |
TokenizerModelResource.getModel()
Retrieves the shared model instance.
|
protected TokenizerModel |
TokenizerModelResourceImpl.loadModel(InputStream in) |
Copyright © 2019 The Apache Software Foundation. All rights reserved.