TokenizerImplEng (Aladin-bot-back-end 0.2.4 API)

java.lang.Object
- net.thexcoders.aladin_bot_backend.nlp_models.NLPModel
- - net.thexcoders.aladin_bot_backend.nlp_models.tokenizer.TokenizerImplEng

All Implemented Interfaces:

Tokenizer
```
public class TokenizerImplEng
extends NLPModel
implements Tokenizer
```
The Tokenizer splits each sentence into an array of tokens that will be used to get the associated Category It is highly linked to Categorizer

- Field Summary
  - Fields inherited from class net.thexcoders.aladin_bot_backend.nlp_models.NLPModel
    fileName, PATH
- Constructor Summary
  
  Constructors
  Constructor Description
  
  TokenizerImplEng()
- Method Summary
  
  All Methods Instance Methods Concrete Methods
  Modifier and Type Method Description
  
  String[] tokenize(String input)
  - Methods inherited from class net.thexcoders.aladin_bot_backend.nlp_models.NLPModel
    fileExists, generateFile
  - Methods inherited from class java.lang.Object
    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - TokenizerImplEng
```
public TokenizerImplEng()
```
- Method Detail
  - tokenize
```
public String[] tokenize(String input)
```
    Specified by:
    
    tokenize in interface Tokenizer