Package net.thexcoders.aladin_bot_backend.nlp_models.tokenizer

Interface Tokenizer

All Known Implementing Classes:

TokenizerImplEng, TokenizerImplFra

public interface Tokenizer

The Tokenizer splits each sentence into an array of tokens that will be used to get the associated Category It is highly linked to Categorizer

Method Summary

Modifier and Type	Method	Description
String[]	tokenize(String input)

Method Detail

tokenize

String[] tokenize(String input)