cc.redpen.tokenizer

Class WhiteSpaceTokenizer

java.lang.Object

cc.redpen.tokenizer.WhiteSpaceTokenizer

All Implemented Interfaces:

RedPenTokenizer

public class WhiteSpaceTokenizer
extends Object
implements RedPenTokenizer

Constructor Summary

Constructors

Constructor and Description
WhiteSpaceTokenizer()

Method Summary

Modifier and Type	Method and Description
List<TokenElement>	tokenize(String content) Tokenize input sentence into tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WhiteSpaceTokenizer

public WhiteSpaceTokenizer()

Method Detail

tokenize

public List<TokenElement> tokenize(String content)

Description copied from interface: RedPenTokenizer

Tokenize input sentence into tokens.

Specified by:

tokenize in interface RedPenTokenizer

Parameters:

content - input sentence

Returns:

a set of tokens in the input sentence