Skip to content
Snippets Groups Projects
Commit 0197cc7d authored by Stephen D's avatar Stephen D
Browse files

better reaction tokenization

parent 119260e7
No related branches found
No related tags found
No related merge requests found
......@@ -42,7 +42,7 @@ impl LineHandler for ReactHandler {
.content
.chars()
.map(|c| match c {
'!' | '?' | ',' | '.' => ' ',
'!' | '?' | ',' | '.' | '(' | ')' | '[' | ']' | '\n' | '\r' => ' ',
_ => c,
})
.group_by(|c| *c == ' ');
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment