I tossed in the removal of an unused variable warning in icu_tokenizer.cpp and also some tokenizer clean-up rather than create a whole new branch.
I tossed in the removal of an unused variable warning in icu_tokenizer.cpp and also some tokenizer clean-up rather than create a whole new branch.