For reference, the upstream recommendation for activating the CJK tokenizer. This is in the perspective of future optional tokenizer, and to avoid an API change while a new official release is not ready yet.
« Back to merge proposal
For reference, the upstream recommendation for activating the CJK tokenizer. This is in the perspective of future optional tokenizer, and to avoid an API change while a new official release is not ready yet.