Do not include heavy Japanese tokenizer by default

valeriansaliou · Sep 4, 2023 · 6f0efef · 6f0efef
1 parent b0a02ba
commit 6f0efef
Show file tree

Hide file tree

Showing 2 changed files with 3 additions and 1 deletion.
diff --git a/Cargo.toml b/Cargo.toml
@@ -50,7 +50,7 @@ tikv-jemallocator = { version = "0.4", optional = true }
 winapi = { version = "0.3", features = ["minwindef", "consoleapi"] }
 
 [features]
-default = ["allocator-jemalloc", "tokenizer-chinese", "tokenizer-japanese"]
+default = ["allocator-jemalloc", "tokenizer-chinese"]
 allocator-jemalloc = ["tikv-jemallocator"]
 tokenizer-chinese = ["jieba-rs"]
 tokenizer-japanese = ["lindera-core", "lindera-dictionary", "lindera-tokenizer"]

diff --git a/README.md b/README.md
@@ -75,6 +75,8 @@ You can find the built binaries in the `./target/release` directory.
 
 _Install `build-essential`, `clang`, `libclang-dev`, `libc6-dev`, `g++` and `llvm-dev` to be able to compile the required RocksDB dependency._
 
+Note that the following optional features can be enabled upon building Sonic: `allocator-jemalloc`, `tokenizer-chinese` and `tokenizer-japanese` (some might be already enabled by default).
+
 **👉 Install from Cargo:**
 
 You can install Sonic directly with `cargo install`: