Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support multiple etymologies #893

Open
wants to merge 55 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
ec62312
reformat code
qicz Jul 7, 2021
cf705e2
optimize code
qicz Jul 8, 2021
64c43a7
crlf to lf
qicz Jul 8, 2021
fd920fc
update config to yml
qicz Jul 9, 2021
a843449
update grant
qicz Jul 9, 2021
8d39443
update yml read logic
qicz Jul 9, 2021
f730a79
optimize code
qicz Jul 10, 2021
0d42c2d
optimize configuration
qicz Jul 10, 2021
df31678
optimize configuration properties
qicz Jul 10, 2021
ed59f32
refactor configuration properties
qicz Jul 11, 2021
3fc5545
refactor configuration properties
qicz Jul 11, 2021
497adf5
rename reload dict
qicz Jul 11, 2021
e61efef
add mysql and redis config
qicz Jul 11, 2021
5a62761
add remote dictionary logic
qicz Jul 11, 2021
222c606
export properties
qicz Jul 11, 2021
a2f698e
optimize dictionary & helper
qicz Jul 11, 2021
bc019df
refactor remote dictionary
qicz Jul 11, 2021
cffb603
fix remote dictionary inital logic
qicz Jul 11, 2021
8a9569a
typo
qicz Jul 11, 2021
2f42e53
optimize dictionary
qicz Jul 11, 2021
9ac1906
mysql remote dictionary implementation
qicz Jul 12, 2021
cca6af1
redis remote dictionary implementation
qicz Jul 12, 2021
c6f7843
fix http schema remote dictionary bug & optimize redis remote dictionary
qicz Jul 12, 2021
baee3e6
update readme
qicz Jul 12, 2021
a64a303
update readme
qicz Jul 12, 2021
b04baad
refactor dictionary logic
qicz Jul 12, 2021
4879120
typo
qicz Jul 12, 2021
b26cd88
optimize configuration
qicz Jul 12, 2021
922e77d
refactor remote dictionary schema logic
qicz Jul 12, 2021
f141420
support custom domain and etymology
qicz Jul 13, 2021
b3f364d
refactor dictionary logic
qicz Jul 13, 2021
4135faa
update readme
qicz Jul 13, 2021
3833fab
typo
qicz Jul 13, 2021
dc612cf
format readme
qicz Jul 13, 2021
6d7187c
fix default-domain logic
qicz Jul 13, 2021
2d6a813
check etymology
qicz Jul 13, 2021
7717197
format log
qicz Jul 13, 2021
8a5b208
refactor mysql remote dictionary logic
qicz Jul 14, 2021
334c670
link starter
qicz Jul 14, 2021
60d5812
import redip
qicz Jul 15, 2021
b4fc23d
update log
qicz Jul 15, 2021
0a080d1
using released redip
qicz Jul 15, 2021
78ac658
using AssertKit
qicz Jul 15, 2021
03e94df
update redip
qicz Jul 16, 2021
fd42a35
optimize dictionary reload logic
qicz Jul 16, 2021
42cd120
update testing
qicz Jul 16, 2021
de55714
testing & using redip 1.0.2, redis remote dictionary using zset store…
qicz Jul 16, 2021
e7961da
using enableMonitor function variable
qicz Jul 16, 2021
c00362f
update test logging
qicz Jul 16, 2021
a0b8d17
update redip1.0.3: add remote dictionary shutdown hook, close the res…
qicz Jul 19, 2021
bb4b9b2
adaptive ik xml configuration
qicz Jul 22, 2021
ad6a830
optimize import
qicz Jul 22, 2021
c87f744
update readme
qicz Jul 22, 2021
8c51480
fix readme bug
qicz Jul 22, 2021
bae22f7
support redis cluster
qicz Aug 24, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
using released redip
qicz committed Jul 15, 2021
commit 0a080d17a9455ce7125772765a8261426a28e360
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -107,7 +107,7 @@ CREATE TABLE `ik_words` (
`domain` varchar(100) NOT NULL COMMENT '所属领域',
`create_time` datetime NOT NULL COMMENT '创建时间',
PRIMARY KEY (`id`),
UNIQUE KEY `domain_word` (`word`,`domain`) USING BTREE
KEY `domain` (`domain`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

SET FOREIGN_KEY_CHECKS = 1;
2 changes: 1 addition & 1 deletion config/redip.sql
Original file line number Diff line number Diff line change
@@ -30,7 +30,7 @@ CREATE TABLE `ik_words` (
`domain` varchar(100) NOT NULL COMMENT '所属领域',
`create_time` datetime NOT NULL COMMENT '创建时间',
PRIMARY KEY (`id`),
UNIQUE KEY `domain` (`domain`) USING BTREE
KEY `domain` (`domain`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

SET FOREIGN_KEY_CHECKS = 1;
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
@@ -91,7 +91,7 @@
<dependency>
<groupId>org.openingo.kits</groupId>
<artifactId>redip</artifactId>
<version>1.0-SNAPSHOT</version>
<version>1.0</version>
</dependency>

<dependency>
6 changes: 3 additions & 3 deletions src/main/java/org/wltea/analyzer/dictionary/Dictionary.java
Original file line number Diff line number Diff line change
@@ -193,7 +193,7 @@ private void loadMainDict() {
Path file = Configuration.getBaseOnDictRoot(Dictionary.PATH_DIC_MAIN);
this.mainDictionary.fillSegment(file, "Main DictFile");
// 加载扩展词典
List<String> mainExtDictFiles = Configuration.getProperties().getMainExtDictFiles();
List<String> mainExtDictFiles = Configuration.getProperties().getLocalMainExtDictFiles();
this.loadLocalExtDict(this.mainDictionary, mainExtDictFiles, "Main Extra DictFile");

// 加载远程自定义词库
@@ -222,7 +222,7 @@ private void loadStopWordDict() {
this.stopWordsDictionary.fillSegment(file, "Main Stopwords");

// 加载扩展停止词典
List<String> extStopDictFiles = Configuration.getProperties().getExtStopDictFiles();
List<String> extStopDictFiles = Configuration.getProperties().getLocalStopExtDictFiles();
this.loadLocalExtDict(this.stopWordsDictionary, extStopDictFiles, "Extra Stopwords");

// 加载远程停用词典
@@ -246,7 +246,7 @@ private void loadRemoteExtDict(DictSegment dictSegment,
DictionaryType dictionaryType) {
log.info("[Remote DictFile Loading] for domain '{}'", this.domainUri);
SpecialPermission.check();
Set<String> remoteWords = RemoteDictionary.getRemoteWords(this, dictionaryType, this.domainUri);
Set<String> remoteWords = RemoteDictionary.getRemoteWords(dictionaryType, this.domainUri);
// 如果找不到扩展的字典,则忽略
if (remoteWords.isEmpty()) {
log.info("[Remote DictFile Loading] no new words for '{}'", this.domainUri);