创作目的

特性

变更日志

更多资料

敏感词控台

敏感词标签文件

支持开源

快速开始

准备

Maven 引入

<dependency>    <groupId>com.github.houbb</groupId>    <artifactId>sensitive-word</artifactId>    <version>0.25.0</version></dependency>

核心方法

判断是否包含敏感词

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";Assert.assertTrue(SensitiveWordHelper.contains(text));

返回第一个敏感词

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";Stringword =SensitiveWordHelper.findFirst(text);Assert.assertEquals("五星红旗",word);

Stringword =SensitiveWordHelper.findFirst(text,WordResultHandlers.word());

返回所有敏感词

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";List<String>wordList =SensitiveWordHelper.findAll(text);Assert.assertEquals("[五星红旗, 毛主席, 天安门]",wordList.toString());

List<String>wordList =SensitiveWordHelper.findAll(text,WordResultHandlers.word());

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";// 默认敏感词标签为空List<WordTagsDto>wordList1 =SensitiveWordHelper.findAll(text,WordResultHandlers.wordTags());Assert.assertEquals("[WordTagsDto{word='五星红旗', tags=[]}, WordTagsDto{word='毛主席', tags=[]}, WordTagsDto{word='天安门', tags=[]}]",wordList1.toString());

默认的替换策略

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";Stringresult =SensitiveWordHelper.replace(text);Assert.assertEquals("****迎风飘扬，***的画像屹立在***前。",result);

指定替换的内容

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";Stringresult =SensitiveWordHelper.replace(text,'0');Assert.assertEquals("0000迎风飘扬，000的画像屹立在000前。",result);

自定义替换策略

/** * 自定替换策略 * @since 0.2.0 */@TestpublicvoiddefineReplaceTest() {finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";ISensitiveWordReplacereplace =newMySensitiveWordReplace();Stringresult =SensitiveWordHelper.replace(text,replace);Assert.assertEquals("国家旗帜迎风飘扬，教员的画像屹立在***前。",result);}

publicclassMyWordReplaceimplementsIWordReplace {@Overridepublicvoidreplace(StringBuilderstringBuilder,finalchar[]rawChars,IWordResultwordResult,IWordContextwordContext) {StringsensitiveWord =InnerWordCharUtils.getString(rawChars,wordResult);// 自定义不同的敏感词替换策略，可以从数据库等地方读取if("五星红旗".equals(sensitiveWord)) {stringBuilder.append("国家旗帜");        }elseif("毛主席".equals(sensitiveWord)) {stringBuilder.append("教员");        }else {// 其他默认使用 * 代替intwordLength =wordResult.endIndex() -wordResult.startIndex();for(inti =0;i <wordLength;i++) {stringBuilder.append('*');            }        }    }}

IWordResultHandler 结果处理类

使用实例

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";List<String>wordList =SensitiveWordHelper.findAll(text);Assert.assertEquals("[五星红旗, 毛主席, 天安门]",wordList.toString());List<String>wordList2 =SensitiveWordHelper.findAll(text,WordResultHandlers.word());Assert.assertEquals("[五星红旗, 毛主席, 天安门]",wordList2.toString());List<IWordResult>wordList3 =SensitiveWordHelper.findAll(text,WordResultHandlers.raw());Assert.assertEquals("[WordResult{startIndex=0, endIndex=4}, WordResult{startIndex=9, endIndex=12}, WordResult{startIndex=18, endIndex=21}]",wordList3.toString());

finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";// 默认敏感词标签为空List<WordTagsDto>wordList1 =SensitiveWordHelper.findAll(text,WordResultHandlers.wordTags());Assert.assertEquals("[WordTagsDto{word='五星红旗', tags=[]}, WordTagsDto{word='毛主席', tags=[]}, WordTagsDto{word='天安门', tags=[]}]",wordList1.toString());List<WordTagsDto>wordList2 =SensitiveWordBs.newInstance()        .wordTag(WordTags.file("dict_tag_test.txt"))        .init()        .findAll(text,WordResultHandlers.wordTags());Assert.assertEquals("[WordTagsDto{word='五星红旗', tags=[政治, 国家]}, WordTagsDto{word='毛主席', tags=[政治, 伟人, 国家]}, WordTagsDto{word='天安门', tags=[政治, 国家, 地址]}]",wordList2.toString());

更多特性

样式处理

忽略大小写

finalStringtext ="fuCK the bad words.";Stringword =SensitiveWordHelper.findFirst(text);Assert.assertEquals("fuCK",word);

忽略半角圆角

finalStringtext ="ｆｕｃｋ the bad words.";Stringword =SensitiveWordHelper.findFirst(text);Assert.assertEquals("ｆｕｃｋ",word);

忽略数字的写法

finalStringtext ="这个是我的微信：9⓿二肆⁹₈③⑸⒋➃㈤㊄";List<String>wordList =SensitiveWordBs.newInstance().enableNumCheck(true).init().findAll(text);Assert.assertEquals("[9⓿二肆⁹₈③⑸⒋➃㈤㊄]",wordList.toString());

忽略繁简体

finalStringtext ="我爱我的祖国和五星紅旗。";List<String>wordList =SensitiveWordHelper.findAll(text);Assert.assertEquals("[五星紅旗]",wordList.toString());

忽略英文的书写格式

finalStringtext ="Ⓕⓤc⒦ the bad words";List<String>wordList =SensitiveWordHelper.findAll(text);Assert.assertEquals("[Ⓕⓤc⒦]",wordList.toString());

忽略重复词

finalStringtext ="ⒻⒻⒻfⓤuⓤ⒰cⓒ⒦ the bad words";List<String>wordList =SensitiveWordBs.newInstance()        .ignoreRepeat(true)        .init()        .findAll(text);Assert.assertEquals("[ⒻⒻⒻfⓤuⓤ⒰cⓒ⒦]",wordList.toString());

更多检测策略

说明

邮箱检测

finalStringtext ="楼主好人，邮箱 sensitiveword@xx.com";List<String>wordList =SensitiveWordBs.newInstance().enableEmailCheck(true).init().findAll(text);Assert.assertEquals("[sensitiveword@xx.com]",wordList.toString());

连续数字检测

finalStringtext ="你懂得：12345678";// 默认检测 8 位List<String>wordList =SensitiveWordBs.newInstance().enableNumCheck(true).init().findAll(text);Assert.assertEquals("[12345678]",wordList.toString());// 指定数字的长度，避免误杀List<String>wordList2 =SensitiveWordBs.newInstance().enableNumCheck(true).numCheckLen(9).init().findAll(text);Assert.assertEquals("[]",wordList2.toString());

网址检测

finalStringtext ="点击链接 https://www.baidu.com 查看答案";finalSensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance().enableUrlCheck(true).init();List<String>wordList =sensitiveWordBs.findAll(text);Assert.assertEquals("[https://www.baidu.com]",wordList.toString());Assert.assertEquals("点击链接 ********************* 查看答案",sensitiveWordBs.replace(text));

finalStringtext ="点击链接 https://www.baidu.com 查看答案，当然也可以是 baidu.com、www.baidu.com";finalSensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance()        .enableUrlCheck(true)// 启用URL检测        .wordCheckUrl(WordChecks.urlNoPrefix())//指定检测的方式        .init();List<String>wordList =sensitiveWordBs.findAll(text);Assert.assertEquals("[www.baidu.com, baidu.com, www.baidu.com]",wordList.toString());Assert.assertEquals("点击链接 https://************* 查看答案，当然也可以是 *********、*************",sensitiveWordBs.replace(text));

IPV4 检测

finalStringtext ="个人网站，如果网址打不开可以访问 127.0.0.1。";finalSensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance().enableIpv4Check(true).init();List<String>wordList =sensitiveWordBs.findAll(text);Assert.assertEquals("[127.0.0.1]",wordList.toString());

引导类特性配置

说明

配置方法

SensitiveWordBswordBs =SensitiveWordBs.newInstance()        .ignoreCase(true)        .ignoreWidth(true)        .ignoreNumStyle(true)        .ignoreChineseStyle(true)        .ignoreEnglishStyle(true)        .ignoreRepeat(false)        .enableNumCheck(false)        .enableEmailCheck(false)        .enableUrlCheck(false)        .enableIpv4Check(false)        .enableWordCheck(true)        .wordCheckNum(WordChecks.num())        .wordCheckEmail(WordChecks.email())        .wordCheckUrl(WordChecks.url())        .wordCheckIpv4(WordChecks.ipv4())        .wordCheckWord(WordChecks.word())        .numCheckLen(8)        .wordTag(WordTags.none())        .charIgnore(SensitiveWordCharIgnores.defaults())        .wordResultCondition(WordResultConditions.alwaysTrue())        .init();finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";Assert.assertTrue(wordBs.contains(text));

配置说明

内存资源的释放

SensitiveWordBswordBs =SensitiveWordBs.newInstance()                .init();// 后续因为一些原因移除了对应信息，希望释放内存。wordBs.destroy();

针对单个黑名单词的新增/删除，无需全量初始化

方法说明

实例代码：

finalStringtext ="测试一下新增敏感词，验证一下删除和新增对不对";SensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance()        .wordAllow(WordAllows.empty())        .wordDeny(WordDenys.empty())        .init();// 当前Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());// 新增单个sensitiveWordBs.addWord("测试");sensitiveWordBs.addWord("新增");Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());// 删除单个sensitiveWordBs.removeWord("新增");Assert.assertEquals("[测试]",sensitiveWordBs.findAll(text).toString());sensitiveWordBs.removeWord("测试");Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());// 新增集合sensitiveWordBs.addWord(Arrays.asList("新增","测试"));Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());// 删除集合sensitiveWordBs.removeWord(Arrays.asList("新增","测试"));Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());// 新增数组sensitiveWordBs.addWord("新增","测试");Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());// 删除集合sensitiveWordBs.removeWord("新增","测试");Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());

针对单个白名单词的新增/删除，无需全量初始化

方法说明

使用例子

finalStringtext ="测试一下新增敏感词白名单，验证一下删除和新增对不对";SensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance()                        .wordAllow(WordAllows.empty())                        .wordDeny(newIWordDeny() {@OverridepublicList<String>deny() {returnArrays.asList("测试","新增");                            }                        })                        .init();// 当前Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());// 新增单个sensitiveWordBs.addWordAllow("测试");sensitiveWordBs.addWordAllow("新增");Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());// 删除单个sensitiveWordBs.removeWordAllow("测试");Assert.assertEquals("[测试]",sensitiveWordBs.findAll(text).toString());sensitiveWordBs.removeWordAllow("新增");Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());// 新增集合sensitiveWordBs.addWordAllow(Arrays.asList("新增","测试"));Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());// 删除集合sensitiveWordBs.removeWordAllow(Arrays.asList("新增","测试"));Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());// 新增数组sensitiveWordBs.addWordAllow("新增","测试");Assert.assertEquals("[]",sensitiveWordBs.findAll(text).toString());// 删除集合sensitiveWordBs.removeWordAllow("新增","测试");Assert.assertEquals("[测试, 新增, 新增]",sensitiveWordBs.findAll(text).toString());

全量初始化

说明

例子

@ComponentpublicclassSensitiveWordService {@AutowiredprivateSensitiveWordBssensitiveWordBs;/**     * 更新词库     *     * 每次数据库的信息发生变化之后，首先调用更新数据库敏感词库的方法。     * 如果需要生效，则调用这个方法。     *     * 说明：重新初始化不影响旧的方法使用。初始化完成后，会以新的为准。     */publicvoidrefresh() {// 每次数据库的信息发生变化之后，首先调用更新数据库敏感词库的方法，然后调用这个方法。sensitiveWordBs.init();    }}

wordResultCondition-针对匹配词进一步判断

说明

内置策略

使用例子

finalStringtext ="I have a nice day。";List<String>wordList =SensitiveWordBs.newInstance()        .wordDeny(newIWordDeny() {@OverridepublicList<String>deny() {returnCollections.singletonList("av");            }        })        .wordResultCondition(WordResultConditions.alwaysTrue())        .init()        .findAll(text);Assert.assertEquals("[av]",wordList.toString());

finalStringtext ="I have a nice day。";List<String>wordList =SensitiveWordBs.newInstance()        .wordDeny(newIWordDeny() {@OverridepublicList<String>deny() {returnCollections.singletonList("av");            }        })        .wordResultCondition(WordResultConditions.englishWordMatch())        .init()        .findAll(text);Assert.assertEquals("[]",wordList.toString());

wordTags 单词标签

/** * 自定义单词标签 * @since 0.23.0 */publicclassMyWordTagextendsAbstractWordTag {privatestaticMap<String,Set<String>>dataMap;static {dataMap =newHashMap<>();dataMap.put("商品",buildSet("广告","中文"));dataMap.put("AV",buildSet("色情","单词","英文"));    }privatestaticSet<String>buildSet(String...tags) {Set<String>set =newHashSet<>();for(Stringtag :tags) {set.add(tag);        }returnset;    }@OverrideprotectedSet<String>doGetTag(Stringword) {returndataMap.get(word);    }}

// 只关心SE情SensitiveWordBssensitiveWordBsYellow =SensitiveWordBs.newInstance()        .wordDeny(newIWordDeny() {@OverridepublicList<String>deny() {returnArrays.asList("商品","AV");            }        })        .wordAllow(WordAllows.empty())        .wordTag(newMyWordTag())        .wordResultCondition(WordResultConditions.wordTags(Arrays.asList("色情")))        .init();// 只关心广告SensitiveWordBssensitiveWordBsAd =SensitiveWordBs.newInstance()        .wordDeny(newIWordDeny() {@OverridepublicList<String>deny() {returnArrays.asList("商品","AV");            }        })        .wordAllow(WordAllows.empty())        .wordTag(newMyWordTag())        .wordResultCondition(WordResultConditions.wordTags(Arrays.asList("广告")))        .init();finalStringtext ="这些 AV 商品什么价格？";Assert.assertEquals("[AV]",sensitiveWordBsYellow.findAll(text).toString());Assert.assertEquals("[商品]",sensitiveWordBsAd.findAll(text).toString());

忽略字符

说明

例子

finalStringtext ="傻@冒，狗+东西";//默认因为有特殊字符分割，无法识别List<String>wordList =SensitiveWordBs.newInstance().init().findAll(text);Assert.assertEquals("[]",wordList.toString());// 指定忽略的字符策略，可自行实现。List<String>wordList2 =SensitiveWordBs.newInstance()        .charIgnore(SensitiveWordCharIgnores.specialChars())        .init()        .findAll(text);Assert.assertEquals("[傻@冒, 狗+东西]",wordList2.toString());

敏感词标签

说明

标签接口

publicinterfaceIWordTag {/**     * 查询标签列表     * @param word 脏词     * @return 结果     */Set<String>getTag(Stringword);}

内置实现

方法列表

格式约定

五星红旗 政治,国家

系统内置实现（默认效果）

SensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance().wordTag(WordTags.system()).init();Set<String>tagSet =sensitiveWordBs.tags("博彩");Assert.assertEquals("[3]",tagSet.toString());

0 政治1 毒品2 色情3 赌博4 违法

文件入门例子

finalStringpath ="~\\test\\resources\\dict_tag_test.txt";// 演示默认方法IWordTagwordTag =WordTags.file(path);SensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance()        .wordTag(wordTag)        .init();Set<String>tagSet =sensitiveWordBs.tags("零售");Assert.assertEquals("[广告, 网络]",tagSet.toString());// 演示指定分隔符IWordTagwordTag2 =WordTags.file(path," ",",");SensitiveWordBssensitiveWordBs2 =SensitiveWordBs.newInstance()        .wordTag(wordTag2)        .init();Set<String>tagSet2 =sensitiveWordBs2.tags("零售");Assert.assertEquals("[广告, 网络]",tagSet2.toString());

零售 广告,网络

单词标签和敏感词发现的联动

// 自定义测试标签类IWordTagwordTag =WordTags.lines(Arrays.asList("天安门 政治,国家,地址"));// 指定初始化SensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance()        .wordTag(wordTag)        .init()        ;List<WordTagsDto>wordTagsDtoList1 =sensitiveWordBs.findAll("天安门",WordResultHandlers.wordTags());Assert.assertEquals("[WordTagsDto{word='天安门', tags=[政治, 国家, 地址]}]",wordTagsDtoList1.toString());

动态加载（用户自定义）

情景说明

接口说明

IWordDeny

/** * 拒绝出现的数据-返回的内容被当做是敏感词 * @author binbin.hou * @since 0.0.13 */publicinterfaceIWordDeny {/**     * 获取结果     * @return 结果     * @since 0.0.13     */List<String>deny();}

publicclassMyWordDenyimplementsIWordDeny {@OverridepublicList<String>deny() {returnArrays.asList("我的自定义敏感词");    }}

IWordAllow

/** * 允许的内容-返回的内容不被当做敏感词 * @author binbin.hou * @since 0.0.13 */publicinterfaceIWordAllow {/**     * 获取结果     * @return 结果     * @since 0.0.13     */List<String>allow();}

publicclassMyWordAllowimplementsIWordAllow {@OverridepublicList<String>allow() {returnArrays.asList("五星红旗");    }}

配置使用

系统的默认配置

SensitiveWordBswordBs =SensitiveWordBs.newInstance()        .wordDeny(WordDenys.defaults())        .wordAllow(WordAllows.defaults())        .init();finalStringtext ="五星红旗迎风飘扬，毛主席的画像屹立在天安门前。";Assert.assertTrue(wordBs.contains(text));

指定自己的实现

Stringtext ="这是一个测试，我的自定义敏感词。";SensitiveWordBswordBs =SensitiveWordBs.newInstance()        .wordDeny(newMyWordDeny())        .wordAllow(newMyWordAllow())        .init();Assert.assertEquals("[我的自定义敏感词]",wordBs.findAll(text).toString());

同时配置多个

Stringtext ="这是一个测试。我的自定义敏感词。";IWordDenywordDeny =WordDenys.chains(WordDenys.defaults(),newMyWordDeny());IWordAllowwordAllow =WordAllows.chains(WordAllows.defaults(),newMyWordAllow());SensitiveWordBswordBs =SensitiveWordBs.newInstance()        .wordDeny(wordDeny)        .wordAllow(wordAllow)        .init();Assert.assertEquals("[我的自定义敏感词]",wordBs.findAll(text).toString());

spring 整合

背景

自定义数据源

@ConfigurationpublicclassSpringSensitiveWordConfig {@AutowiredprivateMyDdWordAllowmyDdWordAllow;@AutowiredprivateMyDdWordDenymyDdWordDeny;/**     * 初始化引导类     * @return 初始化引导类     * @since 1.0.0     */@BeanpublicSensitiveWordBssensitiveWordBs() {SensitiveWordBssensitiveWordBs =SensitiveWordBs.newInstance()                .wordAllow(WordAllows.chains(WordAllows.defaults(),myDdWordAllow))                .wordDeny(myDdWordDeny)// 各种其他配置                .init();returnsensitiveWordBs;    }}

Benchmark

环境

处理器12th Gen Intel(R) Core(TM) i7-1260P   2.10 GHz机带 RAM16.0 GB (15.7 GB 可用)系统类型64 位操作系统, 基于 x64 的处理器

测试效果记录

STAR

后期 road-map

拓展阅读

敏感词系列

NLP 开源矩阵