Movatterモバイル変換

[0]ホーム
URL:

riot

packagemodule
v0.0.0-...-f4c30acLatest Latest
Warning

This package is not in the latest version of its module.

Go to latest Published: Oct 13, 2020 License:Apache-2.0Imports:24Imported by:133

Details

Repository

github.com/go-ego/riot

Links

README

Riot search

CircleCI StatusAppveyorcodecovBuild StatusGo Report CardGoDocGitHub release

Go Open Source, Distributed, Simple and efficient full text search engine.

简体中文

Features

Requirements

Go version >= 1.8

Dependencies

Riot uses go module or dep to manage dependencies.

Installation/Update

go get -u github.com/go-ego/riot

Build-tools

go get -u github.com/go-ego/re
re riot

To create a new riot application

$ re riot my-riotapp
re run

To run the application we just created, you can navigate to the application folder and execute:

$ cd my-riotapp && re run

Usage:

Look at an example
package mainimport ("log""github.com/go-ego/riot""github.com/go-ego/riot/types")var (// searcher is coroutine safesearcher = riot.Engine{})func main() {// Initsearcher.Init(types.EngineOpts{// Using:             4,NotUseGse: true,})defer searcher.Close()text := "Google Is Experimenting With Virtual Reality Advertising"text1 := `Google accidentally pushed Bluetooth update for Homespeaker early`text2 := `Google is testing another Search results layout with rounded cards, new colors, and the 4 mysterious colored dots again`// Add the document to the index, docId starts at 1searcher.Index("1", types.DocData{Content: text})searcher.Index("2", types.DocData{Content: text1}, false)searcher.IndexDoc("3", types.DocData{Content: text2}, true)// Wait for the index to refreshsearcher.Flush()// engine.FlushIndex()// The search output format is found in the types.SearchResp structurelog.Print(searcher.Search(types.SearchReq{Text:"google testing"}))}

It is very simple!

Use default engine:
package mainimport ("log""github.com/go-ego/riot""github.com/go-ego/riot/types")var (searcher = riot.New("zh"))func main() {data := types.DocData{Content: `I wonder how, I wonder why, I wonder where they are`}data1 := types.DocData{Content: "所以, 你好, 再见"}data2 := types.DocData{Content: "没有理由"}searcher.Index("1", data)searcher.Index("2", data1)searcher.Index("3", data2)searcher.Flush()req := types.SearchReq{Text: "你好"}search := searcher.Search(req)log.Println("search...", search)}
Look at more Examples
Look at Store example
Look at Logic search example
Look at Pinyin search example
Look at different dict and language search example
Look at benchmark example
Riot search engine templates, client and dictionaries

Authors

License

Riot is primarily distributed under the terms of the Apache License (Version 2.0), base onwukong.

Documentation

Overview

Package riot is riot engine

Package riot full text search engine

Index

Constants

View Source
const (// Version get the riot versionVersionstring = "v0.10.0.425, Danube River!"// NumNanosecondsInAMillisecond nano-seconds in a milli-second numNumNanosecondsInAMillisecond = 1000000// StoreFilePrefix persistent store file prefixStoreFilePrefix = "riot"// DefaultPath default db pathDefaultPath = "./riot-index")

Variables

View Source
var (// InitMemUsed init mem usedInitMemUseduint64// InitDiskUsed init disk usedInitDiskUseduint64)
View Source
var (Req1 =types.SearchReq{Text: reqText})
View Source
var (TestIndexOpts = rankEngineOpts(rankOptsMax10))

Functions

funcAddDocs

func AddDocs(engine *Engine)

funcAddDocsWithLabels

func AddDocsWithLabels(engine *Engine)

funcCPUInfo

func CPUInfo(args ...int) (string,error)

CPUInfo returns the cpu info

funcCPUPercent

func CPUPercent() ([]float64,error)

CPUPercent returns the amount of use cpu in percent.

funcDiskFree

func DiskFree() (uint64,error)

DiskFree returns the amount of free disk in bytes.

funcDiskPercent

func DiskPercent() (string,error)

DiskPercent returns the amount of use disk in percent.

funcDiskTotal

func DiskTotal() (uint64,error)

DiskTotal returns the amount of total disk in bytes.

funcDiskUsed

func DiskUsed() (uint64,error)

DiskUsed returns the amount of use disk in bytes.

funcGetVersion

func GetVersion()string

GetVersion get the riot version

funcKernelVer

func KernelVer() (string,error)

KernelVer returns the kernel version as a string.

funcMemFree

func MemFree() (uint64,error)

MemFree returns the amount of free memory in bytes.

funcMemPercent

func MemPercent() (string,error)

MemPercent returns the amount of use memory in percent.

funcMemTotal

func MemTotal() (uint64,error)

MemTotal returns the amount of total memory in bytes.

funcMemUsed

func MemUsed() (uint64,error)

MemUsed returns the amount of used memory in bytes.

funcOrderlessOpts

func OrderlessOpts(idOnlybool)types.EngineOpts

funcPlatform

func Platform() (string,error)

Platform returns the platform name and OS Version.

funcPlatformInfo

func PlatformInfo() (platform, family, osVersionstring, errerror)

PlatformInfo fetches system platform information.

funcToGB

func ToGB(datauint64)uint64

ToGB bytes to gb

funcToKB

func ToKB(datauint64)uint64

ToKB bytes to kb

funcToMB

func ToMB(datauint64)uint64

ToMB bytes to mb

funcTry

func Try(fun func(), handler func(interface{}))

Try handler(err)

funcUptime

func Uptime() (uptimeuint64, errerror)

Uptime returns the system uptime in seconds.

Types

typeEngine

type Engine struct {// contains filtered or unexported fields}

Engine initialize the engine

funcNew

func New(conf ...interface{}) *Engine

New create a new engine with mode

funcNewEngine

func NewEngine(conf ...interface{}) *Engine

NewEngine create a new engine

func (*Engine)CheckMem

func (engine *Engine) CheckMem()

CheckMem check the memory when the memory is largerthan 99.99% using the store

func (*Engine)Close

func (engine *Engine) Close()

Close close the engine关闭引擎

func (*Engine)Flush

func (engine *Engine) Flush()

Flush block wait until all indexes are added阻塞等待直到所有索引添加完毕

func (*Engine)FlushIndex

func (engine *Engine) FlushIndex()

FlushIndex block wait until all indexes are added阻塞等待直到所有索引添加完毕

func (*Engine)ForSplitData

func (engine *Engine) ForSplitData(strData []string, numint) (TMap,int)

ForSplitData for split segment's data, segspl

func (*Engine)GetAllDocIds

func (engine *Engine) GetAllDocIds() []string

GetAllDocIds get all the DocId from the storage databaseand return从数据库遍历所有的 DocId, 并返回

func (*Engine)GetDBAllDocs

func (engine *Engine) GetDBAllDocs() (docsId []string, docsData []types.DocData)

GetDBAllDocs get the db all docs

func (*Engine)GetDBAllIds

func (engine *Engine) GetDBAllIds() []string

GetDBAllIds get all the DocId from the storage databaseand return从数据库遍历所有的 DocId, 并返回

func (*Engine)HasDoc

func (engine *Engine) HasDoc(docIdstring)bool

HasDoc if the document is exist return true

func (*Engine)HasDocDB

func (engine *Engine) HasDocDB(docIdstring)bool

HasDocDB if the document is exist in the databasereturn true

func (*Engine)Index

func (engine *Engine) Index(docIdstring, datatypes.DocData,forceUpdate ...bool)

Index add the document to the index

func (*Engine)IndexDoc

func (engine *Engine) IndexDoc(docIdstring, datatypes.DocData,forceUpdate ...bool)

IndexDoc add the document to the index将文档加入索引

输入参数：

docId      标识文档编号，必须唯一，docId == 0 表示非法文档（用于强制刷新索引），[1, +oo) 表示合法文档data      见 DocIndexData 注释forceUpdate 是否强制刷新 cache，如果设为 true，则尽快添加到索引，否则等待 cache 满之后一次全量添加

注意：

  1. 这个函数是线程安全的，请尽可能并发调用以提高索引速度
  2. 这个函数调用是非同步的，也就是说在函数返回时有可能文档还没有加入索引中，因此如果立刻调用Search可能无法查询到这个文档。强制刷新索引请调用FlushIndex函数。

func (*Engine)Indexer

func (engine *Engine) Indexer(optionstypes.EngineOpts)

Indexer initialize the indexer channel

func (*Engine)Init

func (engine *Engine) Init(optionstypes.EngineOpts)

Init initialize the engine

func (*Engine)InitStore

func (engine *Engine) InitStore()

InitStore initialize the persistent store channel

func (*Engine)NotTimeOut

func (engine *Engine) NotTimeOut(requesttypes.SearchReq,rankerReturnChan chan rankerReturnReq) (rankOutArr interface{}, numDocsint)

NotTimeOut not set engine timeout

func (*Engine)NumDocsIndexed

func (engine *Engine) NumDocsIndexed()uint64

NumDocsIndexed documents indexed number, deprecated

func (*Engine)NumDocsRemoved

func (engine *Engine) NumDocsRemoved()uint64

NumDocsRemoved documents removed number, deprecated

func (*Engine)NumIndexed

func (engine *Engine) NumIndexed()uint64

NumIndexed documents indexed number

func (*Engine)NumRemoved

func (engine *Engine) NumRemoved()uint64

NumRemoved documents removed number

func (*Engine)NumTokenAdded

func (engine *Engine) NumTokenAdded()uint64

NumTokenAdded added token index number

func (*Engine)NumTokenIndexAdded

func (engine *Engine) NumTokenIndexAdded()uint64

NumTokenIndexAdded added token index number, deprecated

func (*Engine)PinYin

func (engine *Engine) PinYin(hansstring) []string

PinYin get the Chinese alphabet and abbreviation

func (*Engine)RankID

func (engine *Engine) RankID(requesttypes.SearchReq, rankOptstypes.RankOpts,tokens []string, rankerReturnChan chan rankerReturnReq) (outputtypes.SearchResp)

RankID rank docs by types.ScoredIDs

func (*Engine)Ranker

func (engine *Engine) Ranker(optionstypes.EngineOpts)

Ranker initialize the ranker channel

func (*Engine)Ranks

func (engine *Engine) Ranks(requesttypes.SearchReq, rankOptstypes.RankOpts,tokens []string, rankerReturnChan chan rankerReturnReq) (outputtypes.SearchResp)

Ranks rank docs by types.ScoredDocs

func (*Engine)RemoveDoc

func (engine *Engine) RemoveDoc(docIdstring, forceUpdate ...bool)

RemoveDoc remove the document from the index将文档从索引中删除

输入参数：

docId      标识文档编号，必须唯一，docId == 0 表示非法文档（用于强制刷新索引），[1, +oo) 表示合法文档forceUpdate 是否强制刷新 cache，如果设为 true，则尽快删除索引，否则等待 cache 满之后一次全量删除

注意：

  1. 这个函数是线程安全的，请尽可能并发调用以提高索引速度
  2. 这个函数调用是非同步的，也就是说在函数返回时有可能文档还没有加入索引中，因此如果立刻调用 Search 可能无法查询到这个文档。强制刷新索引请调用 FlushIndex 函数。

func (*Engine)Search

func (engine *Engine) Search(requesttypes.SearchReq) (outputtypes.SearchResp)

Search find the document that satisfies the search criteria.This function is thread safe查找满足搜索条件的文档，此函数线程安全

func (*Engine)SearchDoc

func (engine *Engine) SearchDoc(requesttypes.SearchReq) (outputtypes.SearchDoc)

SearchDoc find the document that satisfies the search criteria.This function is thread safe, return not IDonly

func (*Engine)SearchID

func (engine *Engine) SearchID(requesttypes.SearchReq) (outputtypes.SearchID)

SearchID find the document that satisfies the search criteria.This function is thread safe, return IDonly

func (*Engine)Segment

func (engine *Engine) Segment(contentstring) (keywords []string)

Segment get the word segmentation result of the text获取文本的分词结果, 只分词与过滤弃用词

func (*Engine)Store

func (engine *Engine) Store()

Store start the persistent store work connection

func (*Engine)TimeOut

func (engine *Engine) TimeOut(requesttypes.SearchReq,rankerReturnChan chan rankerReturnReq) (rankOutArr interface{}, numDocsint, isTimeoutbool)

TimeOut set engine timeout

func (*Engine)Tokens

func (engine *Engine) Tokens(requesttypes.SearchReq) (tokens []string)

Tokens get the engine tokens

func (*Engine)UsedDisk

func (engine *Engine) UsedDisk() (uint64,error)

UsedDisk returns the amount of use disk in bytesafter init() func.

func (*Engine)UsedMem

func (engine *Engine) UsedMem() (uint64,error)

UsedMem returns the amount of riot used memory in bytesafter init() func.

func (*Engine)WithGse

func (engine *Engine) WithGse(segmentergse.Segmenter) *Engine

WithGse Using user defined segmenterIf using a not nil segmenter and the dictionary is loaded,the `opt.GseDict` will be ignore.

typeRankByTokenProximity

type RankByTokenProximity struct {}

func (RankByTokenProximity)Score

func (ruleRankByTokenProximity) Score(doctypes.IndexedDoc, fields interface{}) []float32

typeScoringFields

type ScoringFields struct {A, B, Cfloat32}

typeStopTokens

type StopTokens struct {// contains filtered or unexported fields}

StopTokens stop tokens map

func (*StopTokens)Init

func (st *StopTokens) Init(stopTokenFilestring)

Init 从 stopTokenFile 中读入停用词，一个词一行文档索引建立时会跳过这些停用词

func (*StopTokens)IsStopToken

func (st *StopTokens) IsStopToken(tokenstring)bool

IsStopToken to determine whether to stop token

typeTMap

type TMap map[string][]int

TMap defines the tokens map type map[string][]int

Source Files

View all Source files

Directories

PathSynopsis
Package core is riot core
Package core is riot core
Package engine is riot engine
Package engine is riot engine
examples
benchmark
riot 性能测试
riot 性能测试
codelab
一个微博搜索的例子。
一个微博搜索的例子。
pinyin_weibo
一个微博 pinyin 搜索的例子。
一个微博 pinyin 搜索的例子。
Package net is riot net
Package net is riot net
Package types is riot types
Package types is riot types

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f orF : Jump to
y orY : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic.Learn more.
[8]ページ先頭
©2009-2025 Movatter.jp