Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up

Open Information Retrieval Benchmark Framework

NotificationsYou must be signed in to change notification settings

AKSW/irbench

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 

Repository files navigation

An Open Information Retrieval Framework Benchmark for testing, benchmarking and developing IR systems.

Benchmarking

Orbit bridges different benchmark datasets such as TREC and QALD.It contain the main features necessary to benchmark IR systems.It implements parsers for the two main benchmark dataset available TREC and QALD.It contains a list of five meaures that can be used to evaluate the system answers: MAP@?, P@?, Precision, Recall, and F-measure.It also have 9 emmbeded QALD test datasets.Below you find a list of benchmark functionalities implemented.

java -jar orbit.benchmark.jar [option]:-evaluate<benchmarkAnswerFile|datasetID><systemAnswerFile><"measure1","measure2">[-latex|-json]-datasetsList available datasets for evaluation.-measuresList available measures.-questions ["datasetFile1","datasetFile2","datasetID1"...] [-format <format>] [-lang <lang>]

Listing available datasets

Orbit benchmark framework already comes with 9 embedded QALD datasets.

java -jar orbit.benchmark.jar -datasetsqald-5-test-multilingualqald-7-test-multilingualqald-3-test-multilingualqald-6-test-multilingualqald-9-test-multilingualqald-4-test-multilingualqald-2-test-multilingualqald-1-test-multilingualqald-8-test-multilingualqald-9-test-multilingual

Listing available measures

There are totally 7 implemented measures while 2 are extensions (e.g. P@10 and P@100 are PrecisionAt instantiations).

java -jar orbit.benchmark.jar -measuresf-scoreP@100precisionrecallP@10MAP@10MAP@100

Evaluating

java -jar -evaluate "qald-8-test-multilingual" "qald-8-test-multilingual.qald.json" "MAP@10"|Q-ID|MAP@10||22|1.0|...|21|1.0||43|1.0||AVG|1.0|

You can also have the evaluation output in latex, using the pragma -latex.

java -jar -evaluate "qald-8-test-multilingual" "qald-8-test-multilingual.qald.json" "MAP@10" -latex\begin{center}\begin{tabular}{| c | c |}\hline\hlineQ&MAP@10\\\hline\hline22&1.0 \\44&1.0 \\45&1.0 \\...

Outputing dataset(s) content(s)

java -jar orbit.benchmark.jar -questions...{"datasetId":"qald-8-test-multilingual","question":"What is Donald Trump's main business?", "keywords":"Donald Trump's,  business", "sparql":"PREFIX dbo: <http:\/\/dbpedia.org\/ontology\/> PREFIX dbr: <http:\/\/dbpedia.org\/resource\/> SELECT ?owner WHERE { ?owner dbo:owner dbr:Donald_Trump }" , "lang":"en"}{"datasetId":"qald-8-test-multilingual","question":"What is the last work of Dan Brown?", "keywords":"last work,  Dan Brown", "sparql":"PREFIX dbo: <http:\/\/dbpedia.org\/ontology\/> PREFIX dbp: <http:\/\/dbpedia.org\/property\/> PREFIX dbr: <http:\/\/dbpedia.org\/resource\/> SELECT ?book ?date WHERE { ?book dbo:author dbr:Dan_Brown . ?book dbp:releaseDate|dbo:publicationDate ?date } ORDER BY DESC(?date)" , "lang":"en"}{"datasetId":"qald-8-test-multilingual","question":"What other books have been written by the author of The Fault in Our Stars?", "keywords":"books,  author,  The Fault in Our Stars", "sparql":"PREFIX dbo: <http:\/\/dbpedia.org\/ontology\/> SELECT ?books WHERE { ?books dbo:author <http:\/\/dbpedia.org\/resource\/John_Green_(author)> }" , "lang":"en"}{"datasetId":"qald-8-test-multilingual","question":"When was the last episode of the TV series Friends aired?", "keywords":"last episode,  Friends Tv show", "sparql":"PREFIX dbo: <http:\/\/dbpedia.org\/ontology\/> PREFIX dbr: <http:\/\/dbpedia.org\/resource\/> SELECT ?date WHERE { dbr:Friends dbo:completionDate ?date }" , "lang":"en"}
[8]ページ先頭
©2009-2025 Movatter.jp