The following material is available for download from the CIIR. It is provided without warranty and without support. If there are problems accessing or using any of this material, we would appreciate being told (info at ciir.cs.umass.edu), in case we can address the issue.
Passage Similarity and Diversification in Non-factoid Question Answering - Dataset - ICTIR - July 2021
Lakshmi Vikraman, Ali Montazeralghaem, Helia Hashemi, W. Bruce Croft, James Allan
This consists of two datasets:
(a) NFPassageQA_Sim: Answer Passage Similarity task with 128 questions and 18,314 judgements
(b) NFPassageQA_Div: Answer Passage Diversification task with 93 questions and 1602 judgements
For more details, please refer to the paper given below.
Link to the paper: https://ciir-publications.cs.umass.edu/getpdf.php?id=1416
Link to the dataset: https://ciir.cs.umass.edu/downloads/NFPassageQA/
Lemur Project Downloads - June 2020
The Lemur Project (an NSF-funded collaboration with CMU and the CIIR) develops search engines, browser toolbars, text analysis tools, and data resources that support research and development of information retrieval and text mining software. The project is best known for its Indri search engine, Lemur Toolbar, and ClueWeb09 dataset. Lemur software and datasets are available at: http://www.lemurproject.org.
Open-Retrieval Conversational Question Answering - May 2020
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, Mohit Iyyer
The OR-QuAC dataset enhances QuAC by adapting it to an open-retrieval setting. It is an aggregation of three existing datasets:
(1) the QuAC dataset that offers information-seeking conversations,
(2) the CANARD dataset that consists of context-independent rewrites of QuAC questions, and
(3) the Wikipedia corpus that serves as the knowledge source of answering questions.
Link to the dataset: https://ciir.cs.umass.edu/downloads/ORConvQA/
Incorporating Hierarchical Domain Information to Disambiguate Very Short Queries - Dataset - ICTIR - October 2019
Hamed Bonab, Mohammad Aliannejadi, John Foley, James Allan
This data is prepared based on 50 of NIST relevance assessors queries from TREC 2017 Core Track (https://trec-core.github.io/2017/) data collection. It has over 1.8 million New York Times (NYT) annotated articles, containing nearly every published article between January 1, 1987, and June 19, 2007. We provide Rel and RetRel query with domain pairs for each query. We also provide the NYT corpus hierarchy in xml file format. For more information please refer to the published paper.
Link to the paper: https://ciir-publications.cs.umass.edu/pub/web/getpdf.php?id=1313
Link to the dataset: https://ciir.cs.umass.edu/downloads/ictir19_nyt_hierarchy/
Simulating CLIR Translation Resource Scarcity using High-resource Languages - Dataset - ICTIR - October 2019
Hamed Bonab, James Allan, and Ramesh Sitaraman
This data is prepared based on 200 queries of the Cross-Language Evaluation Forum (CLEF) 2000-2003 campaign for bilingual ad-hoc retrieval tracks (http://catalog.elra.info/en-us/repository/browse/ELRA-E0008/). The Swahili and Somali queries are the translation of English queries from C001-C200 topic set. We hired a translation organization to translate the title and description of each topic into Somali and Swahili. For more information please refer to the published paper.
Link to the paper: https://ciir-publications.cs.umass.edu/pub/web/getpdf.php?id=1357
Link to the dataset: https://ciir.cs.umass.edu/downloads/ictir19_simulate_low_resource/
ANTIQUE: A Non-Factoid Question Answering Benchmark - May 2019
This dataset contains 2626 non-factoid questions and 34k manual judgments ranged from 1 to 4. All questions and answers are collected from a CQA website. Note that the test relevance labels were obtained through depth-k pooling (k=10).
Link to the dataset: https://ciir.cs.umass.edu/downloads/Antique/
Link to the paper: https://arxiv.org/abs/1905.08957
PsgRobust - Answer Passage Retrieval Dataset - December 2018
Keping Bi, Qingyao Ai, W. Bruce Croft
PsgRobust is an answer passage collection built based on the Robust04 collection without manual annotation. It was built for research on iterative relevance feedback based on two assumptions:
1. For the passages that are ranked in the top positions by a powerful ranker, if they are in the relevant documents, we assume that they are relevant.
2. All the passages in the non-relevant document are irrelevant.
There are 383036 passages from 22403 unique documents in total, and 6589 relevant passages in total for the 246 queries, which are from 3544 documents. For detailed information, please check the readme.txt (linked to https://ciir.cs.umass.edu/downloads/PsgRobust/readme.txt) file and our paper.
Keping Bi, Qingyao Ai, W. Bruce Croft. "Iterative Relevance Feedback for Answer Passage Retrieval with Passage-level Semantic Match." in the Proceedings of the European Conference on Information Retrieval (ECIR 19), Cologne, Germany, April 14-18, 2019, pp. 558-572.
Link to the dataset: https://ciir.cs.umass.edu/downloads/PsgRobust/
Link to the paper: https://ciir-publications.cs.umass.edu/pub/web/getpdf.php?id=1295
Citation Worthiness Dataset - SIGIR - July 2018
Hamed Bonab, Hamed Zamani, Erik Learned-Miller, James Allan
This data is prepared based on SEPID corpus (http://pars.ie/lr/sepid-corpus). All the sentence IDs are consistent with the SEPID corpus, in case you need to do other interesting research with this data. It includes sentence-level segmentation of 10,921 articles from ACL ARC 1.0, up to February 2007.
Citation worthiness dataset contains 85,778 sentences with the "cite" label and 1,142,275 sentences with the "not_cite" label. 10% of data (test data) randomly selected and divided based on the section name into 7 chunks. See the paper for more explanation.
Link to the paper: https://ciir-publications.cs.umass.edu/pub/web/getpdf.php?id=1314
Link to the dataset: https://ciir.cs.umass.edu/downloads/sigir18_citation/
Hierarchical Embedding Model for Personal Product Selection - April 2017 Software
Qingyao Ai, Bruce Croft
This software is an implementation of the Hierarchical Embedding Model deep neural network trained to learn personal product selections through user product queries and selections. The probability of a product item being purchased by a user with a query is computed using their corresponding latent representations.
The software is Python and Java based, using Python for neural net processing via Tensorflow and Java Galago indexing and search software from the Lemur Project.
See the README.txt file for this release or the following paper for more details.
Hierarchical Embedding Model for Personal Product Selections
DeepMerge: Merging Multiple Search Result Lists - December 2015
C.J. Lee, Bruce Croft
DeepMerge Merging of Multiple Search Result Lists
nfL6: Yahoo Non-Factoid Question Dataset - November 2015
Daniel Cohen, Bruce Croft
This dataset contains 87,361 questions derived from the Yahoo Webscope L6 collection and their corresponding best, and additional, answers submitted by users. Only the best answer was reviewed in determining answer quality.
The dataset file is in JSON format and may be downloaded in rar or gzip compressed format.
nfL6: Yahoo Non-Factoid Question Dataset
"Search Engines" Galago Source Code
Trevor Strohman
An old and now obsolete version (1.04) of the Galago Java source code that was referenced as a learning resource in the textbook "Search Engines: Information Retrieval in Practice" by Croft, Metzler and Strohman (2009).
It has been moved from its former Google Code repository to here.
"Search Engines" Galago Source Code
Web Annotated Passages - September 2015 Dataset
Liu Yang, Bruce Croft
This dataset contains 7,499 documents with four grades of sentence level relevance judgment annotations for 82 queries derived from the TREC Gov2 web collection. The dataset is archived in rar or gzipʼed tar file formats. The dataset is described and used in Keikha, Park and Croft, SIGIR 2014.
Web Annotated Passages Dataset
Wikipedia Bullet Points June 2013 Dataset
John Foley, James Allan
This dataset contains over 40,000 bullet-point "facts" mined from English Wikipedia year pages in the June 2013 english XML dump. (3.3M, gzipped JSON)
Wikipedia Bullet Points Dataset
Twitter June - July 2014 Dataset
Nada Naji, James Allan
This dataset consists of 71,564,914 Twitter IDs that have been automatically crawled over the period from mid June through early July 2014.
Query Facet and Facet Feedback Annotations
Weize Kong, James Allan
This dataset consists of query facet and facet feedback annotations used in work, "Weize Kong and James Allan. Extending Faceted Search to the General Web. CIKM 2014".
KB Bridge Entity Linking System
Jeffrey Dalton, Pat Verga, Laura Dietz
KB Bridge is an entity linking system which identifies named entities in free text and links them to entries in a semistructured knowledge base, such as Freebase or Wikipedia. See also Dalton, J. and Dietz, L., "A Neighborhood Relevance Model for Entity Linking," OAIR 2013.
KB Bridge Code and Instructions
Online Appendix for Entity Query Feature Expansion with Knowledge Base Links
Jeffrey Dalton, Laura Dietz, James Allan
This online appendix provides additional material for entity query feature expansion, such as additional gold standard annotations, produces rankings, entity-based features, and software. See also Dalton, J., Dietz, L. and Allan, J., "Entity Query Feature Expansion using Knowledge Base Links," SIGIR 2014.
Entity Query Feature Expansion Appendix
Controversy Annotation Dataset
Shiri Dori-Hacohen, James Allan
This collection consists of controversy annotations for 445 webpages and 2060 Wikipedia articles. See also Dori-Hacohen, S. and Allan, J., "Detecting Controversy on the Web," CIKM 2013; Dori-Hacohen, S. and Allan, J., "Automated Controversy Detection on the Web," ECIR 2015.
Controversy Annotation Dataset
Open Library
Henry Feild
This collection consists of 46,561,553 metadata records crawled from the Open Library on November 30, 2011 and click distributions over records for 22,622 queries recorded over the year October 2010 through September 2011.
RETAS OCR Evaluation Dataset
Zeki Yalniz, R. Manmatha
This dataset was created to evaluate the optical character recognition (OCR) accuracy of scanned books. It is provided here for research purposes. The dataset is extracted from books in Project Gutenberg and the Internet Archive.
Book Translation Detection Dataset
Zeki Yalniz, R. Manmatha, Kriste Krstovski, David A. Smith
These datasets were created to evaluate the effectiveness of the translation detection frameworks. The 2K dataset was created by Krstovski and Smith (2011), and the list of translation pairs was updated for use in Yalniz and Manmatha (2012). The 30-book dataset was created for use in Yalniz and Manmatha (2012). This dataset is for research purposes only.
Book Duplicate Detection Dataset
Zeki Yalniz, E. F. Can, R. Manmatha
This dataset was created to evaluate the effectiveness of the partial duplicate detection framework for scanned books proposed by Yalniz, Can and Manmatha (2011). This dataset is for research purposes only.
Searcher Frustration User Study Data
Henry Feild
This is a dataset collected during a user study of frustration during web search at the University of Massachusetts Amherst in October 2009. The study consists of query logs and sensor readings for thirty participants.
This is available under an Open Database/Database Content license. Feel free to use, redistribute, and modify the dataset, but make sure to make it available under the same license and to give due attribution in any public use of the dataset.
Word Image Data Sets
Toni Rath
Data sets containing word images from the George Washington collection with meta-data for retrieval performance evaluation.
Stemming Class from Stemming and Cooccurrence on a Larger Corpus
Jeremy Pickens
Three sets of experiments were done, using initial classes created by (1) the Porter stemmer, (2) K-Stem, and (3) the Porter stemmer classes merged in a connected component manner with the K-Stem classes.
Event Threading Experiment
Nallapati, R., Feng, A., Peng, F., and Allan, J.
This is the experimental data from "Event Threading within News Topics" in the Proceedings of CIKM 2004 conference, pp. 446-453.
Novelty Track, TREC
Alvaro Bolivar
This is a collection building toolkit to assemble the training set used by the CIIR in its participation in the Novelty track at TREC 2002. For details check conference proceedings.