# Information Retrieval

## Books

* [Semantic Search on Text and Knowledge Bases](https://www.researchgate.net/publication/304364705_Semantic_Search_on_Text_and_Knowledge_Bases), by Bast, Hannah & Buchhold, Björn & Haussmann, Elmar, 2016.\
  A comprehensive overview of the broad area of semantic search on text and knowledge bases.\
  It was written in 2016, which means the analysis stops before Transformers were widely adopted in IR. \
  This is the book to read in order to understand the IR field prior to the introduction of Transformers.
* [Pretrained Transformers for Text Ranking: BERT and Beyond](https://arxiv.org/pdf/2010.06467.pdf), by Jimmy Lin, Rodrigo Nogueira, and Andrew Yates, 2021.\
  The previous book was about the IR field prior to 2016. This one is about the recent years. In particular, about Transformers usage in text ranking.\
  This is the book to read in order to uderstand the present of IR (at least at the time when I'm writing this).<br>

## Papers

* [Multi-Stage Document Ranking with BERT](https://arxiv.org/abs/1910.14424) by [Rodrigo Nogueira](https://arxiv.org/search/cs?searchtype=author\&query=Nogueira%2C+R), [Wei Yang](https://arxiv.org/search/cs?searchtype=author\&query=Yang%2C+W), [Kyunghyun Cho](https://arxiv.org/search/cs?searchtype=author\&query=Cho%2C+K), [Jimmy Lin](https://arxiv.org/search/cs?searchtype=author\&query=Lin%2C+J), 2019\
  Rodrigo Nogueira is co-author of the second book I mentioned at the start of this page. This is a must-read article, as R.N. has important contributions in this field.&#x20;
* [Passage Re-ranking with BERT](https://arxiv.org/abs/1901.04085), by [Rodrigo Nogueira](https://arxiv.org/search/cs?searchtype=author\&query=Nogueira%2C+R), [Kyunghyun Cho](https://arxiv.org/search/cs?searchtype=author\&query=Cho%2C+K), 2019.
* [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://arxiv.org/abs/2101.08751), by [Luyu Gao](https://arxiv.org/search/cs?searchtype=author\&query=Gao%2C+L), [Zhuyun Dai](https://arxiv.org/search/cs?searchtype=author\&query=Dai%2C+Z), [Jamie Callan](https://arxiv.org/search/cs?searchtype=author\&query=Callan%2C+J).\
  They propose Localized Contrastive Estimation (LCE) for training rerankers and demonstrate it significantly improves deep two-stage models
* [What happens to BERT embeddings during fine-tuning](https://arxiv.org/abs/2004.14448), by [Amil Merchant](https://arxiv.org/search/cs?searchtype=author\&query=Merchant%2C+A), [Elahe Rahimtoroghi](https://arxiv.org/search/cs?searchtype=author\&query=Rahimtoroghi%2C+E), [Ellie Pavlick](https://arxiv.org/search/cs?searchtype=author\&query=Pavlick%2C+E), [Ian Tenney](https://arxiv.org/search/cs?searchtype=author\&query=Tenney%2C+I)\
  They investigate how fine-tuning affects the representations of the BERT model. \
  Findings:\
  \- Fine-tuning does not lead to catastrophic forgetting of linguistic phenomena \
  \- fine-tuning primarily affects the top layers of BERT, but with noteworthy variation across tasks\
  \- weaker effect on representations of out-of-domain sentences
* [Understanding the Behaviors of BERT in Ranking](https://arxiv.org/abs/1904.07531), by [Yifan Qiao](https://arxiv.org/search/cs?searchtype=author\&query=Qiao%2C+Y), [Chenyan Xiong](https://arxiv.org/search/cs?searchtype=author\&query=Xiong%2C+C), [Zhenghao Liu](https://arxiv.org/search/cs?searchtype=author\&query=Liu%2C+Z), [Zhiyuan Liu](https://arxiv.org/search/cs?searchtype=author\&query=Liu%2C+Z)
* [Language models as knowledge bases ?](https://arxiv.org/abs/1909.01066) by [Fabio Petroni](https://arxiv.org/search/cs?searchtype=author\&query=Petroni%2C+F), [Tim Rocktäschel](https://arxiv.org/search/cs?searchtype=author\&query=Rockt%C3%A4schel%2C+T), [Patrick Lewis](https://arxiv.org/search/cs?searchtype=author\&query=Lewis%2C+P), [Anton Bakhtin](https://arxiv.org/search/cs?searchtype=author\&query=Bakhtin%2C+A), [Yuxiang Wu](https://arxiv.org/search/cs?searchtype=author\&query=Wu%2C+Y), [Alexander H. Miller](https://arxiv.org/search/cs?searchtype=author\&query=Miller%2C+A+H), [Sebastian Riedel](https://arxiv.org/search/cs?searchtype=author\&query=Riedel%2C+S) &#x20;
* [Understanding BERT Rankers Under Distillation](https://arxiv.org/abs/2007.11088), by [Luyu Gao](https://arxiv.org/search/cs?searchtype=author\&query=Gao%2C+L), [Zhuyun Dai](https://arxiv.org/search/cs?searchtype=author\&query=Dai%2C+Z), [Jamie Callan](https://arxiv.org/search/cs?searchtype=author\&query=Callan%2C+J)
* [Deep Learning for Matching in Search and Recommendation](http://staff.ustc.edu.cn/~hexn/papers/www18-tutorial-deep-matching-paper.pdf), by Jun Xu, Hang Li, Xiangnan He, 2019
* [Project PIAF: Building a Native French Question-Answering Dataset](https://arxiv.org/abs/2007.00968), by [Rachel Keraron](https://arxiv.org/search/cs?searchtype=author\&query=Keraron%2C+R), [Guillaume Lancrenon](https://arxiv.org/search/cs?searchtype=author\&query=Lancrenon%2C+G), [Mathilde Bras](https://arxiv.org/search/cs?searchtype=author\&query=Bras%2C+M), [Frédéric Allary](https://arxiv.org/search/cs?searchtype=author\&query=Allary%2C+F), [Gilles Moyse](https://arxiv.org/search/cs?searchtype=author\&query=Moyse%2C+G), [Thomas Scialom](https://arxiv.org/search/cs?searchtype=author\&query=Scialom%2C+T), [Edmundo-Pavel Soriano-Morales](https://arxiv.org/search/cs?searchtype=author\&query=Soriano-Morales%2C+E), [Jacopo Staiano](https://arxiv.org/search/cs?searchtype=author\&query=Staiano%2C+J), 2020

## Other

* [Zero-shot, One Kill: BERT for Neural Information Retrieval](https://uu.diva-portal.org/smash/get/diva2:1563081/FULLTEXT01.pdf), by Stergios Efes\
  This is an interesting work done as a master thess

<table data-header-hidden><thead><tr><th width="169.7142857142857"></th><th width="150"></th><th width="150"></th><th></th></tr></thead><tbody><tr><td>Resource Name</td><td>Type</td><td>Tags</td><td>Description</td></tr><tr><td><a href="https://dl.acm.org/doi/10.1145/3397271.3401331">Personalized Query Suggestionsdid not find the full paper</a></td><td>short-paper</td><td>query suggestion, personalization</td><td>a sequence-to-sequence-model--based query suggestion framework that is capable of modeling structured, personalized features and unstructured query texts naturally.</td></tr><tr><td><a href="https://arxiv.org/pdf/1905.01386.pdf">Personalized query Auto-Completion Through a Lightweight Representation of the User Context</a></td><td>article</td><td>query suggestion, personalization</td><td>using fastText library for personalized autosuggest</td></tr><tr><td><a href="https://arxiv.org/pdf/1805.02816.pdf">Attention-based Hierarchical Neural Query Suggestion</a></td><td>article</td><td>query suggestion, personalization</td><td>using RNNs</td></tr><tr><td><a href="https://www.irit.fr/CIRCLE/wp-content/uploads/2020/06/CIRCLE20_06.pdf">Using BERT and BART for Query Suggestion</a></td><td>article</td><td><br></td><td><br></td></tr><tr><td><a href="https://arxiv.org/pdf/2008.06759.pdf">Deep Search Query Intent Understanding</a></td><td>article</td><td>query intent</td><td>what they do is actually use these models to predict only the type of document targeted by the user through this search (that i, to predict whether the user is looking for a user profile, a job posting, feed post etc)</td></tr><tr><td><a href="https://arxiv.org/pdf/2102.07662.pdf">OVERVIEW OF THE TREC 2020 DEEP LEARNING TRACK</a></td><td>article</td><td>passage and document ranking</td><td>This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime. </td></tr><tr><td><a href="https://arxiv.org/abs/1905.05583">How to Fine-Tune BERT for Text Classification?</a></td><td>article</td><td>BERT, fine-tuning, classification</td><td>experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning.</td></tr><tr><td><a href="https://jalammar.github.io/illustrated-bert/">The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)</a></td><td>blog article</td><td>BERT</td><td><br></td></tr><tr><td><a href="http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.673.pdf">Project PIAF: Building a Native French Question-Answering Dataset</a></td><td>paper</td><td>french Q&#x26;A dataset</td><td><br></td></tr><tr><td><a href="https://www.microsoft.com/en-us/research/uploads/prod/2020/09/dnnmem.pdf">Estimating GPU Memory Consumption of Deep Learning Models</a></td><td>paper</td><td>GPU memory usage</td><td>most ML projects that fail while running / training actually run out of memory. They proposed a way to estimate memory needs so your experiment doesn't fail half way through. (only read abstract)</td></tr><tr><td><a href="https://sigir.org/wp-content/uploads/2018/01/p032.pdfa">Some Common Mistakes In IR Evaluation, And How They Can Be Avoided</a></td><td>paper</td><td>mistakes metrics in information retrieval</td><td>most research in information retrieval evaluates results using MRR, MAP</td></tr><tr><td><a href="https://medium.com/mlearning-ai/semantic-search-with-s-bert-is-all-you-need-951bc710e160">Semantic Search with S-BERT</a></td><td>blog post</td><td><br></td><td>SBERT, FAISS on MS MARCO for document retrieval using FAISS to store the embeddings and search; presumably easier than setting up Elasticsearch</td></tr><tr><td><a href="https://kmwllc.com/index.php/2020/03/20/understanding-tf-idf-and-bm-25/">BM25 vs TF-IDF</a></td><td>blog post</td><td><br></td><td>it's a nice overview of what TF-IDF does and in which cases BM25 is better</td></tr><tr><td><a href="https://dl.acm.org/doi/10.1145/3331184.3331373">Enhanced News Retrieval: Passages Lead the Way!</a></td><td>paper</td><td>passage retrieval</td><td>Adapting BM25 into BM25P to identify relevant candidate documents for the passage retrieval task</td></tr></tbody></table>
