Recently, retrieval-augmented language models have shown to improve over standard neural language models, by accessing information retrieved from a large datastore, in addition to their standard, parametric, next-word prediction.
To this end, we perform a careful analysis of the various dimensions over which knn-lm diverges from standard lms, and investigate these dimensions one by one.
Large pretrained language models (lms) can not only perform remarkably well on a range of natural language processing (nlp) tasks but also start improving on reasoning tasks such as arithmetic induction, symbolic manipulation, and commonsense reasoning with increasing size of models.
However, it is still unclear what the underlying capabilities of these models are.
Relational knowledge bases (kbs) are established tools for world knowledgerepresentation in machines, but they usually sacrifice some data modeling flexibility for these advantages because they adhere to a manually engineered schema.
We propose a novel taxonomy for relational knowledgerepresentation in contextual lms based on the level of kb supervision provided, considering both works that probe lms for implicit relational knowledge acquired during self-supervised pretraining on unstructured text alone, and works that explicitly supervise lms at the level of kb entities and/or relationships.