Abstraction not Memory: BERT and the English Article System
Harish Tayyar Madabushi, Dagmar Divjak, Petar Milin
Article prediction is a task that has long defied accurate linguistic
description. As such, this task is ideally suited to evaluate models on their
ability to emulate native-speaker intuition. To this end, we compare the
performance of native English speakers and pre-trained models on the task of
article prediction set up as a three way choice (a/an, the, zero). Our
experiments with BERT show that BERT outperforms humans on this task across all
articles. In particular, BERT is far superior to humans at detecting the zero
article, possibly because we insert them using rules that the deep neural model
can easily pick up. More interestingly, we find that BERT tends to agree more
with annotators than with the corpus when inter-annotator agreement is high but
switches to agreeing more with the corpus as inter-annotator agreement drops.
We contend that this alignment with annotators, despite being trained on the
corpus, suggests that BERT is not memorising article use, but captures a high
level generalisation of article use akin to human intuition.