文献类型： 会议论文

第一作者： Mark Bo Chu

作者： Mark Bo Chu ¹ ; Bhargav Srinivasa Desikan ² ; Ethan O. Nadler ³ ; Ruggerio L. Sardo ⁴ ; Elise Darragh-Ford ⁵ ; Douglas Guilbeault ⁶ ;

作者机构： 1.Columbia University

2.Ecole Polytechnique Federale de Lausanne

3.Carnegie Observatories University of Southern California

4.Sapienza University of Rome

5.Stanford University KIPAC & Department of Physics

6.University of California, Berkeley Haas Business School

会议名称： Annual meeting of the Association for Computational Linguistics

主办单位：

页码： 7120-7134

摘要： Natural language processing models learn word representations based on the distributional hypothesis, which asserts that word context (e.g., co-occurrence) correlates with meaning. We propose that n-grams composed of random character sequences, or garble, provide a novel context for studying word meaning both within and beyond extant language. In particular, randomly generated character n-grams lack meaning but contain primitive information based on the distribution of characters they contain. By studying the embeddings of a large corpus of garble, extant language, and pseudowords using CharacterBERT, we identify an axis in the model's high-dimensional embedding space that separates these classes of n-grams. Furthermore, we show that this axis relates to structure within extant language, including word part-of-speech, morphology, and concept concreteness. Thus, in contrast to studies that are mainly limited to extant language, our work reveals that meaning and primitive information are intrinsically linked.

分类号： tp391