Tensors and tensor decompositions for combining external information with knowledge graph embeddings

Abstract

The task of knowledge graph (KG) completion, where one is given an incomplete KG as a list of facts, and is asked to give high scores to correct but unseen triples, has been a well-studied problem in the NLP community. A simple but surprisingly robust approach for solving this task emerged as learning low dimensional embeddings for entities and relations by approximating the underlying KG directly through a scoring function. Knowledge graphs have a natural representation as a binary three way array, also known as a 3rd order tensor, and certain classes of scoring functions can be characterized as finding a low-rank decomposition of this tensor. This dissertation extends this characterization, and investigates the suitability of tensors for modelling both knowledge graphs and related data, for learning low-rank representations of entities and relations that incorporate information from heterogeneous sources, and for reasoning with paths and rules using the learned representations. Specifically, we present two joint tensor decomposition models for integrating external information in the process of learning KG embeddings. Our first model is a joint tensor-tensor decomposition model that learns representations based on both KG facts and type information on entities and relations. Our second model is a joint tensor-matrix decomposition for integrating cooccurrence information between entities and words from an entity linked corpus into knowledge graph embeddings, in order to learn better representations for the entities that are rarely seen in the knowledge graph. We also investigate tensors as tools for enabling multi-step reasoning using learned embedding representations. To this end, we extend theoretical results for semiring weighted logic programs to tensors of semirings. Our results are broadly applicable to any area that uses dynamic programming algorithms for calculating tensor values. Such applications include incorporating embeddings of paths and rules for knowledge graph completion, and syntactic parsing with latent variable grammars