Inductive knowledge graph completion

Retriever-Augmented Generation, or RAG, is a type of language generation model that combines pre-trained parametric and non-parametric memory for language generation. Specifically, the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. For query $x$, Maximum Inner Product Search (MIPS) is used to find the top-K documents $z\_{i}$. For final prediction $y$, we treat $z$ as a latent variable and marginalize over seq2seq predictions given different documents.

Hyperboloid Embeddings (HypE) is a novel self-supervised dynamic reasoning framework, that utilizes positive first-order existential queries on a KG to learn representations of its entities and relations as hyperboloids in a Poincaré ball. HypE models the positive first-order queries as geometrical translation (t), intersection ($\cap$), and union ($\cup$). For the problem of KG reasoning in real-world datasets, the proposed HypE model significantly outperforms the state-of-the art results. HypE is also applied to an anomaly detection task on a popular e-commerce website product taxonomy as well as hierarchically organized web articles and demonstrate significant performance improvements compared to existing baseline methods. Finally, HypE embeddings can also be visualized in a Poincaré ball to clearly interpret and comprehend the representation space.

RotatE is a method for generating graph embeddings which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition. Specifically, the RotatE model defines each relation as a rotation from the source entity to the target entity in the complex vector space. The RotatE model is trained using a self-adversarial negative sampling technique.

CodeGen is an autoregressive transformers with next-token prediction language modeling as the learning objective trained on a natural language corpus and programming language data curated from GitHub.

An RGCN, or Relational Graph Convolution Network, is a an application of the GCN framework to modeling relational data, specifically to link prediction and entity classification tasks. See here for an in-depth explanation of RGCNs by DGL.

MEI introduces the multi-partition embedding interaction technique with block term tensor format to systematically address the efficiency--expressiveness trade-off in knowledge graph embedding. It divides the embedding vector into multiple partitions and learns the local interaction patterns from data instead of using fixed special patterns as in ComplEx or SimplE models. This enables MEI to achieve optimal efficiency--expressiveness trade-off, not just being fully expressive. Previous methods such as TuckER, RESCAL, DistMult, ComplEx, and SimplE are suboptimal restricted special cases of MEI.

To avoid the problem caused by low-frequent entity-relation pairs, our MBS uses the estimated probabilities from a trained model $\mathbf{\theta}'$ to calculate frequencies for each triplet and query. By using $\mathbf{\theta}'$, the NS loss in KGE with MBS is represented as follows: \begin{align} &\ell_{mbs}(\mathbf{\theta};\mathbf{\theta}') \nonumber \ =&-\frac{1}{|D|}\sum_{(x,y) \in D} \Bigl[A_{mbs}(\mathbf{\theta}')\log(\sigma(s_{\mathbf{\theta}}(x,y)+\gamma))\nonumber\ &+\frac{1}{\nu}sum_{y_{i}\sim p_n(y_{i}|x)}^{\nu}B_{mbs}(\mathbf{\theta}')\log(\sigma(-s_{\mathbf{\theta}}(x,y_i)-\gamma))\Bigr], \end{align}

Galactica is a language model which uses a Transformer architecture in a decoder-only setup with the following modifications: - It uses GeLU activations on all model sizes - It uses a 2048 length context window for all model sizes - It does not use biases in any of the dense kernels or layer norms - It uses learned positional embeddings for the model - A vocabulary of 50k tokens was constructed using BPE. The vocabulary was generated from a randomly selected 2% subset of the training data

The composed video retrieval (CoVR) task is a new task, where the goal is to find a video that matches both a query image and a query text. The query image represents a visual concept that the user is interested in, and the query text specifies how the concept should be modified or refined. For example, given an image of a fountain and the text during show at night, the CoVR task is to retrieve a video that shows the fountain at night with a show.

Symbolic rule learning methods find regularities in data that can be expressed in the form of 'if-then' rules based on symbolic representations of the data.

Graph Path Feature Learning is a probabilistic rule learner optimized to mine instantiated first-order logic rules from knowledge graphs. Instantiated rules contain constants extracted from KGs. Compared to abstract rules that contain no constants, instantiated rules are capable of explaining and expressing concepts in more detail. GPFL utilizes a novel two-stage rule generation mechanism that first generalizes extracted paths into templates that are acyclic abstract rules until a certain degree of template saturation is achieved, then specializes the generated templates into instantiated rules.

LLM-SR pioneers the use of LLMs for scientific equation discovery and symbolic regression and shows how LLMs, with their vast scientific knowledge and coding capability, enhance equation discovery across various scientific fields.

Recurrent Event Network (RE-NET) is an autoregressive architecture for predicting future interactions. The occurrence of a fact (event) is modeled as a probability distribution conditioned on temporal sequences of past knowledge graphs. RE-NET employs a recurrent event encoder to encode past facts and uses a neighborhood aggregator to model the connection of facts at the same timestamp. Future facts can then be inferred in a sequential manner based on the two modules.

SAFRAN is a rule application framework which aggregates rules through a scalable clustering algorithm.

CP with N3 Regularizer and Relation Prediction

Child-Tuning is a fine-tuning technique that updates a subset of parameters (called child network) of large pretrained models via strategically masking out the gradients of the non-child network during the backward process. It decreases the hypothesis space of the model via a task-specific mask applied to the full gradients, helping to effectively adapt the large-scale pretrained model to various tasks and meanwhile aiming to maintain its original generalization ability.

ComplEx model trained with a nuclear norm regularizer; A relation prediction objective is added on top of the commonly used 1vsAll objective.

RESCAL model trained with a relation prediction objective on top of the 1vsAll loss

TuckER model trained with a relation prediction objective on top of the 1vsAll loss

Vulnerability-constrained Decoding, is a sequence decoding approach that aims to avoid generating vulnerabilities in generated code.