Transfer Learning

A HaloNet is a self-attention based model for efficient image classification. It relies on a local self-attention architecture that efficiently maps to existing hardware with haloing. The formulation breaks translational equivariance, but the authors observe that it improves throughput and accuracies over the centered local self-attention used in regular self-attention. The approach also utilises a strided self-attentive downsampling operation for multi-scale feature extraction.

Shape Adaptor is a novel resizing module for neural networks. It is a drop-in enhancement built on top of traditional resizing layers, such as pooling, bilinear sampling, and strided convolution. This module allows for a learnable shaping factor which differs from the traditional resizing layers that are fixed and deterministic. Image Source: Liu et al.

YOLOP is a panoptic driving perception network for handling traffic object detection, drivable area segmentation and lane detection simultaneously. It is composed of one encoder for feature extraction and three decoders to handle the specific tasks. It can be thought of a lightweight version of Tesla's HydraNet model for self-driving cars. A lightweight CNN, from Scaled-yolov4, is used as the encoder to extract features from the image. Then these feature maps are fed to three decoders to complete their respective tasks. The detection decoder is based on the current best-performing single-stage detection network, YOLOv4, for two main reasons: (1) The single-stage detection network is faster than the two-stage detection network. (2) The grid-based prediction mechanism of the single-stage detector is more related to the other two semantic segmentation tasks, while instance segmentation is usually combined with the region based detector as in Mask R-CNN. The feature map output by the encoder incorporates semantic features of different levels and scales, and our segmentation branch can use these feature maps to complete pixel-wise semantic prediction.

A method that randomly mask out (replaces with some numerical value, usually zero) all features coming from a specific band in multi-spectral data.

Contrastive Predictive Coding v2 (CPC v2) is a self-supervised learning approach that builds upon the original CPC with several improvements. These improvements include: - Model capacity - The third residual stack of ResNet-101 (originally containing 23 blocks, 1024-dimensional feature maps, and 256-dimensional bottleneck layers), is converted to use 46 blocks, with 4096-dimensional feature maps and 512-dimensional bottleneck layers: ResNet-161. - Layer Normalization - The authors find CPC with batch normalization harms downstream performance. They hypothesize this is due to batch normalization allowing large models to find a trivial solution to CPC: it introduces a dependency between patches (through the batch statistics) that can be exploited to bypass the constraints on the receptive field. They replace batch normalization with layer normalization. - Predicting lengths and directions - patches are predicted with contexts from both directions rather than just spatially underneath. - Patch-based Augmentation - Utilising "color dropping" which randomly drops two of the three color channels in each patch, as well as random horizontal flips. Consistent with prior results, this new architecture delivers better performance regardless of

Approximating Spatiotemporal Representations Using a 2DCNN

MixLoRA is a type of PEFT method which construct a resource-efficient sparse MoE model based on LoRA.

Models Genesis, or Generic Autodidactic Models, is a self-supervised approach for learning 3D image representations. The objective of Models Genesis is to learn a common image representation that is transferable and generalizable across diseases, organs, and modalities. It consists of an encoder-decoder architecture with skip connections in between, and is trained to learn a common image representation by restoring the original sub-volume $x\_{i}$ (as ground truth) from the transformed one $\bar{x}\_{i}$ (as input), in which the reconstruction loss (MSE) is computed between the model prediction $x'\_{0}$ and ground truth $x\_{i}$. Once trained, the encoder alone can be fine-tuned for target classification tasks; while the encoder and decoder together can be fine-tuned for target segmentation tasks.

NAIL OR is a two-dimensional activation function which takes two logits as input and produces a single output. It is a piece-wise linear approximation of the probabilistic OR operator in logit-space. $$\begin{equation} \text{OR_{AIL}}(x, y) := \begin{cases} x + y, & x > 0, \, y > 0 \\ \max(x, y), & \text{otherwise} \end{cases} \end{equation} $$

PRNet+ is a multi-task neural network for outdoor position recovery from measurement record (MR) data. PRNet+ develops a feature extraction module to learn common local-, short- and long-term spatio-temporal locality from heterogeneous MR samples, with a convolutional neural network (CNN), long short-term memory cells (LSTM), and attention mechanisms. Specifically, PRNet+ 1) allows the various-length sequences of MR samples, such that the two components (CNN and LSTM) are able to capture spatial locality from the samples within each MR sequence, 2) exploits two attention mechanisms for the time-interval between neighbouring MR samples, together with the one between neighbouring MR sequences, to capture temporal locality, and 3) incorporates the detected transportation modes and predicted locations of heterogeneous MR data into a joint loss for better result.

Protagonist Antagonist Induced Regret Environment Design, or PAIRED, is an adversarial method for approximate minimax regret to generate environments for reinforcement learning. It introduces an antagonist which is allied with the environment generating adversary. The primary agent we are trying to train is the protagonist. The environment adversary’s goal is to design environments in which the antagonist achieves high reward and the protagonist receives low reward. If the adversary generates unsolvable environments, the antagonist and protagonist would perform the same and the adversary would get a score of zero, but if the adversary finds environments the antagonist solves and the protagonist does not solve, the adversary achieves a positive score. Thus, the environment adversary is incentivized to create challenging but feasible environments, in which the antagonist can outperform the protagonist. Moreover, as the protagonist learns to solves the simple environments, the antagonist must generate more complex environments to make the protagonist fail, increasing the complexity of the generated tasks and leading to automatic curriculum generation.

Model to adapt: We use Ultra Fast Structure-aware Deep Lane Detection (UFLD) as baseline and strictly adopt its training scheme and hyperparameters. UFLD treats lane detection as a row-based classification problem and utilizes the row anchors defined by TuSimple. Unsupervised Domain Adaptation with SGPCS: SGPCS builds upon PCS and performs in-domain contrastive learning and crossdomain self-supervised learning via cluster prototypes. We reformulate the pseudo label selection mechanism from SGADA. For each lane, we select the highest confidence value from the griding cells of each row anchor. Based on their griding cell position, the confidence values are divided into two cases: absent lane points and present lane points. Thereby, the last griding cell represents absent lane points as in. For each case, we calculate the mean confidence over the corresponding lanes. We then use the thresholds defined by SGADA to decide whether the prediction is treated as a pseudo label. Our overall objective function comprises the in-domain and cross-domain loss from PCS, the losses defined by UFLD, and our adopted pseudo loss from SGADA. We adjust the momentum for memory bank feature updates to 0.5 and use spherical K-means with K = 2,500 to cluster them into prototypes.

SqueezeBERT is an efficient architectural variant of BERT for natural language processing that uses grouped convolutions. It is much like BERT-base, but with positional feedforward connection layers implemented as convolutions, and grouped convolution for many of the layers.

Talking-Heads Attention is a variation on multi-head attention which includes linear projections across the attention-heads dimension, immediately before and after the softmax operation. In multi-head attention, the different attention heads perform separate computations, which are then summed at the end. Talking-Heads Attention breaks that separation. Two additional learned linear projections are inserted, $P\_{l}$ and $P\_{w}$, which transform the attention-logits and the attention weights respectively, moving information across attention heads. Instead of one "heads" dimension $h$ across the whole computation, we now have three separate heads dimensions: $h\_{k}$, $h$, and $h\_{v}$, which can optionally differ in size (number of "heads"). $h\_{k}$ refers to the number of attention heads for the keys and the queries. $h$ refers to the number of attention heads for the logits and the weights, and $h\_{v}$ refers to the number of attention heads for the values.

TransferQA is a transferable generative QA model, built upon T5 that combines extractive QA and multi-choice QA via a text-to-text transformer framework, and tracks both categorical slots and non-categorical slots in DST. In addition, it introduces two effective ways to construct unanswerable questions, namely, negative question sampling and context truncation, which enable the model to handle “none” value slots in the zero-shot DST setting.

using transfer learning to transfer knowledge from one building to predict the energy consumption of another building with scarce data

BARThez is a self-supervised transfer learning model for the French language based on BART. Compared to existing BERT-based French language models such as CamemBERT and FlauBERT, BARThez is well-suited for generative tasks, since not only its encoder but also its decoder is pretrained.