AutoML

DenseNAS is a neural architecture search method that utilises a densely connected search space. The search space is represented as a dense super network, which is built upon designed routing blocks. In the super network, routing blocks are densely connected and we search for the best path between them to derive the final architecture. A chained cost estimation algorithm is used to approximate the model cost during the search.

DenseNAS-A is a mobile convolutional neural network discovered through the DenseNAS neural architecture search method. The basic building block is MBConvs, or inverted bottleneck residuals, from the MobileNet architectures.

DenseNAS-B is a mobile convolutional neural network discovered through the DenseNAS neural architecture search method. The basic building block is MBConvs, or inverted bottleneck residuals, from the MobileNet architectures.

DetNASNet is a convolutional neural network designed to be an object detection backbone and discovered through DetNAS architecture search. It uses ShuffleNet V2 blocks as its basic building block.

A DiCE Unit is an image model block that is built using dimension-wise convolutions and dimension-wise fusion. The dimension-wise convolutions apply light-weight convolutional filtering across each dimension of the input tensor while dimension-wise fusion efficiently combines these dimension-wise representations; allowing the DiCE unit to efficiently encode spatial and channel-wise information contained in the input tensor. Standard convolutions encode spatial and channel-wise information simultaneously, but they are computationally expensive. To improve the efficiency of standard convolutions, separable convolution are introduced, where spatial and channelwise information are encoded separately using depth-wise and point-wise convolutions, respectively. Though this factorization is effective, it puts a significant computational load on point-wise convolutions and makes them a computational bottleneck. DiCE Units utilize a dimension-wise convolution to encode depth-wise, width-wise, and height-wise information independently. The dimension-wise convolutions encode local information from different dimensions of the input tensor, but do not capture global information. One approach is a pointwise convolution, but it is computationally expensive, so instead dimension-wise fusion factorizes the point-wise convolution in two steps: (1) local fusion and (2) global fusion.

DiCENet is a convolutional neural network architecture that utilizes dimensional convolutions (and dimension-wise fusion). The dimension-wise convolutions apply light-weight convolutional filtering across each dimension of the input tensor while dimension-wise fusion efficiently combines these dimension-wise representations; allowing the DiCE Unit in the network to efficiently encode spatial and channel-wise information contained in the input tensor.

Differentiable simultaneous optimization of hyperparameters and neural network architecture. Also a Neural Architecture Search (NAS) method.

GreedyNAS is a one-shot neural architecture search method. Previous methods held the assumption that a supernet should give a reasonable ranking over all paths. They thus treat all paths equally, and spare much effort to train paths. However, it is harsh for a single supernet to evaluate accurately on such a huge-scale search space (eg, $7^{21}$). GreedyNAS eases the burden of supernet by encouraging focus more on evaluation of potentially-good candidates, which are identified using a surrogate portion of validation data. Concretely, during training, GreedyNAS utilizes a multi-path sampling strategy with rejection, and greedily filters the weak paths. The training efficiency is thus boosted since the training space has been greedily shrunk from all paths to those potentially-good ones. An exploration and exploitation policy is adopted by introducing an empirical candidate path pool.

GreedyNAS-B is a convolutional neural network discovered using the GreedyNAS neural architecture search method. The basic building blocks used are inverted residual blocks (from MobileNetV2) and squeeze-and-excitation blocks.

GreedyNAS-C is a convolutional neural network discovered using the GreedyNAS neural architecture search method. The basic building blocks used are inverted residual blocks (from MobileNetV2) and squeeze-and-excitation blocks.

Hit-Detector is a neural architectures search algorithm that simultaneously searches all components of an object detector in an end-to-end manner. It is a hierarchical approach to mine the proper subsearch space from the large volume of operation candidates. It consists of two main procedures. First, given a large search space containing all the operation candidates, we screen out the customized sub search space suitable for each part of detector with the help of group sparsity regularization. Secondly, we search the architectures for each part within the corresponding sub search space by adopting the differentiable manner.

Optimize combinations of various neural network models for multimodal data with bayseian optimization.

LightAutoML is an AutoML solution targeted for financial services companies. A typical LightAutoML pipeline scheme is presented in the Figure, each pipeline containing: - Reader: object that receives raw data and task as input, calculates some useful metadata, performs initial data cleaning and decides about data manipulations that should be done before fitting different model types. - LightAutoML inner datasets that contains metadata and CV iterators that implements validation scheme for the datasets. - Multiple ML Pipelines that are stacked and/or blended to get a single prediction. An ML pipeline in LightAutoML is one or multiple ML models that share a single data preprocessing and validation scheme. The preprocessing step may have up to two feature selection steps, a feature engineering step or even just be empty if no preprocessing is needed. The ML pipelines can be computed independently on the same datasets and then blended together using averaging (or weighted averaging). Alternatively, a stacking ensemble scheme can be used to build multi level ensemble architectures.

MoGA-B is a convolutional neural network optimized for mobile latency and discovered via Mobile GPU-Aware (MoGA) neural architecture search. The basic building block is MBConvs (inverted residual blocks) from MobileNetV2. Squeeze-and-excitation layers are also experimented with.

MoGA-C is a convolutional neural network optimized for mobile latency and discovered via Mobile GPU-Aware (MoGA) neural architecture search. The basic building block is MBConvs (inverted residual blocks) from MobileNetV2. Squeeze-and-excitation layers are also experimented with.

We present the latest generation of MobileNets, known as MobileNetV4 (MNv4), featuring universally efficient architecture designs for mobile devices. At its core, we introduce the Universal Inverted Bottleneck (UIB) search block, a unified and flexible structure that merges Inverted Bottleneck (IB), ConvNext, Feed Forward Network (FFN), and a novel Extra Depthwise (ExtraDW) variant. Alongside UIB, we present Mobile MQA, an attention block tailored for mobile accelerators, delivering a significant 39% speedup. An optimized neural architecture search (NAS) recipe is also introduced which improves MNv4 search effectiveness. The integration of UIB, Mobile MQA and the refined NAS recipe results in a new suite of MNv4 models that are mostly Pareto optimal across mobile CPUs, DSPs, GPUs, as well as specialized accelerators like Apple Neural Engine and Google Pixel EdgeTPU - a characteristic not found in any other models tested. Finally, to further boost accuracy, we introduce a novel distillation technique. Enhanced by this technique, our MNv4-Hybrid-Large model delivers 87% ImageNet-1K accuracy, with a Pixel 8 EdgeTPU runtime of just 3.8ms.

SCARLET-NAS is a type of neural architecture search that utilises a learnable stabilizer to calibrate feature deviation, named the Equivariant Learnable Stabilizer (ELS). Previous one-shot approaches can be limited by fixed-depth search spaces. With SCARLET-NAS, we use the equivariant learnable stabilizer on each skip connection. This can lead to improved convergence, more reliable evaluation, and retained equivalence. The third benefit is deemed most important by the authors for scalability.