Power system operators are increasingly looking toward distributed optimization to address various challenges facing electric power systems. To assess their capabilities in environments with nonideal communications, this paper investigates the impacts of data quality on the performance of distributed optimization algorithms. Specifically, this paper compares the performance of the Alternating Direction Method of Multipliers (ADMM), Analytical Target Cascading (ATC), and Auxiliary Problem Principle (APP) algorithms in the context of DC Optimal Power Flow (DC OPF) problems. Using several test systems, this paper characterizes the performance of these algorithms in terms of their convergence rates and solution quality under three data quality nonidealities: (1) additive Gaussian noise, (2) bad data (large error), and (3) intermittent communication failure.
Lensless on-chip microscopy has shown great potential for biomedical imaging due to its large-area and high-throughput imaging capabilities. By combining the pixel super-resolution (PSR) technique, it can improve the resolution beyond the limit of the imaging detector. However, existing PSR techniques are restricted to the feature size and crosstalk of modulation components (such as spatial light modulator), which cannot efficiently encode target information. Besides, the reconstruction algorithms suffer from the trade-off between image quality, reconstruction resolution and computational efficiency. In this work, we constructed a novel integrated lensless on-chip sensor via scattering multiplexing, and reported a robust PSR algorithm for sample reconstruction. The sensor employed a scattering layer as a modulator, which was permanently integrated with the detector. Benefiting from the high-degree-of-freedom reconstruction of the scattering layer, we realized fine wavefront modulation with a small feature size. The integration engineering avoided repetitious calibration and reduce the measurement complexity. The reported PSR algorithm combines both model-driven and data-driven strategies to efficiently exploit the high-frequency information from the fine modulation. A series of experiments validated that the reported sensor provides a low-cost solution for large-scale microscopic imaging, with significant advantages in resolution, image contrast and noise robustness.
In networks of autonomous agents (e.g., fleets of vehicles, scattered sensors), the problem of minimizing the sum of the agents' local functions has received a lot of interest. We tackle here this distributed optimization problem in the case of open networks when agents can join and leave the network at any time. Leveraging recent online optimization techniques, we propose and analyze the convergence of a decentralized asynchronous optimization method for open networks.
To address the communication bottleneck problem in distributed optimization within a master-worker framework, we propose LocalNewton, a distributed second-order algorithm with local averaging. In LocalNewton, the worker machines update their model in every iteration by finding a suitable second-order descent direction using only the data and model stored in their own local memory. We let the workers run multiple such iterations locally and communicate the models to the master node only once every few (say L) iterations. LocalNewton is highly practical since it requires only one hyperparameter, the number L of local iterations. We use novel matrix concentration-based techniques to obtain theoretical guarantees for LocalNewton, and we validate them with detailed empirical evaluation. To enhance practicability, we devise an adaptive scheme to choose L, and we show that this reduces the number of local iterations in worker machines between two model synchronizations as the training proceeds, successively refining the model quality at the master. Via extensive experiments using several real-world datasets with AWS Lambda workers and an AWS EC2 master, we show that LocalNewton requires fewer than 60% of the communication rounds (between master and workers) and less than 40% of the end-to-end running time, compared to state-of-the-art algorithms, to reach the same training~loss.
Information compression is essential to reduce communication cost in distributed optimization over peer-to-peer networks. This paper proposes a communication-efficient linearly convergent distributed (COLD) algorithm to solve strongly convex optimization problems. By compressing innovation vectors, which are the differences between decision vectors and their estimates, COLD is able to achieve linear convergence for a class of $\delta$-contracted compressors. We explicitly quantify how the compression affects the convergence rate and show that COLD matches the same rate of its uncompressed version. To accommodate a wider class of compressors that includes the binary quantizer, we further design a novel dynamical scaling mechanism and obtain the linearly convergent Dyna-COLD. Importantly, our results strictly improve existing results for the quantized consensus problem. Numerical experiments demonstrate the advantages of both algorithms under different compressors.
XGBoost is one of the most widely used machine learning models in the industry due to its superior learning accuracy and efficiency. Targeting at data isolation issues in the big data problems, it is crucial to deploy a secure and efficient federated XGBoost (FedXGB) model. Existing FedXGB models either have data leakage issues or are only applicable to the two-party setting with heavy communication and computation overheads. In this paper, a lossless multi-party federated XGB learning framework is proposed with a security guarantee, which reshapes the XGBoost's split criterion calculation process under a secret sharing setting and solves the leaf weight calculation problem by leveraging distributed optimization. Remarkably, a thorough analysis of model security is provided as well, and multiple numerical results showcase the superiority of the proposed FedXGB compared with the state-of-the-art models on benchmark datasets.
We consider the distributed optimization problem where $n$ agents each possessing a local cost function, collaboratively minimize the average of the $n$ cost functions over a connected network. Assuming stochastic gradient information is available, we study a distributed stochastic gradient algorithm, called exact diffusion with adaptive stepsizes (EDAS) adapted from the Exact Diffusion method and NIDS and perform a non-asymptotic convergence analysis. We not only show that EDAS asymptotically achieves the same network independent convergence rate as centralized stochastic gradient descent (SGD) for minimizing strongly convex and smooth objective functions, but also characterize the transient time needed for the algorithm to approach the asymptotic convergence rate, which behaves as $K_T=\mathcal{O}\left(\frac{n}{1-\lambda_2}\right)$, where $1-\lambda_2$ stands for the spectral gap of the mixing matrix. To the best of our knowledge, EDAS achieves the shortest transient time when the average of the $n$ cost functions is strongly convex and each cost function is smooth. Numerical simulations further corroborate and strengthen the obtained theoretical results.
In this paper, a novel bandwidth negotiation mechanism is proposed for massive devices wireless spectrum sharing, in which individual device locally negotiates bandwidth usage with neighbor devices and globally optimal spectrum utilization is achieved through distributed decision-making. Since only sparse feedback is needed, the proposed mechanism can greatly reduce the signaling overhead. In order to solve the distributed optimization problem when massive devices coexist, mean field multi-agent reinforcement learning (MF-MARL) based bandwidth decision algorithm is proposed, which allow device make globally optimal decision leveraging only neighborhood observation. In simulation, distributed bandwidth negotiation between 1000 devices is demonstrated and the spectrum utilization rate is above 95%. The proposed method is beneficial to reduce spectrum conflicts, increase spectrum utilization for massive devices spectrum sharing.
Heating, ventilating, and air-conditioning (HVAC) systems consume a large amount of energy in residential houses and buildings. Effective energy management of HVAC is a cost-effective way to improve energy efficiency and reduce the energy cost of residential users. This work develops a novel distributed method for the residential transactive energy system that enables multiple users to interactively optimize their energy management of HVAC systems and behind-the-meter batteries. Specifically, this method effectively reduces the cost of smart homes by employing energy trading among users to leverage their power usage flexibility without compromising the users' privacy. To achieve this goal, we design a distributed optimization algorithm based on the alternating direction method of multipliers (ADMM) to automatically operate the HVAC system and batteries, which minimizes the energy costs of users. Specifically, we decouple the optimization problem into a primal subproblem and a dual subproblem. The primal subproblem is solved by the users, and the dual subproblem is solved by the grid operator. Unlike the existing centralized method, our approach only uses the users' private information locally for solving the primal subproblem hence preserves the users' privacy. Using real-world data, we validate our proposed algorithm through extensive simulations in Matlab. The results demonstrate that our method effectively incentivizes the energy trading among the users to reduce users' peak load and reduce the overall energy cost of the system by 23% on average.
We address the problem of resource allocation (RA) in a cognitive radio (CR) communication system with multiple secondary operators sharing spectrum with an incumbent primary operator. The key challenge of the RA problem is the inter-operator coordination arising in the optimization problem so that the aggregated interference at the primary users (PUs) does not exceed the target threshold. While this problem is easily solvable if a centralized unit could access information of all secondary operators, it becomes challenging in a realistic scenario. In this paper, considering a satellite setting, we alleviate this problem by proposing two approaches to reduce the information exchange level among the secondary operators. In the first approach, we formulate an RA scheme based on a partial information sharing method which enables distributed optimization across secondary operators. In the second approach, instead of exchanging secondary users (SUs) information, the operators only exchange their contributions of the interference-level and RA is performed locally across secondary operators. These two approaches, for the first time in this context, provide a trade-off between performance and level of inter-operator information exchange. Through the numerical simulations, we explain this trade-off and illustrate the penalty resulting from partial information exchange.
A widely-distributed radar system is a promising architecture to enhance radar imaging performance. However, most existing algorithms rely on isotropic scattering assumption, which is only satisfied in collocated radar systems. Moreover, due to noise and imaging model imperfections, artifacts such as layovers are common in radar images. In this paper, a novel $l_1$-regularized, consensus alternating direction method of multipliers (CADMM) based algorithm is proposed to mitigate artifacts by exploiting a widely-distributed radar system's spatial diversity. By imposing the consensus constraints on the local images formed by distributed antenna clusters and solving the resulting distributed optimization problem, the scenario's spatial-invariant common features are retained. Simultaneously, the spatial-variant artifacts are mitigated, and it will finally converge to a high-quality global image in the consensus of all distributed measurements. The proposed algorithm outperforms the joint sparsity-based composite imaging (JSC) algorithm in terms of artifacts mitigation. It can also reduce the computation and storage burden of large-scale imaging problems through its distributed and parallelizable optimization scheme.
This thesis explores a particular class of distributed optimization methods for various separable resource allocation problems, which are of high interest in a wide array of multi-agent settings. A distinctly motivating application for this thesis is real-time power dispatch of distributed energy resources for providing frequency control in a distribution grid or microgrid with high renewable energy penetration. In this application, it is paramount that agent data be shared as sparsely as possible in the interest of conserving user privacy, and it is required that algorithms scale gracefully as the network size increases to the order of thousands or millions of resources and devices. Distributed algorithms are naturally well-poised to address these challenges, in contrast to more traditional centralized algorithms which scale poorly and require global access to information. The class of distributed optimization methods explored here can be broadly described as Newton-like or second-order, implying utilization of second-derivative information of the cost functions, in contrast to well-studied gradient-based or first-order methods. We consider three formulations of separable resource-allocation problems and develop a Newton-like algorithm for each. The analysis and simulation studies in the subsequent chapters demonstrate the advantages of our approaches over existing methods; most commonly, we note that convergence rates are substantially improved. We supplement our algorithm development for these three problem formulations with a network design technique, in which we can construct a maximally-connected network by adding some edges to the underlying communication graph, and a real demonstration of distributed algorithms on a large set of heterogeneous devices on the UC San Diego microgrid.
This paper focuses on distributed learning-based control of decentralized multi-agent systems where the agents' dynamics are modeled by Gaussian Processes (GPs). Two fundamental problems are considered: the optimal design of experiment for concurrent learning of the agents' GP models, and the distributed coordination given the learned models. Using a Distributed Model Predictive Control (DMPC) approach, the two problems are formulated as distributed optimization problems, where each agent's sub-problem includes both local and shared objectives and constraints. To solve the resulting complex and non-convex DMPC problems efficiently, we develop an algorithm called Alternating Direction Method of Multipliers with Convexification (ADMM-C) that combines a distributed ADMM algorithm and a Sequential Convexification method. The computational efficiency of our proposed method comes from the facts that the computation for solving the DMPC problem is distributed to all agents and that efficient convex optimization solvers are used at the agents for solving the convexified sub-problems. We also prove that, under some technical assumptions, the ADMM-C algorithm converges to a stationary point of the penalized optimization problem. The effectiveness of our approach is demonstrated in numerical simulations of a multi-vehicle formation control example.
We study first-order optimization algorithms under the constraint that the descent direction is quantized using a pre-specified budget of $R$-bits per dimension, where $R \in (0 ,\infty)$. We propose computationally efficient optimization algorithms with convergence rates matching the information-theoretic performance lower bounds for: (i) Smooth and Strongly-Convex objectives with access to an Exact Gradient oracle, as well as (ii) General Convex and Non-Smooth objectives with access to a Noisy Subgradient oracle. The crux of these algorithms is a polynomial complexity source coding scheme that embeds a vector into a random subspace before quantizing it. These embeddings are such that with high probability, their projection along any of the canonical directions of the transform space is small. As a consequence, quantizing these embeddings followed by an inverse transform to the original space yields a source coding method with optimal covering efficiency while utilizing just $R$-bits per dimension. Our algorithms guarantee optimality for arbitrary values of the bit-budget $R$, which includes both the sub-linear budget regime ($R < 1$), as well as the high-budget regime ($R \geq 1$), while requiring $O\left(n^2\right)$ multiplications, where $n$ is the dimension. We also propose an efficient relaxation of this coding scheme using Hadamard subspaces that requires a near-linear time, i.e., $O\left(n \log n\right)$ additions.Furthermore, we show that the utility of our proposed embeddings can be extended to significantly improve the performance of gradient sparsification schemes. Numerical simulations validate our theoretical claims. Our implementations are available at https://github.com/rajarshisaha95/DistOptConstrComm.
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of computers via specialized message-passing protocols such as Ring All-Reduce. However, running these protocols at scale requires reliable high-speed networking that is only available in dedicated clusters. In contrast, many real-world applications, such as federated learning and cloud-based distributed training, operate on unreliable devices with unstable network bandwidth. As a result, these applications are restricted to using parameter servers or gossip-based averaging protocols. In this work, we lift that restriction by proposing Moshpit All-Reduce - an iterative averaging protocol that exponentially converges to the global average. We demonstrate the efficiency of our protocol for distributed optimization with strong theoretical guarantees. The experiments show 1.3x speedup for ResNet-50 training on ImageNet compared to competitive gossip-based strategies and 1.5x speedup when training ALBERT-large from scratch using preemptible compute nodes.
We are concerned with finding Nash Equilibria in agent-based multi-cluster games, where agents are separated into distinct clusters. While the agents inside each cluster collaborate to achieve a common goal, the clusters are considered to be virtual players that compete against each other in a non-cooperative game with respect to a coupled cost function. In such scenarios, the inner-cluster problem and the game between the clusters need to be solved simultaneously. Therefore, the resulting inter-cluster Nash Equilibrium should also be a minimizer of the social welfare problem inside the clusters. In this work, this setup is cast as a distributed optimization problem with sparse state information. Hence, critical information, such as the agent's cost functions, remain private. We present a distributed algorithm that converges with a linear rate to the optimal solution. Furthermore, we apply our algorithm to an extended cournot game to verify our theoretical results.
We consider a distributed non-convex optimization where a network of agents aims at minimizing a global function over the Stiefel manifold. The global function is represented as a finite sum of smooth local functions, where each local function is associated with one agent and agents communicate with each other over an undirected connected graph. The problem is non-convex as local functions are possibly non-convex (but smooth) and the Steifel manifold is a non-convex set. We present a decentralized Riemannian stochastic gradient method (DRSGD) with the convergence rate of $\mathcal{O}(1/\sqrt{K})$ to a stationary point. To have exact convergence with constant stepsize, we also propose a decentralized Riemannian gradient tracking algorithm (DRGTA) with the convergence rate of $\mathcal{O}(1/K)$ to a stationary point. We use multi-step consensus to preserve the iteration in the local (consensus) region. DRGTA is the first decentralized algorithm with exact convergence for distributed optimization on Stiefel manifold.
Large scale distributed optimization has become the default tool for the training of supervised machine learning models with a large number of parameters and training data. Recent advancements in the field provide several mechanisms for speeding up the training, including {\em compressed communication}, {\em variance reduction} and {\em acceleration}. However, none of these methods is capable of exploiting the inherently rich data-dependent smoothness structure of the local losses beyond standard smoothness constants. In this paper, we argue that when training supervised models, {\em smoothness matrices} -- information-rich generalizations of the ubiquitous smoothness constants -- can and should be exploited for further dramatic gains, both in theory and practice. In order to further alleviate the communication burden inherent in distributed optimization, we propose a novel communication sparsification strategy that can take full advantage of the smoothness matrices associated with local losses. To showcase the power of this tool, we describe how our sparsification technique can be adapted to three distributed optimization algorithms -- DCGD, DIANA and ADIANA -- yielding significant savings in terms of communication complexity. The new methods always outperform the baselines, often dramatically so.
We develop several new communication-efficient second-order methods for distributed optimization. Our first method, NEWTON-STAR, is a variant of Newton's method from which it inherits its fast local quadratic rate. However, unlike Newton's method, NEWTON-STAR enjoys the same per iteration communication cost as gradient descent. While this method is impractical as it relies on the use of certain unknown parameters characterizing the Hessian of the objective function at the optimum, it serves as the starting point which enables us design practical variants thereof with strong theoretical guarantees. In particular, we design a stochastic sparsification strategy for learning the unknown parameters in an iterative fashion in a communication efficient manner. Applying this strategy to NEWTON-STAR leads to our next method, NEWTON-LEARN, for which we prove local linear and superlinear rates independent of the condition number. When applicable, this method can have dramatically superior convergence behavior when compared to state-of-the-art methods. Finally, we develop a globalization strategy using cubic regularization which leads to our next method, CUBIC-NEWTON-LEARN, for which we prove global sublinear and linear convergence rates, and a fast superlinear rate. Our results are supported with experimental results on real datasets, and show several orders of magnitude improvement on baseline and state-of-the-art methods in terms of communication complexity.
With the increasing demand for large-scale training of machine learning models, consensus-based distributed optimization methods have recently been advocated as alternatives to the popular parameter server framework. In this paradigm, each worker maintains a local estimate of the optimal parameter vector, and iteratively updates it by waiting and averaging all estimates obtained from its neighbors, and then corrects it on the basis of its local dataset. However, the synchronization phase can be time consuming due to the need to wait for \textit{stragglers}, i.e., slower workers. An efficient way to mitigate this effect is to let each worker wait only for updates from the fastest neighbors before updating its local parameter. The remaining neighbors are called \textit{backup workers.} To minimize the globally training time over the network, we propose a fully distributed algorithm to dynamically determine the number of backup workers for each worker. We show that our algorithm achieves a linear speedup for convergence (i.e., convergence performance increases linearly with respect to the number of workers). We conduct extensive experiments on MNIST and CIFAR-10 to verify our theoretical results.