Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic data sets possible. We evaluate our approach on two challenging data sets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task, and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.
Robotic systems, working together as a team, are becoming valuable players in different real-world applications, from disaster response to warehouse fulfillment services. Centralized solutions for coordinating multi-robot teams often suffer from poor scalability and vulnerability to communication disruptions. This paper develops a decentralized multi-agent task allocation (Dec-MATA) algorithm for multi-robot applications. The task planning problem is posed as a maximum-weighted matching of a bipartite graph, the solution of which using the blossom algorithm allows each robot to autonomously identify the optimal sequence of tasks it should undertake. The graph weights are determined based on a soft clustering process, which also plays a problem decomposition role seeking to reduce the complexity of the individual-agents' task assignment problems. To evaluate the new Dec-MATA algorithm, a series of case studies (of varying complexity) are performed, with tasks being distributed randomly over an observable 2D environment. A centralized approach, based on a state-of-the-art MILP formulation of the multi-Traveling Salesman problem is used for comparative analysis. While getting within 7-28% of the optimal cost obtained by the centralized algorithm, the Dec-MATA algorithm is found to be 1-3 orders of magnitude faster and minimally sensitive to task-to-robot ratios, unlike the centralized algorithm.
We aim to reduce the burden of programming and deploying autonomous systems to work in concert with people in time-critical domains, such as military field operations and disaster response. Deployment plans for these operations are frequently negotiated on-the-fly by teams of human planners. A human operator then translates the agreed upon plan into machine instructions for the robots. We present an algorithm that reduces this translation burden by inferring the final plan from a processed form of the human team's planning conversation. Our approach combines probabilistic generative modeling with logical plan validation used to compute a highly structured prior over possible plans. This hybrid approach enables us to overcome the challenge of performing inference over the large solution space with only a small amount of noisy data from the team planning session. We validate the algorithm through human subject experimentation and show we are able to infer a human team's final plan with 83% accuracy on average. We also describe a robot demonstration in which two people plan and execute a first-response collaborative task with a PR2 robot. To the best of our knowledge, this is the first work that integrates a logical planning technique within a generative model to perform plan inference.
Many multi-agent coordination problems can be represented as DCOPs. Motivated by task allocation in disaster response, we extend standard DCOP models to consider uncertain task rewards where the outcome of completing a task depends on its current state, which is randomly drawn from unknown distributions. The goal of solving this problem is to find a solution for all agents that minimizes the overall worst-case loss. This is a challenging problem for centralized algorithms because the search space grows exponentially with the number of agents and is nontrivial for standard DCOP algorithms we have. To address this, we propose a novel decentralized algorithm that incorporates Max-Sum with iterative constraint generation to solve the problem by passing messages among agents. By so doing, our approach scales well and can solve instances of the task allocation problem with hundreds of agents and tasks.
Disaster response agencies have started to incorporate social media as a source of fast-breaking information to understand the needs of people affected by the many crises that occur around the world. These agencies look for tweets from within the region affected by the crisis to get the latest updates of the status of the affected region. However only 1% of all tweets are geotagged with explicit location information. First responders lose valuable information because they cannot assess the origin of many of the tweets they collect. In this work we seek to identify non-geotagged tweets that originate from within the crisis region. Towards this, we address three questions: (1) is there a difference between the language of tweets originating within a crisis region and tweets originating outside the region, (2) what are the linguistic patterns that can be used to differentiate within-region and outside-region tweets, and (3) for non-geotagged tweets, can we automatically identify those originating within the crisis region in real-time?
Microblogging platforms such as Twitter provide active communication channels during mass convergence and emergency events such as earthquakes, typhoons. During the sudden onset of a crisis situation, affected people post useful information on Twitter that can be used for situational awareness and other humanitarian disaster response efforts, if processed timely and effectively. Processing social media information pose multiple challenges such as parsing noisy, brief and informal messages, learning information categories from the incoming stream of messages and classifying them into different classes among others. One of the basic necessities of many of these tasks is the availability of data, in particular human-annotated data. In this paper, we present human-annotated Twitter corpora collected during 19 different crises that took place between 2013 and 2015. To demonstrate the utility of the annotations, we train machine learning classifiers. Moreover, we publish first largest word2vec word embeddings trained on 52 million crisis-related tweets. To deal with tweets language issues, we present human-annotated normalized lexical resources for different lexical variations.
During natural or man-made disasters, humanitarian response organizations look for useful information to support their decision-making processes. Social media platforms such as Twitter have been considered as a vital source of useful information for disaster response and management. Despite advances in natural language processing techniques, processing short and informal Twitter messages is a challenging task. In this paper, we propose to use Deep Neural Network (DNN) to address two types of information needs of response organizations: 1) identifying informative tweets and 2) classifying them into topical classes. DNNs use distributed representation of words and learn the representation as well as higher level features automatically for the classification task. We propose a new online algorithm based on stochastic gradient descent to train DNNs in an online fashion during disaster situations. We test our models using a crisis-related real-world Twitter dataset.
High resolution datasets of population density which accurately map sparsely-distributed human populations do not exist at a global scale. Typically, population data is obtained using censuses and statistical modeling. More recently, methods using remotely-sensed data have emerged, capable of effectively identifying urbanized areas. Obtaining high accuracy in estimation of population distribution in rural areas remains a very challenging task due to the simultaneous requirements of sufficient sensitivity and resolution to detect very sparse populations through remote sensing as well as reliable performance at a global scale. Here, we present a computer vision method based on machine learning to create population maps from satellite imagery at a global scale, with a spatial sensitivity corresponding to individual buildings and suitable for global deployment. By combining this settlement data with census data, we create population maps with ~30 meter resolution for 18 countries. We validate our method, and find that the building identification has an average precision and recall of 0.95 and 0.91, respectively and that the population estimates have a standard error of a factor ~2 or less. Based on our data, we analyze 29 percent of the world population, and show that 99 percent lives within 36 km of the nearest urban cluster. The resulting high-resolution population datasets have applications in infrastructure planning, vaccination campaign planning, disaster response efforts and risk analysis such as high accuracy flood risk analysis.
대규모 재난이 발생하면 대응 인력은 몇 시간 내에 어떤 건물이 피해를 입었는지 나타내는 지도가 필요합니다. 공개 벤치마크에서 우수한 성능을 보이는 모델들은 재난 전후로 매칭된 이미지와 유사한 과거 사건에서 추출한 학습 세트를 전제로 하지만, 새로운 재난의 첫날에는 이 두 가지 모두 일반적으로 확보할 수 없습니다. 우리는 HASTE(High-speed Assessment and Satellite Tracking for Emergencies)를 소개합니다. 이는 머신러닝 엔지니어가 아닌 분석가들도 재난 후 위성 이미지로부터 건물별 피해 지도를 생성할 수 있게 해주는 노코드(no-code) 웹 플랫폼입니다. HASTE는 하나의 인터페이스를 공유하는 두 가지 방법을 구현합니다. 첫 번째 방법은 사용자가 재난 후 장면에 폴리곤을 레이블링하면, 해당 단일 장면에서 소규모 시맨틱 분할(semantic segmentation) 모델을 학습하고 전체 이미지에 실행한 뒤 픽셀별 출력을 기존 건물 풋프린트(building footprints)와 결합합니다. 두 번째 방법은 사전학습된 비전 모델로 모든 풋프린트를 임베딩하고, 사용자가 소수의 건물만 레이블링하면 브라우저에서 로지스틱 회귀(logistic regression)를 적합하여 나머지 장면을 몇 초 만에 평가합니다. 우리는 플랫폼과 두 가지 방법, 그리고 이를 지원하는 엔지니어링을 설명합니다. 또한 xBD에 대한 예비 실험 결과를 보고하는데, 풋프린트에 걸쳐 풀링된 파운데이션 모델(foundation model) 임베딩이 재난 후 이미지만으로도 손상된 건물과 온전한 건물을 구분하며, 레이블의 1/20만으로 완전 지도 학습된 ResNet-50 기준선과 동등한 성능을 보임을 확인하였습니다. HASTE와 그 전신 시스템들은 2023년 이후 지진, 허리케인, 사이클론, 홍수, 산불, 토네이도를 포함한 30건 이상의 실제 재난 대응을 지원하였으며, 이미지가 확보된 후 수 시간에서 수 일 내에 인도주의 파트너들에게 결과를 제공하였습니다. 마지막으로 비전-언어(vision-language) 평가, 능동 학습(active learning), 도로 및 기타 인프라를 위한 피해 모델을 포함하여 가장 유망하다고 판단되는 향후 방향을 제시합니다. HASTE는 https://github.com/microsoft/haste 에서 오픈 소스로 공개되어 있습니다.
대규모 언어 모델(LLM)은 조직 경계를 넘나들며 협력하는 자율 에이전트로 빠르게 진화하고 있으며, 데이터 소유권을 포기하지 않으면서도 분산된 전문성을 요구하는 공동 재난 대응, 공급망 최적화 및 기타 과업을 가능하게 한다. 그러나 교차 도메인 협업은 현재의 정렬(alignment) 및 격리(containment) 기법이 전제로 하는 통합된 신뢰 가정을 무너뜨린다. 격리된 상태에서는 양성인 에이전트라도 신뢰할 수 없는 피어(peer)로부터 메시지를 수신하면 비밀을 유출하거나 정책을 위반할 수 있으며, 이는 고전적인 소프트웨어 버그가 아니라 창발적 다중 에이전트 역학에 의해 유발되는 위험을 낳는다. 본 포지션 논문(position paper)은 교차 도메인 다중 에이전트 LLM 시스템의 보안 의제를 정리한다. 우리는 새로운 보안 과제의 7가지 범주를 제시하고, 각 범주에 대해 그럴듯한 공격, 보안 평가(evaluation) 지표, 향후 연구 지침도 함께 제시한다.
의사결정과 관련된 건물 피해 평가는 재난 이후 자원 우선순위 결정 및 복구에 있어 매우 중요하지만, 대부분의 자동화 방법은 피해를 단일 심각도 척도(피해 없음, 경미, 심각, 전파)로 평탄화하거나, 신흥 재해에는 종종 이용 불가능한 사전·사후 이벤트 이미지 쌍을 필요로 한다. 이 논문은 Damage-TriageFormer를 제시한다. 이는 심각도 척도 대신 피해 유형(damage typology)을 산출하는 단일 이미지, 사후 이벤트, 건물 외곽선 조건부 모델이다. 본 연구의 기여는 다음과 같다: (1) DamageTriage-Bench—2018년 Hurricane Michael, 2024년 Hurricane Helene, 2025년 로스앤젤레스 산불 복합 재해에 걸친 NOAA 긴급 대응 이미지로 구축된 새로운 벤치마크로, 지붕 피해와 구조적 피해를 구분하고 각각에서 부분 피해와 전체 피해를 구별하는 5개 유형 클래스를 포함; (2) Damage-TriageFormer—더 높은 해상도의 인스턴스 풀링을 위한 Simple Feature Pyramid, 2단계 게이팅 피해 헤드, 보조 심각도 회귀 목적함수를 추가하여 DINOv3 ViT-L 백본을 확장한 모델. 본 모델은 검증 세트에서 매크로 F1 0.624, 홀드아웃 계층화 테스트 세트에서 0.619를 달성하며, 운용적 분류가 가장 필요한 곳에서 가장 강한 성능을 보였다. 클래스별 F1은 피해 없는 건물에서 0.91, 구조 전파에서 0.84이다. 희귀 클래스인 Total Roof Damage는 제한된 사례 수와 내재적으로 모호한 레이블 경계로 인해 여전히 어렵지만, 본 결과는 단일 이미지 사후 이벤트 이미지가 실행 가능한 건물 피해 유형 분류를 지원할 수 있음을 보여주며, 사전 이벤트 참조 이미지 없이도 표적화된 긴급 대응 및 자원 배분을 가능하게 한다.