PyTorchKR Papers

최신 논문

총 2편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2607.11607v2cs.AI

When Can Learned Uncertainty Be Trusted? An Audit of Distributional Reinforcement Learning

Many learned predictive models represent their uncertainty as a probability distribution, and planners and policies use this distribution to judge risk. However, the fact that a model outputs a distribution does not tell us whether that distribution captures the uncertainty in the environment which is relevant for a decision. In this work we study this question in distributional reinforcement learning, where the learned return distribution P_{\theta}(Z | s,a) can be compared directly with an environment-grounded distribution P_env(Z | s,a) estimated by snapshot-restart Monte Carlo. Our audit first screens for the claims that a risk-sensitive user would act upon, using the excess Wasserstein gap between the top two actions (which is equal to the mass by which first-order stochastic dominance is violated). Each claim is then tested against ground truth using permutation nulls, bootstrap refutation and FDR control. Across QR-DQN, C51 and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, their placement is statistically indistinguishable from a truth-blind placement, and almost none of them can be confirmed. A similar pattern is seen for a pretrained full-Atari QR-DQN. On positive controls of known magnitude the same audit confirms 96-100% of real claims (correlation 0.89-0.92), so it is able to detect real risk when it is present. Acting on the learned CVaR advice at the most-flagged states gives results ranging from beneficial to significantly worse than chance. Ensembling, risk-sensitive training and recalibration do not fix the problem, and the recalibration experiment indicates that the learned spread has no usable information about the true risk. Based on these results, we suggest that uncertainty estimates, including those of learned world models, should be checked against realized outcomes at the decisions where they are used..

arXiv:2607.01880v2cs.LG

Learning the Supports for Categorical Critic in Reinforcement Learning

Gaussian Histogram Loss(HL-Gauss)를 강화학습(RL)의 가치 함수 학습에 적용할 때 사전에 고정된 지지 구간(support interval)을 지정해야 하는 문제를 해결하기 위해, 이 연구는 지지 구간의 하한과 상한을 동적으로 학습하는 DySEL 알고리즘을 제안한다. 제약 최적화 문제를 적대적 min-max 게임으로 공식화하여 지지 구간 너비를 최소화하면서도 목표 분포를 충분히 포함하도록 하며, 연속 제어 태스크에서 고정 지지 구간 기반 HL-Gauss와 동등하거나 일부 태스크에서 더 우수한 성능을 달성한다.