FG-DSA: Frequency-Guided Depthwise Separable Attention for Efficient Dense Prediction

원문 초록

Dense prediction and generative modeling in resource-constrained scenarios remain challenging, as repeated downsampling substantially reduces spatial detail and degrades boundaries. Although recent attention mechanisms have alleviated the problem of semantic ambiguity, they often fail to preserve high-frequency features or do so at the expense of a very high computational cost. In this paper, we propose the Frequency Guided Depthwise Separable Attention for Efficient Dense Prediction (FG-DSA), a novel framework for efficient dense visual understanding and generation. FG-DSA employs a 2D FFT high-pass filtering module to explicitly inject frequency domain priors into the spatial attention mechanism, allowing lightweight networks to recover sharp boundary transitions with linear complexity. A range of experiments show that FG-DSA is effective at recovering fine object boundaries and leads to a consistent performance gain in the task of semantic segmentation. Importantly, we show that FGA-DSA can be used as a module in diffusion-based generative U-Nets, leading to a significant improvement in the detail of the generated images with very little additional parameter and Floating Point Operation (FLOP) cost.