Unsupervised Domain Adaptation for Skeleton Recognition with Fourier Analysis

Unsupervised domain adaptation (UDA) methods have recently been explored for their use in skeleton recognition tasks. Much work along this line has been focusing on the “close set” problems, which often deviate from reality as human actions vary in application scenarios. Thus, there remains a need to thoroughly study the “open-set” problems with UDA methods for skeleton recognition, aiming to support those models capable of self-adapting to action changes in different scenarios. To this end, we delve into the “open-set” problems from a feature alignment perspective under UDA settings in reaching domain and class alignment. Specifically, the domain-wise alignment was achieved by the Maximum Mean Discrepancy (MMD) combined with supervision signals from the source domain, which form clear feature boundaries between the “known” and “unknown” classes. Then, the class-wise alignment was achieved by contrastive learning methods, which are distinguished from previous binary classification methods, in reaching compactness inside of “unknown” or “known” classes. Moreover, we conducted the Fourier Analysis during the evaluation phases to verify the model’s robustness. To our knowledge, we are the first to apply the Fourier Heatmap in UDA methods for skeleton recognition. The heatmap visualizes the model’s sensitivity steered for interpretability. Significant performance improvements are observed on the NTU and PKU datasets when adding the domain-wise alignment module to other contrastive learning methods. Furthermore, experimental results demonstrate that our approach, termed CStrCRL-UDA, is consistent with robustness and efficiency on these two benchmark datasets.