Abstract: Diffusion models are effective purification methods, where the noises or adversarial attacks are removed using generative approaches before pre-existing classifiers conducting classification tasks. However, the efficiency of diffusion models is still a concern, and existing solutions are based on knowledge distillation which can jeopardize the generation quality because of the small number of generation steps. Hence, we propose TendiffPure as a tensorized and compressed diffusion model for purification. Unlike the knowledge distillation methods, we directly compress U-Nets as backbones of diffusion models using tensor-train decomposition, which reduces the number of parameters and captures more spatial information in multi-dimensional data such as images. The space complexity is reduced from O(N²) to O(NR²) with R≤4 as the tensor-train rank and N as the number of channels. Experimental results show that TendiffPure can more efficiently obtain high-quality purification results and outperforms the baseline purification methods on CIFAR-10, Fashion-MNIST, and MNIST datasets for two noises and one adversarial attack.

TendiffPure：一种用于纯化的卷积张量链去噪扩散模型

[1]Carroll JD, Chang JJ, 1970. Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition. Psychometrika, 35(3):283-319.

[2]Croce F, Hein M, 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. Proc 37^th Int Conf on Machine Learning, Article 206.

[3]Dhariwal P, Nichol A, 2021. Diffusion models beat GANs on image synthesis. Proc 35^th Conf on Neural Information Processing Systems, p.8780-8794.

[4]Gao Q, Li ZL, Zhang JP, et al., 2023. CoreDiff: contextual error-modulated generalized diffusion model for low-dose CT denoising and generalization. IEEE Trans Med Imag, early access.

[5]Giovannetti V, Montangero S, Fazio R, 2008. Quantum multiscale entanglement renormalization ansatz channels. Phys Rev Lett, 101(18):180503.

[6]Hitchcock FL, 1927. The expression of a tensor or a polyadic as a sum of products. J Math Phys, 6(1-4):164-189.

[7]Ho J, Salimans T, 2021. Classifier-free diffusion guidance. Proc Workshop on Deep Generative Models and Downstream Applications.

[8]Ho J, Jain A, Abbeel P, 2020. Denoising diffusion probabilistic models. Proc 34^th Int Conf on Neural Information Processing Systems, Article 574.

[9]Hu EJ, Shen YL, Wallis P, et al., 2022. LoRA: low-rank adaptation of large language models. Proc 10^th Int Conf on Learning Representations.

[10]Krizhevsky A, Hinton G, 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto, Toronto, Canada.

[11]LeCun Y, Bottou L, Bengio Y, et al., 1998. Gradient-based learning applied to document recognition. Proc IEEE, 86(11):2278-2324.

[12]Li C, Sun Z, Yu JS, et al., 2019. Low-rank embedding of kernels in convolutional neural networks under random shuffling. Proc IEEE Int Conf on Acoustics, p.3022-3026.

[13]Luo YS, Zhao XL, Meng DY, et al., 2022. HLRTF: hierarchical low-rank tensor factorization for inverse problems in multi-dimensional imaging. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.19281-19290.

[14]Meng CL, Rombach R, Gao RQ, et al., 2023. On distillation of guided diffusion models. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.14297-14306.

[15]Nichol AQ, Dhariwal P, 2021. Improved denoising diffusion probabilistic models. Proc 38^th Int Conf on Machine Learning, p.8162-8171.

[16]Nie WL, Guo B, Huang YJ, et al., 2022. Diffusion models for adversarial purification. Proc 39^th Int Conf on Machine Learning, p.16805-16827.

[17]Oseledets IV, 2011. Tensor-train decomposition. SIAM J Sci Comput, 33(5):2295-2317.

[18]Ronneberger O, Fischer P, Brox T, 2015. U-Net: convolutional networks for biomedical image segmentation. Proc 18^th Int Conf on Medical Image Computing and Computer-Assisted Intervention, p.234-241.

[19]Song JM, Meng CL, Ermon S, 2021. Denoising diffusion implicit models. Proc 9^th Int Conf on Learning Representations.

[20]Song Y, Garg S, Shi JX, et al., 2020. Sliced score matching: a scalable approach to density and score estimation. Proc 35^th Uncertainty in Artificial Intelligence Conf, p.574-584.

[21]Song Y, Dhariwal P, Chen M, et al., 2023. Consistency models. Proc 40^th Int Conf on Machine Learning, Article 1335.

[22]Su JH, Byeon W, Kossaifi J, et al., 2020. Convolutional tensor-train LSTM for spatio-temporal learning. Proc 34^th Int Conf on Neural Information Processing Systems, Article 1150.

[23]Tucker LR, 1966. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279-311.

[24]Vahdat A, Kreis K, Kautz J, 2021. Score-based generative modeling in latent space. Proc 35^th Conf on Neural Information Processing Systems.

[25]Vincent P, 2011. A connection between score matching and denoising autoencoders. Neur Comput, 23(7):1661-1674.

[26]Xiao H, Rasul K, Vollgraf R, 2017. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. https://arxiv.org/abs/1708.07747

[27]Zhao QB, Zhou GX, Xie SL, et al., 2016. Tensor ring decomposition. https://arxiv.org/abs/1606.05535