Abstract: Isolated data islands are prevalent in intelligent automated optical inspection (AOI) systems, limiting the full utilization of data resources and impeding the potential of AOI systems. Establishing a collaborative ecology involving software providers, hardware manufacturers, and factories offers an encouraging solution to build a closed-loop data flow and achieve optimal data resource utilization. However, concerns about privacy issues, rights infringement, and threats from other participants present challenges in establishing an efficient and effective community. In this paper, we propose a novel framework, AOI-OPEN, which first creates a trustworthy AOI ecology to gather related entities with decentralized autonomous organization (DAO) mechanisms. Then, a parallel data pipeline is proposed to generate large-scale virtual samples from small-scale real data for AOI systems. Finally, federated learning (FL) is adopted to use the distributed data resources among multiple entities and build privacy-preserving big models. Experiments on defect classification tasks show that, with privacy preserved, AOI-OPEN greatly strengthens the utilization of distributed data resources and improves the accuracy of inspection models.

AOI-OPEN：基于去中心化自治组织的可信智能自动光学检测生态的联邦操作与控制

[1]Bommasani R, Hudson DA, Adeli E, et al., 2021. On the opportunities and risks of foundation models. https://arxiv.org/abs/2108.07258

[2]Cubero S, Aleixos N, Moltó E, et al., 2011. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioproc Technol, 4(4):487-504.

[3]Defard T, Setkov A, Loesch A, et al., 2021. PaDiM: a patch distribution modeling framework for anomaly detection and localization. Int Conf on Pattern Recognition, p.475-489.

[4]Dosovitskiy A, Beyer L, Kolesnikov A, et al., 2021. An image is worth 16×16 words: Transformers for image recognition at scale. Proc 9^th Int Conf on Learning Representations.

[5]Dwork C, Roth A, 2014. The algorithmic foundations of differential privacy. Found Trends Theor Comput Sci, 9(3-4):211-407.

[6]El Saddik A, 2018. Digital twins: the convergence of multimedia technologies. IEEE Multim, 25(2):87-92.

[7]Goldberg M, Schär F, 2023. Metaverse governance: an empirical analysis of voting within decentralized autonomous organizations. J Bus Res, 160: 113764.

[8]Gou JP, Yu BS, Maybank SJ, et al., 2021. Knowledge distillation: a survey. Int J Comput Vis, 129(6):1789-1819.

[9]Kim S, Kim W, Noh YK, et al., 2017. Transfer learning for automated optical inspection. Proc Int Joint Conf on Neural Networks, p.2517-2524.

[10]Kingma DP, Welling M, 2013. Auto-encoding variational Bayes. https://arxiv.org/abs/1312.6114

[11]Li JJ, Qin R, Olaverri-Monreal C, et al., 2023. Logistics 5.0: from intelligent networks to sustainable ecosystems. IEEE Trans Intell Veh, 8(7):3771-3774.

[12]Li T, Sahu AK, Talwalkar A, et al., 2020. Federated learning: challenges, methods, and future directions. IEEE Signal Process Mag, 37(3):50-60.

[13]Li YT, Guo JI, 2018. A VGG-16 based faster RCNN model for PCB error inspection in industrial AOI applications. Proc IEEE Int Conf on Consumer Electronics-Taiwan, p.1-2.

[14]Li ZY, Yang Q, 2011. System design for PCB defects detection based on AOI technology. Proc 4^th Int Congress on Image Signal Processing, p.1988-1991.

[15]Liao HC, Lim ZY, Hu YX, et al., 2018. Guidelines of automated optical inspection (AOI) system development. Proc IEEE 3^rd Int Conf on Signal Image Processing, p.362-366.

[16]Liu X, Wang X, Zhang WS, et al., 2017. Parallel data: from big data to data intelligence. Patt Recogn Artif Intell, 30(8):673-681 (in Chinese).

[17]Lu RS, Wu A, Zhang TD, et al., 2018. Review on automated optical (visual) inspection and its applications in defect detection. Acta Opt Sin, 38(8): 0815002.

[18]Miao QH, Lv YS, Huang M, et al., 2023. Parallel learning: overview and perspective for computational learning across Syn2Real and Sim2Real. IEEE/CAA J Autom Sin, 10(3):603-631.

[19]Monrat AA, Schelén O, Andersson K, 2019. A survey of blockchain from the perspectives of applications, challenges, and opportunities. IEEE Access, 7:117134-117151.

[20]Ouyang LW, Wang FY, Tian YL, et al., 2023. Artificial identification: a novel privacy framework for federated learning based on blockchain. IEEE Trans Comput Soc Syst, 10(6):3576-3585.

[21]Pan SJ, Yang Q, 2010. A survey on transfer learning. IEEE Trans Knowl Data Eng, 22(10):1345-1359.

[22]Reed JM, Hutchinson S, 1996. Image fusion and subpixel parameter estimation for automated optical inspection of electronic components. IEEE Trans Ind Electron, 43(3):346-354.

[23]Tian YL, Wang JG, Wang YT, et al., 2022. Federated vehicular Transformers and their federations: privacy-preserving computing and cooperation for autonomous driving. IEEE Trans Intell Veh, 7(3):456-465.

[24]Touvron H, Cord M, Douze M, et al., 2021. Training data-efficient image Transformers & distillation through attention. Proc 38^th Int Conf on Machine Learning, p.10347-10357.

[25]Vepakomma P, Gupta O, Swedish T, et al., 2018. Split learning for health: distributed deep learning without sharing raw patient data. https://arxiv.org/abs/1812.00564

[26]Wang FY, Qin R, Chen YZ, et al., 2021a. Federated ecology: steps toward confederated intelligence. IEEE Trans Comput Soc Syst, 8(2):271-278.

[27]Wang FY, Zhang WS, Tian YL, et al., 2021b. Federated data: toward new generation of credible and trustable artificial intelligence. IEEE Trans Comput Soc Syst, 8(3):538-545.

[28]Wang FY, Zhu J, Qin R, et al., 2021c. Federated control: toward information security and rights protection. IEEE Trans Comput Soc Syst, 8(4):793-798.

[29]Wang KF, Gou C, Zheng NN, et al., 2017. Parallel vision for perception and understanding of complex scenes: methods, framework, and perspectives. Artif Intell Rev, 48(3):299-329.

[30]Wang L, Yoon KJ, 2022. Knowledge distillation and student-teacher learning for visual intelligence: a review and new outlooks. IEEE Trans Patt Anal Mach Intell, 44(6):3048-3068.

[31]Wang S, Yuan Y, Wang X, et al., 2018. An overview of smart contract: architecture, applications, and future trends. Proc IEEE Intelligent Vehicles Symp, p.108-113.

[32]Wang S, Ding WW, Li JJ, et al., 2019. Decentralized autonomous organizations: concept, model, and applications. IEEE Trans Comput Soc Syst, 6(5):870-878.

[33]Wang WC, Chen SL, Chen LB, et al., 2017. A machine vision based automatic optical inspection system for measuring drilling quality of printed circuit boards. IEEE Access, 5:10817-10833.

[34]Wang Y, Xiong W, Cheng JR, et al., 2015. Vision based hole crack detection. Proc IEEE 10^th Conf on Industrial Electronics and Applications, p.1932-1936.

[35]Wang YT, Tian YL, Wang JG, et al., 2022. Integrated inspection of QoM, QoP, and QoS for AOI industries in metaverses. IEEE/CAA J Autom Sin, 9(12):2071-2078.

[36]Wang ZH, Li HJ, Li JG, et al., 2024. Federated learning on non-IID and long-tailed data via dual-decoupling. Front Inform Technol Electron Eng, 25(5):728-741.

[37]Weiss K, Khoshgoftaar TM, Wang DD, 2016. A survey of transfer learning. J Big Data, 3(1):9.

[38]Wu FP, Zhang XM, Kuan YC, et al., 2008. An AOI algorithm for PCB based on feature extraction. Proc 7^th World Congress on Intelligent Control and Automation, p.240-247.

[39]Yan K, Shu NN, Wu T, et al., 2024. A survey of energy-efficient strategies for federated learning in mobile edge computing. Front Inform Technol Electron Eng, 25(5):645-663.

[40]Yang Q, Liu Y, Cheng Y, et al., 2019. Federated Learning. Springer, Cham, Germany, p.1-207.

[41]Yang YT, Pan LH, Ma JX, et al., 2020. A high-performance deep learning algorithm for the automated optical inspection of laser welding. Appl Sci, 10(3):933.

[42]Yao SY, Yu JR, Yu Y, et al., 2023. Towards integrated traffic control with operating decentralized autonomous organization. Proc IEEE 26^th Int Conf on Intelligent Transportation Systems, p.6126-6131.

[43]Yu HF, Bai XP, 2024. Identity-based searchable attribute signcryption in lattice for a blockchain-based medical system. Front Inform Technol Electron Eng, 25(3):461-471.

[44]Yuan K, Guo SP, Liu ZW, et al., 2021. Incorporating convolution designs into visual Transformers. Proc IEEE/CVF Int Conf on Computer Vision, p.579-588.

[45]Yuan L, Chen YP, Wang T, et al., 2021. Tokens-to-token ViT: training vision Transformers from scratch on ImageNet. Proc IEEE/CVF Int Conf on Computer Vision, p.558-567.

[46]Zhang BY, Chen T, Wang X, et al., 2024. Decoding activist public opinion in decentralized self-organized protests using LLM. IEEE Trans Comput Soc Syst, 11(4):5525-5535.

[47]Zhang R, Xue R, Liu L, 2020. Security and privacy on blockchain. ACM Comput Surv, 52(3):51.

[48]Zhang WS, Sun G, Xu L, et al., 2022. A trustworthy safety inspection framework using performance-security balanced blockchain. IEEE Int Things J, 9(11):8178-8190.

[49]Zhang WS, Chen X, He K, et al., 2023. Semi-asynchronous personalized federated learning for short-term photovoltaic power forecasting. Dig Commun Netw, 9(5):1221-1229.

[50]Zhang WS, Wang YQ, Chen LM, et al., 2024. Dynamic circular network-based federated dual-view learning for multivariate time series anomaly detection. Bus Inform Syst Eng, 66(1):19-42.

[51]Zhuang FZ, Qi ZY, Duan KY, et al., 2021. A comprehensive survey on transfer learning. Proc IEEE, 109(1):43-76.