Abstract: Integration testing is an integral part of software testing. Prior studies have focused on reducing test cost in integration test order generation. However, there are no studies concerning the testing priorities of critical classes when generating integration test orders. Such priorities greatly affect testing efficiency. In this study, we propose an effective strategy that considers both test cost and efficiency when generating test orders. According to a series of dynamic execution scenarios, the software is mapped into a multi-layer dynamic execution network (MDEN) model. By analyzing the dynamic structural complexity, an evaluation scheme is proposed to quantify the class testing priority with the defined class risk index. Cost–benefit analysis is used to perform cycle-breaking operations, satisfying two principles: assigning higher priorities to higher-risk classes and minimizing the total complexity of test stubs. We also present a strategy to evaluate the effectiveness of integration test order algorithms by calculating the reduction of software risk during their testing process. Experiment results show that our approach performs better across software of different scales, in comparison with the existing algorithms that aim only to minimize test cost. Finally, we implement a tool, ITOsolution, to help practitioners automatically generate test orders.

基于成本收益分析的集成测试序列生成优化方法

[1]Abdurazik A, Offutt J, 2006. Coupling-based class integration and test order. Proc Int Workshop on Automation of Software Test, p.50-56.

[2]Al Mannai WI, Lewis TG, 2008. A general defender-attacker risk model for networks. J Risk Finance, 9(3):244-261.

[3]Amland S, 2000. Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J Syst Softw, 53(3):287-295.

[4]Assunção WKG, Colanzi TE, Vergilio SR, et al., 2014. A multi-objective optimization approach for the integration and test order problem. Inform Sci, 267:119-139.

[5]Bang L, Aydin A, Bultan T, 2015. Automatically computing path complexity of programs. Proc 10^th Joint Meeting on Foundations of Software Engineering, p.61-72.

[6]Bansal P, Sabharwal S, Sidhu P, 2009. An investigation of strategies for finding test order during integration testing of object oriented applications. Proc Int Conf on Methods and Models in Computer Science, p.1-8.

[7]Binder RV, 1996. Testing object-oriented software: a survey. Softw Test Verif Reliab, 6(3-4):125-252.

[8]Bowring JF, Rehg JM, Harrold MJ, 2004. Active learning for automatic classification of software behavior. ACM SIGSOFT Softw Eng Notes, 29(4):195-205.

[9]Briand LC, Jie F, Labiche Y, 2002. Experimenting with genetic algorithms to devise optimal integration test orders. Technical Report, No. SCE-02-03, Carleton University, Ottawa, Canada.

[10]Briand LC, Labiche Y, Wang YH, 2003. An investigation of graph-based class integration test order strategies. IEEE Trans Softw Eng, 29(7):594-607.

[11]Cai KY, Yin BB, 2009. Software execution processes as an evolving complex network. Inform Sci, 179(12):1903-1928.

[12]da Veiga Cabral R, Pozo A, Vergilio SR, 2010. A Pareto ant colony algorithm applied to the class integration and test order problem. Proc 22^nd IFIP WG 6.1 Int Conf on Testing Software and Systems, p.16-29.

[13]Deb K, Pratap A, Agarwal S, et al., 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput, 6(2):182-197.

[14]Floyd RW, 1962. Algorithm 97: shortest path. Commun ACM, 5(6):345.

[15]Frankl PG, Weyuker EJ, 2000. Testing software to detect and reduce risk. J Syst Softw, 53(3):275-286.

[16]Goseva-Popstojanova K, Hassan A, Guedem A, et al., 2003. Architectural-level risk analysis using UML. IEEE Trans Softw Eng, 29(10):946-960.

[17]Hao D, Zhang L, Zang L, et al., 2016. To be optimal or not in test-case prioritization. IEEE Trans Softw Eng, 42(5):490-505.

[18]Henry S, Kafura D, 1981. Software structure metrics based on information flow. IEEE Trans Softw Eng, 7(5):510-518.

[19]Huang CY, Lyu MR, 2005. Optimal testing resource allocation, and sensitivity analysis in software development. IEEE Trans Reliab, 54(4):592-603.

[20]Jiang SJ, Zhang YM, Li HY, et al., 2011. An approach for inter-class integration test order determination based on coupling measures. Chin J Comput, 34(6):1062-1074.

[21]Jiang SJ, Zhang M, Zhang YM, et al., 2021. An integration test order strategy to consider control coupling. IEEE Trans Softw Eng, 47(7):1350-1367.

[22]Jorgensen PC, Erickson C, 1994. Object-oriented integration testing. Commun ACM, 37(9):30-38.

[23]Knowles JD, Corne DW, 2000. Approximating the nondominated front using the Pareto archived evolution strategy. Evol Comput, 8(2):149-172.

[24]Kung D, Gao J, Hsia P, et al., 1995. A test strategy for object-oriented programs. Proc 19^th Annual Int Computer Software and Applications Conf, p.239-244.

[25]Le Traon Y, Jeron T, Jezequel JM, et al., 2000. Efficient object-oriented integration and regression testing. IEEE Trans Reliab, 49(1):12-25.

[26]Lipow M, 1982. Number of faults per line of code. IEEE Trans Softw Eng, 8(4):437-439.

[27]McCabe TJ, 1976. A complexity measure. IEEE Trans Softw Eng, 2(4):308-320.

[28]Monden A, Hayashi T, Shinoda S, et al., 2013. Assessing the cost effectiveness of fault prediction in acceptance testing. IEEE Trans Reliab, 39(10):1345-1357.

[29]Myers CR, 2003. Software systems as complex networks: structure, function, and evolvability of software collaboration graphs. Phys Rev E, 68(2):046116.

[30]NASA, 1999. Pyroshock Test Crite. Technical Report, No. NASA-STD-7003A. NASA, Washington, USA.

[31]Sharma C, Sibal R, 2013. Application of different metaheuristic techniques for finding optimal test order during integration testing of object oriented systems and their comparative study. Int J Soft Comput Eng, 3(12):1-19.

[32]Tai KC, Daniels JF, 1999. Interclass test order for object-oriented software. J Obj-Orient Progr, 12(4):18-25.

[33]Vergilio SR, Pozo A, Árias JCG, et al., 2012. Multi-objective optimization algorithms applied to the class integration and test order problem. Int J Softw Tools Technol Transf, 14(4):461-475.

[34]Walters C, Ludwig D, 1994. Calculation of Bayes posterior probability distributions for key population parameters. Can J Fish Aquat Sci, 51(3):713-722.

[35]Wang Y, Zhu ZL, Yang B, et al., 2018a. Using reliability risk analysis to prioritize test cases. J Syst Softw, 139:14-31.

[36]Wang Y, Zhu ZL, Yu H, et al., 2018b. Risk analysis on multi-granular flow network for software integration testing. IEEE Trans Circ Syst II Expr Briefs, 65(8):1059-1063.

[37]Wang ZS, Li BX, Wang LL, et al., 2011. A brief survey on automatic integration test order generation. Proc 23^rd Int Conf on Software Engineering & Knowledge Engineering, p.254-257.

[38]Weyuker EJ, 1988. Evaluating software complexity measures. IEEE Trans Softw Eng, 14(9):1357-1365.

[39]Xu C, Qin Y, Yu P, et al., 2020. Theories and techniques for growing software: paradigm and beyond. Sci Sin Inform, 50(11):1595-1611.