Abstract: Task-oriented virtual assistants are software systems that provide users with a natural language interface to complete domain-specific tasks. With the recent technological advances in natural language processing and machine learning, an increasing number of task-oriented virtual assistants have been developed. However, due to the well-known complexity and difficulties of the natural language understanding problem, it is challenging to manage a task-oriented virtual assistant software project. Meanwhile, the management and experience related to the development of virtual assistants are hardly studied or shared in the research community or industry, to the best of our knowledge. To bridge this knowledge gap, in this paper, we share our experience and the lessons that we have learned at managing a task-oriented virtual assistant software project at Microsoft. We believe that our practices and the lessons learned can provide a useful reference for other researchers and practitioners who aim to develop a virtual assistant system. Finally, we have developed a requirement management tool, named SpecSpace, which can facilitate the management of virtual assistant projects.

管理面向任务的虚拟助手软件系统的经验性研究

[1]Amershi S, Begel A, Bird C, et al., 2019. Software engineering for machine learning: a case study. Proc IEEE/ACM 41^st Int Conf on Software Engineering: Software Engineering in Practice, p.291-300.

[2]Arpteg A, Brinne B, Crnkovic-Friis L, et al., 2018. Software engineering challenges of deep learning. Proc 44^th Euromicro Conf on Software Engineering and Advanced Applications, p.50-59.

[3]Bender EM, Koller A, 2020. Climbing towards NLU: on meaning, form, and understanding in the age of data. Proc 58^th Annual Meeting of the Association for Computational Linguistics, p.5185-5198.

[4]Bonwell CC, Eison JA, 1991. Active Learning: Creating Excitement in the Classroom. ERIC Number ED336049. The George Washington University, Washington, USA.

[5]Bradley AJ, 2020. Brace Yourself for an Explosion of Virtual Assistants. https://blogs.gartner.com/anthony_bradley/2020/08/10/brace-yourself-for-an-explosion-of-virtual-assistants/ [Accessed on Aug. 10, 2020].

[6]Breck E, Cai SQ, Nielsen E, et al., 2017. The ML test score: a rubric for ML production readiness and technical debt reduction. Proc IEEE Int Conf on Big Data, p.1123-1132.

[7]Campagna G, Xu SL, Moradshahi M, et al., 2019. Genie: a generator of natural language semantic parsers for virtual assistant commands. Proc 40^th ACM SIGPLAN Conf on Programming Language Design and Implementation, p.394-410.

[8]Dhamdhere K, McCurley KS, Nahmias R, et al., 2017. Analyza: exploring data with conversation. Proc 22^nd Int Conf on Intelligent User Interfaces, p.493-504.

[9]Facebook, 2020. Surveybot. https://surveybot.io/ [Accessed on Aug. 10, 2020].

[10]Gao Y, Lou JG, Zhang DM, 2019. A hybrid semantic parsing approach for tabular data analysis. https://arxiv.org/abs/1910.10363

[11]Hains G, Jakobsson A, Khmelevsky Y, 2018. Towards formal methods and software engineering for deep learning: security, safety and productivity for DL systems development. Proc Annual IEEE Int Systems Conf, p.1-5.

[12]Horkoff J, 2019. Non-functional requirements for machine learning: challenges and new directions. Proc 27^th Int Requirements Engineering Conf, p.386-391.

[13]Huang XW, Kroening D, Kwiatkowska M, et al., 2018. Safety and trustworthiness of deep neural networks: a survey. https://arxiv.org/abs/1812.08342v1

[14]Islam J, Nguyen HA, Pan R, et al., 2019. What do developers ask about ML libraries? A large-scale study using stack overflow. https://arxiv.org/abs/1906.11940

[15]Krishnan S, Wang JN, Wu E, et al., 2016. ActiveClean: interactive data cleaning for statistical modeling. Proc VLDB Endow, 9(12):948-959.

[16]Krishnan S, Franklin MJ, Goldberg K, et al., 2017. BoostClean: automated error detection and repair for machine learning. https://arxiv.org/abs/1711.01299

[17]Lee DTS, Zhou ZQ, Tse TH, 2020. Metamorphic robustness testing of Google Translate. Proc 42^nd Int Conf on Software Engineering Workshops, p.388-395.

[18]Marijan D, Gotlieb A, Ahuja MK, 2019. Challenges of testing machine learning based systems. Proc IEEE Int Conf on Artificial Intelligence Testing, p.101-102.

[19]Mason L, Baxter J, Bartlett PL, et al., 1999. Boosting algorithms as gradient descent. Proc 12^th Int Conf on Neural Information Processing Systems, p.512-518.

[20]Masuda S, Ono K, Yasue T, et al., 2018. A survey of software quality for machine learning applications. IEEE Int Conf on Software Testing, Verification and Validation Workshops, p.279-284.

[21]Microsoft, 2020. Power BI. https://powerbi.microsoft.com/ [Accessed on Aug. 10, 2020].

[22]Oram R, 2019. Meeting Edward: Chatbots and the Changing Face of the Hotel Guest Experience. https://blogs.oracle.com/hospitality/chatbots-and-the-changing-the-face-of-the-hotel-guest-experience [Accessed on Aug. 10, 2020].

[23]Polyzotis N, Roy S, Whang SE, et al., 2018. Data lifecycle challenges in production machine learning: a survey. ACM SIGMOD Rec, 47(2):17-28.

[24]Radford A, Wu J, Child R, et al., 2019. Language Models are Unsupervised Multitask Learners. https://openai.com/blog/ [Accessed on Jan. 1, 2020].

[25]Schapire RE, 1990. The strength of weak learnability. Mach Learn, 5(2):197-227.

[26]Schelter S, Lange D, Schmidt P, et al., 2018a. Automating large-scale data quality verification. Proc VLDB Endow, 11(12):1781-1794.

[27]Schelter S, Biessmann F, Januschowski T, et al., 2018b. On challenges in machine learning model management. IEEE Data Eng Bull, 41(4):5-15.

[28]Sculley D, Holt G, Golovin D, et al., 2015. Hidden technical debt in machine learning systems. Proc 28^th Int Conf on Neural Information Processing Systems, p.2503-2511.

[29]Sun NY, Yang XF, Liu YF, 2020. TableQA: a large-scale Chinese text-to-SQL dataset for table-aware SQL generation. https://arxiv.org/abs/2006.06434v1

[30]Tableau, 2020a. Ask Data. https://www.tableau.com/products/new-features/ask-data/ [Accessed on Aug. 10, 2020].

[31]Tableau, 2020b. Tableau. https://www.tableau.com/ [Accessed on Aug. 10, 2020].

[32]Task Virtual, 2020. TaskVirtual. https://taskvirtual.com/ [Accessed on Aug. 10, 2020].

[33]Thrun S, 1998. Lifelong learning algorithms. In: Thrun S, Pratt L (Eds.), Learning to Learn. Kluwer Academic Publishers, Norwell, USA, p.181-209.

[34]Vogelsang A, Borg M, 2019. Requirements engineering for machine learning: perspectives from data scientists. Proc 27^th Int Requirements Engineering Conf Workshops, p.245-251.

[35]Voigt P, von dem Bussche A, 2017. The EU General Data Protection Regulation (GDPR): a Practical Guide. Springer, Cham, Germany.

[36]Wikipedia, 2021. Virtual Assistant. https://en.wikipedia.org/wiki/Virtual_assistant [Accessed on Aug. 10, 2020].

[37]Yao ZY, Su Y, Sun H, et al., 2019. Model-based interactive semantic parsing: a unified framework and a text-to-SQL case study. Proc Conf on Empirical Methods in Natural Language Processing and the 9^th Int Joint Conf on Natural Language Processing, p.5447-5458.

[38]Young SWH, 2014. Improving library user experience with A/B testing: principles and process. Weave J Libr User Exper.

[39]Zhang JM, Harman M, Ma L, et al., 2022. Machine learning testing: survey, landscapes and horizons. IEEE Trans Softw Eng, 48(1):1-36.

[40]Zhang TY, Gao CY, Ma L, et al., 2019. An empirical study of common challenges in developing deep learning applications. Proc 30^th Int Symp on Software Reliability Engineering, p.104-115.

[41]Zhong V, Xiong CM, Socher R, 2017. Seq2SQL: generating structured queries from natural language using reinforcement learning. https://arxiv.org/abs/1709.00103v5