Analysis of expertise in intelligence analysis for human-centered AI

Recently published paper 

S. Hepenstal, L. Zhang and B. L. William Wong, "An analysis of expertise in intelligence analysis to support the design of Human-Centered Artificial Intelligence," 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2021, pp. 107-112, doi: 10.1109/SMC52423.2021.9659095.

ttps://doi.org/10.1109/SMC52423.2021.9659095 Abstract Intelligence analysis involves unpredictable processes and decision making about complex domains where analysts rely upon expertise. Artificial Intelligence (AI) systems could support analysts as they perform analysis tasks, to enhance their expertise. However, systems must also be cognisant about how expertise is gained and designed so that this is not impinged. In this paper, we describe the results of Cognitive Task Analysis interviews with 6 experienced intelligence analysts. We capture themes, in terms of their decision making paths during an analysis task, and highlight how each theme is both influenced by expertise and an influence upon expertise. We also identify important interdependencies between themes. We propose that our findings can be used to help design Human-Centered AI (HCAI) systems for supporting intelligence analysts. To read more go to: https://doi.org/10.1109/SMC52423.2021.9659095 References 1.   G. A. Klein, R. Calderwood and D. MacGregor, "Critical decision method for eliciting knowledge", IEEE Transactions on Systems Man and Cybernetics, vol. 19, no. 3, pp. 462-472, 1989. https://doi.org/10.1109/21.31053   2.   B. L. W. Wong, Critical Decision Method Data Analysis, Lawrence Erlbaun Associates, pp. 327-346, 1993.     3.   M. Gerber, B. W. Wong and N. Kodagoda, "How analysts think: Intuition leap of faith and insight", Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 60, no. 1, pp. 173-177, 2016, [online] Available: https://doi.org/10.1177/1541931213601039. https://doi.org/10.1177/1541931213601039     4.   "National intelligence machinery", [online] Available: https://www.gov.uk/government/publications/national-intelligence-machinery.     5.   B. W. Wong and N. Kodagoda, "How analysts think: Inference making strategies", Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 59, no. 1, pp. 269-273, 2015, [online] Available: https://doi.org/10.1177/1541931215591055. https://doi.org/10.1177/1541931215591055     6.   G. Tecuci, D. Schum, M. Boicu, D. Marcu and B. Hamilton, "Intelligence analysis as agent-assisted discovery of evidence hypotheses and arguments" in Advances in Intelligent Decision Technologies, Berlin, Heidelberg:Springer Berlin Heidelberg, pp. 1-10, 2010. https://doi.org/10.1007/978-3-642-14616-9_1     7.   J. D. Corkill, T. K. Cunow, E. Ashton and A. East, "Attributes of an analyst: What we can learn from the intelligence analysts job description", 8th Australian Security and Intelligence Conference, 2015.     8.   N. Quarmby and L. J. Young, "Managing intelligence: The art of influence" in , Sydney:The Federation Press, 2010.     9.   D. M. Allen, "Building a better strategic analyst: a critical review of the U.S. army's all source analyst training program" in , Fort Leavenworth:United States Army Command and General Staff College, pp. 59, 2008.     10.   G. Klein, A recognition-primed decision (RPD) model of rapid decision making, Abex Publishing, pp. 138-147, 1993.   11.   J. Rasmussen, "Taxonomy for cognitive work analysis", [online] Available: .     12.   D. Leslie, "Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of ai systems in the public sector", 2019, [online] Available: https://doi.org/10.5281/zenodo.3240529. https://doi.org/10.2139/ssrn.3403301   13.   S. Hepenstal, N. Kodagoda, L. Zhang, P. Paudyal and B. L. W. Wong, "Algorithmic transparency of conversational agents", Joint Proceedings of the ACM IUI 2019 Workshops co-located with the 24th ACM Conference on Intelligent User Interfaces (ACM IUI 2019), vol. 2327, March 20, 2019, [online] Available: http://ceur-ws.org/Vol-2327/IUI19WS-IUIATEC-1.pdf.     14.   W. Xu, "Toward human-centered ai: A perspective from human-computer interaction", Interactions, vol. 26, no. 4, pp. 42-46, Jun. 2019, [online] Available: https://doi.org/10.1145/3328485. https://doi.org/10.1145/3328485   15.   B. L. W. Wong and A. Blandford, "Analysing ambulance dispatcher decision making: trialing emergent themes analysis" in , Canberra, Australia:Ergonomics Society of Australia, 2002.   16.   B. L. W. Wong and A. Blandford, "Describing situation awareness at an emergency medical dispatch centre", Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 48, pp. 285-289, 2004. https://doi.org/10.1177/154193120404800302     17.   M. R. Endsley, "Toward a theory of situation awareness in dynamic systems", Human Factors, vol. 37, no. 1, pp. 32-64, 1995, [online] Available: https://doi.org/10.1518/001872095779049543. https://doi.org/10.1518/001872095779049543   18.   G. Klein, Seeing what others don't. The Remarkable Ways We Gain Insights, Nicholas Brealey Publishing, 2017.   19.   F. Qiao, P. Li, X. Zhang, Z. Ding, J. Cheng and H. Wang, "Predicting social unrest events with hidden markov models using gdelt", Discrete Dynamics in Nature and Society, vol. 2017, pp. 1-13, 2017. https://doi.org/10.1155/2017/8180272   References & Cited By Select All 1. G. A. Klein, R. Calderwood and D. MacGregor, "Critical decision method for eliciting knowledge", IEEE Transactions on Systems Man and Cybernetics, vol. 19, no. 3, pp. 462-472, 1989. Show in Context View Article   Google Scholar  2. B. L. W. Wong, Critical Decision Method Data Analysis, Lawrence Erlbaun Associates, pp. 327-346, 1993. Show in Context Google Scholar  3. M. Gerber, B. W. Wong and N. Kodagoda, "How analysts think: Intuition leap of faith and insight", Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 60, no. 1, pp. 173-177, 2016, [online] Available: https://doi.org/10.1177/1541931213601039. Show in Context CrossRef  Google Scholar  4. "National intelligence machinery", [online] Available: https://www.gov.uk/government/publications/national-intelligence-machinery. Show in Context Google Scholar  5. B. W. Wong and N. Kodagoda, "How analysts think: Inference making strategies", Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 59, no. 1, pp. 269-273, 2015, [online] Available: https://doi.org/10.1177/1541931215591055. Show in Context CrossRef  Google Scholar  6. G. Tecuci, D. Schum, M. Boicu, D. Marcu and B. Hamilton, "Intelligence analysis as agent-assisted discovery of evidence hypotheses and arguments" in Advances in Intelligent Decision Technologies, Berlin, Heidelberg:Springer Berlin Heidelberg, pp. 1-10, 2010. Show in Context CrossRef  Google Scholar  7. J. D. Corkill, T. K. Cunow, E. Ashton and A. East, "Attributes of an analyst: What we can learn from the intelligence analysts job description", 8th Australian Security and Intelligence Conference, 2015. Show in Context Google Scholar  8. N. Quarmby and L. J. Young, "Managing intelligence: The art of influence" in , Sydney:The Federation Press, 2010. Show in Context Google Scholar  9. D. M. Allen, "Building a better strategic analyst: a critical review of the U.S. army’s all source analyst training program" in , Fort Leavenworth:United States Army Command and General Staff College, pp. 59, 2008. Show in Context Google Scholar  10. G. Klein, A recognition-primed decision (RPD) model of rapid decision making, Abex Publishing, pp. 138-147, 1993. Show in Context Google Scholar  11. J. Rasmussen, "Taxonomy for cognitive work analysis", [online] Available: . Show in Context View Article   Google Scholar  12. D. Leslie, "Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of ai systems in the public sector", 2019, [online] Available: https://doi.org/10.5281/zenodo.3240529. Show in Context CrossRef  Google Scholar  13. S. Hepenstal, N. Kodagoda, L. Zhang, P. Paudyal and B. L. W. Wong, "Algorithmic transparency of conversational agents", Joint Proceedings of the ACM IUI 2019 Workshops co-located with the 24th ACM Conference on Intelligent User Interfaces (ACM IUI 2019), vol. 2327, March 20, 2019, [online] Available: http://ceur-ws.org/Vol-2327/IUI19WS-IUIATEC-1.pdf. Show in Context Google Scholar  14. W. Xu, "Toward human-centered ai: A perspective from human-computer interaction", Interactions, vol. 26, no. 4, pp. 42-46, Jun. 2019, [online] Available: https://doi.org/10.1145/3328485. Show in Context Google Scholar  15. B. L. W. Wong and A. Blandford, "Analysing ambulance dispatcher decision making: trialing emergent themes analysis" in , Canberra, Australia:Ergonomics Society of Australia, 2002. Show in Context Google Scholar  16. B. L. W. Wong and A. Blandford, "Describing situation awareness at an emergency medical dispatch centre", Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 48, pp. 285-289, 2004. Show in Context CrossRef  Google Scholar  17. M. R. Endsley, "Toward a theory of situation awareness in dynamic systems", Human Factors, vol. 37, no. 1, pp. 32-64, 1995, [online] Available: https://doi.org/10.1518/001872095779049543. Show in Context CrossRef  Google Scholar  18. G. Klein, Seeing what others don’t. The Remarkable Ways We Gain Insights, Nicholas Brealey Publishing, 2017. Show in Context Google Scholar  19. F. Qiao, P. Li, X. Zhang, Z. Ding, J. Cheng and H. Wang, "Predicting social unrest events with hidden markov models using gdelt", Discrete Dynamics in Nature and Society, vol. 2017, pp. 1-13, 2017. CrossRef  Google Scholar