Professional Auditing Research

Professional Auditing Research

Classification and Auditing of Large Language Models (LLMs)

Authors
Morteza Asadi 1
Mina Farnoud Ahmadi 2
1 Master of Accounting, Management Faculty, University of Tehran, Tehran, Iran.
2 Ph.D Student in Accounting, Islamic Azad University. Tabriz. Iran
10.22034/jpar.2026.2086351.1515
Abstract
Large language models represent a major advance in artificial intelligence research. These models are based on complex neural networks trained on large datasets and can provide unique capabilities in various domains. The application of large language models in the audit field has also created challenges in performing repetitive tasks, evaluating internal controls, providing procedures, and planning audits. Studies have also defined AI auditing as a mechanism that ensures that AI systems are ethically, legally, and technically sound. However, existing auditing practices have failed to address the challenges posed by large language models that exhibit emerging capabilities. The present study, by reviewing the existing literature, while defining and classifying these models, has raised the opportunities and challenges of using large language models in the field of auditing and has presented a three-level framework for their auditing. The results of this study provide researchers with a structured basis for classifying large language models. The study also shows that audits, when conducted in a structured and coordinated manner at all three levels, can be a practical mechanism for managing some of the ethical and social risks posed by large language models.

Highlights

علوی، سید کمال؛ نعمتی، علی و دارابی، رویا. (1404). الگوی بهینه و کاربردی فناوری اطلاعات در حسابرسی با توجه به آزمون های محتوا، کنترل و ریسک های حسابرسی. پژوهش‌های حسابرسی حرفه‌ای، 5(20), 30-57.‎

نقدی، سجاد؛ احمدیان، وحید، فضل زاده، علیرضا و ولی پور، احمد.(1403) .طراحی مدل استقرار اثربخش حسابرسی فناوری اطلاعات در سازمان تأمین اجتماعی. پژوهش‌های حسابرسی حرفه‌ای، 5(18), 59-34.

 

Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., ... & Venkatasubramanian, S. (2018). Auditing black-box models for indirect influence. Knowledge and Information Systems, 54(1), 95-122. DOI: 10.1007/s10115-017-1116-3.

AICPA. (1993). Code of Professional Conduct. New York: AICPA. https://aicpa.org/research/standards/code-of-professional-conduct.html.

Alavi, S. K., Nemati, A., & Darabi, R. (2025). An Optimal and Practical Model of Information Technology in Auditing Considering Substantive and Control Testing, and Audit Risks. (In Persian). DOI: 10.22034/jpar.2024.2031862.1327.

Avin, S., Belfield, H., Brundage, M., Krueger, G., Wang, J., Weller, A., ... & Zilberman, N. (2021). Filling gaps in trustworthy development of AI. Science, 374(6573), 1327-1329. DOI: 10.1126/science.abj0689.

Bandy, J. (2021). Problematic machine behavior: A systematic literature review of algorithm audits. Proceedings of the acm on human-computer interaction, 5(CSCW1), 1-34. DOI: 10.1145/3449144.

Bauer, J. (2016). The necessity of auditing artificial intelligence. SSRN, 577, 1-16. DOI: 10.2139/ssrn.2740657.

Berghout, E., Fijneman, R., Hendriks, L., de Boer, M., & Butijn, B. J. (2023). Advanced Digital auditing: Theory and practice of auditing complex information systems and technologies (p. 256). Springer Nature. DOI: 10.1007/978-3-031-34918-7.

Boskou, G., Kirkos, E., Chatzipetrou, E., Tiakas, E., & Spathis, C. (2024). Exploring the boundaries of financial statement fraud detection with large language models. Available at SSRN 4842962. https://ssrn.com/abstract=4842962.

Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., ... & Anderljung, M. (2020). Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv preprint arXiv:2004.07213. https://arxiv.org/abs/2004.07213.

Buolamwini, J., & Gebru, T. (2018, January). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency (pp. 77-91). PMLR. https://proceedings.mlr.press/v81/buolamwini18a.html.

Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186. 10.1126/science.aal4230.

Chopra, A. K., & Singh, M. P. (2018, December). Sociotechnical systems and ethics in the large. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 48-53). https://dl.acm.org/doi/10.1145/3278721.3278727.

Contractor, D., McDuff, D., Haines, J. K., Lee, J., Hines, C., Hecht, B., ... & Li, H. (2022, June). Behavioral use licensing for responsible AI. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 778-788). DOI: 10.1145/3531146.3533084.

Deloitte. (2024). How we harness Generative AI for a smarter, digital audit. https://www2.deloitte.com/us/en/pages/strategy/articles/harnessing-generative-ai-for-a-smarter-digital-audit.html.

Dong, M. M., Stratopoulos, T. C., & Wang, V. X. (2024). A scoping review of ChatGPT research in accounting and finance. International Journal of Accounting Information Systems, 55, 100715. DOI: 10.1016/j.isa.2024.100715.

Elad, F., & Mugwira, T. (2025). Classifying and Evaluating Large Language Models in External Audits. Available at SSRN 5325170. https://ssrn.com/abstract=5325170.

Emett, S., Eulerich, M., Lipinski, E., Prien, N., & Wood, D. A. (2024). Leveraging ChatGPT for Enhancing the Internal Audit Process—A Real-world Example from Uniper, a Large Multinational Company. Accounting Horizons, 1-11. https://doi.org/10.2308/HORIZONS-2023-094.

Engler, A. (2023). Early thoughts on regulating generative AI like ChatGPT. https://www.brookings.edu/articles/early-thoughts-on-regulating-generative-ai-like-chatgpt/.

Engler, A. C. (2021). Independent auditors are struggling to hold AI companies accountable. Fast Company. https://www.fastcompany.com/90695092/independent-auditors-are-struggling-to-hold-ai-companies-accountable.

Eulerich, M., & Wood, D. A. (2025). A Demonstration of How ChatGPT and Generative AI Can be Used in the Internal Auditing Process. Journal of Emerging Technologies in Accounting, 1-31. https://doi.org/10.2308/JETA-2023-030.

Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024). Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. Review of Accounting Studies, 29(3), 2318-2349. DOI: 10.2308/RET-2349-2349.

Evans, O., Cotton-Barratt, O., Finnveden, L., Bales, A., Balwit, A., Wills, P., ... & Saunders, W. (2021). Truthful AI: Developing and governing AI that does not lie. arXiv preprint arXiv:2110.06674. https://arxiv.org/abs/2110.06674.

Falco, G., Shneiderman, B., Badger, J., Carrier, R., Dahbura, A., Danks, D., ... & Yeong, Z. K. (2021). Governing AI safety through independent audits. Nature Machine Intelligence, 3(7), 566-571. DOI: 10.1038/s42256-021-00383-x.

Flint, D. (1988). Philosophy and principles of auditing: an introduction. (No Title).

Floridi, L., Holweg, M., Taddeo, M., Amaya, J., Mökander, J., & Wen, Y. (2022). CapAI-A procedure for conducting conformity assessment of AI systems in line with the EU artificial intelligence act. Available at SSRN 4064091. https://ssrn.com/abstract=4064091.

Föhr, T. L., Schreyer, M., Juppe, T. A., & Marten, K. U. (2023). Assuring sustainable futures: Auditing sustainability reports using AI foundation models. Available at SSRN 4502549. https://ssrn.com/abstract=4502549.

Fotoh, L. E., & Mugwira, T. (2025). Exploring Large Language Models in external audits: Implications and ethical considerations. International Journal of Accounting Information Systems, 56, 100748. DOI: 10.1016/j.accinfosec.2025.100748.

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92. DOI: 10.1145/3442188.

Goel, K., Rajani, N., Vig, J., Tan, S., Wu, J., Zheng, S., ... & Ré, C. (2021). Robustness gym: Unifying the NLP evaluation landscape. arXiv preprint arXiv:2101.04840. https://arxiv.org/abs/2101.04840.

Green, N., Procope, C., Cheema, A., & Adediji, A. (2022). System cards, a new resource for understanding how AI systems work. Meta AI. https://ai.meta.com/blog/system-cards-understanding-ai-systems/.

Green, R. M., & Donovan, A. (2009). The methods of business ethics.

Gu, H., Schreyer, M., Moffitt, K., & Vasarhelyi, M. (2024). Artificial intelligence co-piloted auditing. International Journal of Accounting Information Systems, 54, 100698. https://doi.org/10.1016/j.accinf.2024.100698.

Gu, H., Schreyer, M., Moffitt, K., & Vasarhelyi, M. (2024). Artificial intelligence co-piloted auditing. International Journal of Accounting Information Systems, 54, 100698.

Hodges, C. (2015). Ethics in business practice and regulation. Law and Corporate Behaviour: Integrating Theories of Regulation, Enforcement, Compliance and Ethics, 1-21. https://www.elgaronline.com/abstract/9781784710690.00010.xml.

IESBA. 2023. Handbook of the international code of ethics for professional accountants. New York: International Federation of Accountants, Professional Code. https://www.ifac.org/system/files/publications/files/IESBA-2023-Handbook-Code-of-Ethics.pdf.

Iliescu, F. M. (2010). Auditing IT governance. Informatica Economica, 14(1), 93. http://www.revistaie.ase.ro/content/53/09%20-%20Iliescu.pdf.

Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023, July). A watermark for large language models. In International Conference on Machine Learning (pp. 17061-17084). PMLR. https://proceedings.mlr.press/v202/kirchenbauer23a.html

LaBrie, R. C., & Steinke, G. (2019). Towards a framework for ethical audits of AI algorithms. https://www.researchgate.net/publication/337828146_Towards_a_framework_for_ethical_audits_of_AI_algorithms.

Laux, J., Wachter, S., & Mittelstadt, B. (2021). Taming the few: Platform regulation, independent audits, and the risks of capture created by the DMA and DSA. Computer law & Security review, 43, 105613. https://doi.org/10.1016/j.clsr.2021.105613.

Li, H., & Vasarhelyi, M. A. (2024). Applying large language models in accounting: A comparative analysis of different methodologies and off-the-shelf examples. Journal of Emerging Technologies in Accounting, 21(2), 133-152. https://doi.org/10.2308/JETA-2023-043.

Li, H., Freitas, M. M. D., Lee, H., & Vasarhelyi, M. (2024). Enhancing continuous auditing with large language models: AI-assisted real-time accounting information cross-verification. Available at SSRN 4692960. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4692960.

Marda, V., & Narayan, S. (2021). On the importance of ethnographic methods in AI research. Nature Machine Intelligence, 3(3), 187-189. https://doi.org/10.1038/s42256-021-00316-8.

Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends® in Human–Computer Interaction, 14(4), 272-344. https://doi.org/10.1561/1100000077.

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019, January). Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency (pp. 220-229). https://doi.org/10.1145/3287560.3287596.

Mökander, J., Axente, M., Casolari, F., & Floridi, L. (2022). Conformity assessments and post-market monitoring: a guide to the role of auditing in the proposed European AI regulation. Minds and Machines, 32(2), 241-268. https://doi.org/10.1007/s11023-022-09598-y.

Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024). Auditing large language models: a three-layered approach. AI and Ethics, 4(4), 1085-1115. https://doi.org/10.1007/s43681-024-00393-y.

Mökander, J., Sheth, M., Gersbro-Sundler, M., Blomgren, P., & Floridi, L. (2022). Challenges and best practices in corporate AI governance: Lessons from the biopharmaceutical industry. Frontiers in Computer Science, 4, 1068361. https://doi.org/10.3389/fcomp.2022.1068361.

Munoko, I., Brown-Liburd, H. L., & Vasarhelyi, M. (2020). The ethical implications of using artificial intelligence in auditing. Journal of business ethics, 209-234. https://doi.org/10.1007/s10551-019-04407-1.

Naghdi, S., ّFazlzadeh, A., & Valipour, A. (2025). Designing an Effective Deployment Model of Information Technology Audit in the Social Security Organization. Professional Auditing Research, 5(18), 34-59. (In Persian). DOI:10.22034/jpar.2024.2015837.1243.

O'Leary, D. E. (2023). Enterprise large language models: Knowledge characteristics, risks, and organizational activities. Intelligent systems in accounting, finance and management, 30(3), 113-119. https://doi.org/10.1002/isaf.1542.

Otero, A. R., & Agu, M. (2025). Benefits and Drawbacks of Incorporating ChatGPT in Financial Audits. Current Issues in Auditing, 1-8.

Papachristou, G., & Bekiaris, M. (2020). Ethics in auditing. The Routledge Handbook of Accounting Ethics, 169-185. https://doi.org/10.4324/9780429200606-15.

Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021). Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11). https://doi.org/10.1016/j.patter.2021.100336.

Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022, July). Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (pp. 557-571). https://doi.org/10.1145/3514094.3534168.

Ross, M., & Zhang, J. (2025). ChatGPT is Ready for the Profession—But is the Profession Ready for ChatGPT? Accounting Horizons, 39(2), 185-203. https://doi.org/10.2308/HORIZONS-2023-057.

Sandvig, C., Hamilton, K., Karahalios, K., & Langbort, C. (2014). Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry, 22(2014), 4349-4357. https://pdfs.semanticscholar.org/1f3d/c9a2dba0d5b3cef9a6a8c4d6c6d2d1e8d3e5.pdf.

Senft, S., & Gallegos, F. (2008). Information technology control and audit. Auerbach publications. https://www.taylorfrancis.com/books/mono/10.1201/9781420065506/information-technology-control-audit-sandra-senft-frederick-gallegos.

Stodt, J., & Reich, C. (2021). Machine learning development audit framework: assessment and inspection of risk and quality of data, model and development process. International Journal of Computer and Information Engineering, 15(3), 187-193. http://www.waset.org/publications/10012046

Street, D., & Wilck, J. (2023). 'Let’s Have a Chat': Principles for the Effective Application of ChatGPT and Large Language Models in the Practice of Forensic Accounting. Journal of Forensic and Investigative Accounting, July to December. https://doi.org/10.2308/JFIA-2023-011.

Street, D., Wilck, J., & Chism, Z. (2023). Six Principles for the Effective Use of Artificial Intelligence Large Language Models: How to Leverage ChatGPT, Bard, and Bing Chat in Accounting Work. The CPA Journal, 93(11-12), 50-57. https://www.cpajournal.com/2023/11/13/six-principles-for-the-effective-use-of-artificial-intelligence-large-language-models/.

Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the capabilities, limitations, and societal impact of large language models. https://arxiv.org/abs/2102.02503.

Vasarhelyi, M. A., Moffitt, K. C., Stewart, T., & Sunderland, D. (2023). Large language models: An emerging technology in accounting. Journal of emerging technologies in accounting, 20(2), 1-10. https://doi.org/10.2308/JETA-2023-010.

Wang, Z.J., Choi, D., Xu, S., Yang, D. (2021). Putting Humans in the Natural Language Processing Loop: A Survey. In: Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pp. 47–52. https://aclanthology.org/2021.hcinlp-1.7/.

Wei, T., Wu, H., & Chu, G. (2023). Is ChatGPT competent? Heterogeneity in the cognitive schemas of financial auditors and robots. International Review of Economics & Finance, 88, 1389-1396. https://doi.org/10.1016/j.iref.2023.03.018.

Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P. S., Mellor, J., ... & Gabriel, I. (2022, June). Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency (pp. 214-229). https://doi.org/10.1145/3531146.3533088.

Zhao, J., & Wang, X. (2024). Unleashing efficiency and insights: Exploring the potential applications and challenges of ChatGPT in accounting. Journal of Corporate Accounting & Finance, 35(1), 269-276. https://doi.org/10.1002/jcaf.22688.

Keywords
Large language models
Governance audit
Model audit
Application audit
Subjects