Skip to main content
Kent Academic Repository

New Tricks to Old Codes: Can AI Chatbots Replace Static Code Analysis Tools?

Ozturk, Omer Said, Ekmekcioglu, Emre, Cetin, Orcun, Arief, Budi, Hernandez-Castro, Julio C. (2023) New Tricks to Old Codes: Can AI Chatbots Replace Static Code Analysis Tools? In: EICC '23: Proceedings of the 2023 European Interdisciplinary Cybersecurity Conference. . pp. 13-18. ACM ISBN 978-1-4503-9829-9. (doi:10.1145/3590777.3590780) (KAR id:102124)

PDF Author's Accepted Manuscript
Language: English

Restricted to Repository staff only
Contact us about this Publication
[thumbnail of EICC2023-Paper3196-ChatGPT-PHP-AAM.pdf]
PDF Publisher pdf
Language: English


Download this file
(PDF/587kB)
[thumbnail of 3590777.3590780.pdf]
Preview
Request a format suitable for use with assistive technology e.g. a screenreader
Official URL:
https://doi.org/10.1145/3590777.3590780

Abstract

The prevalence and significance of web services in our daily lives make it imperative to ensure that they are – as much as possible – free from vulnerabilities. However, developing a complex piece of software free from any security vulnerabilities is hard, if not impossible. One way to progress towards achieving this holy grail is by using static code analysis tools to root out any common or known vulnerabilities that may accidentally be introduced during the development process. Static code analysis tools have significantly contributed to addressing the problem above, but are imperfect. It is conceivable that static code analysis can be improved by using AI-powered tools, which have recently increased in popularity. However, there is still very little work in analysing both types of tools’ effectiveness, and this is a research gap that our paper aims to fill. We carried out a study involving 11 static code analysers, and one AI-powered chatbot named ChatGPT, to assess their effectiveness in detecting 92 vulnerabilities representing the top 10 known vulnerability categories in web applications, as classified by OWASP. We particularly focused on PHP vulnerabilities since it is one of the most widely used languages in web applications. However, it has few security mechanisms to help its software developers. We found that the success rate of ChatGPT in terms of finding security vulnerabilities in PHP is around 62-68%. At the same time, the best traditional static code analyser tested has a success rate of 32%. Even combining several traditional static code analysers (with the best features on certain aspects of detection) would only achieve a rate of 53%, which is still significantly lower than ChatGPT’s success rate. Nonetheless, ChatGPT has a very high false positive rate of 91%. In comparison, the worst false positive rate of any traditional static code analyser is 82%. These findings highlight the promising potential of ChatGPT for improving the static code analysis process but reveal certain caveats (especially regarding accuracy) in its current state. Our findings suggest that one interesting possibility to explore in future works would be to pick the best of both worlds by combining traditional static code analysers with ChatGPT to find security vulnerabilities more effectively.

Item Type: Conference or workshop item (Proceeding)
DOI/Identification number: 10.1145/3590777.3590780
Projects: HEROES
Uncontrolled keywords: ChatGPT, AI, Static code analysis, PHP vulnerabilities, Tools evaluation, Vulnerability detection, AI in cyber security
Subjects: Q Science > QA Mathematics (inc Computing science)
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
University-wide institutes > Institute of Cyber Security for Society
Funders: European Commission (https://ror.org/00k4n6c32)
Depositing User: Budi Arief
Date Deposited: 18 Jul 2023 16:12 UTC
Last Modified: 04 Mar 2024 17:54 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/102124 (The current URI for this page, for reference purposes)

University of Kent Author Information

  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.