The study of the operational practices of national CSIRTS regarding the Use of free tools and public data in supporting computer security incident response

Computer Security Incident Response Teams (CSIRTs) have been established at national and organisational levels to respond and coordinate responses to computer security incidents. It is known that many CSIRTs, including national CSIRTs, routinely use different types of tools and data, including free tools, open-source tools, and public data, in their daily work to support incident responses. However, a lack of public information and systematic discussions was observed regarding how national CSIRTs use and perceive free tools and public data in their operational practices.

To achieve a more comprehensive and systematic understanding of how national CSIRTs use and perceive free tools and public data in the operational practices, a systematic literature review (SLR) of the research literature and websites of national CSIRTs and cross-CSIRT organisations was conducted in this research. A primary finding from the SLR is that most discussions concerning free tools and public data used in national CSIRTs' operations are largely incomplete, ad hoc, and fragmented. This includes a lack of discussions on how the staff of national CSIRTs perceive the usefulness of free tools and public data to facilitate incident responses. Such gaps can prevent us from understanding how national CSIRTs can benefit from free tools and public data and how other organisations, individuals and researchers can help by providing such tools and data to improve national CSIRTs' operations. The findings from the SLR call for more empirical research on how national CSIRTs use and perceive free tools and public data. This should also include how such data and tools can be leveraged to support incident responses at national CSIRTs' operations.

Hence, a survey and twelve follow-up semi-structured interviews with staff members of thirteen national CSIRTs worldwide were undertaken in this research to gain insights into how free tools and public data are used and perceived in national CSIRTs. The study was conducted in two phases: first, with staff members of the Malaysia Computer Emergency Response Team (MyCERT) to get some initial results, and then with twelve other national CSIRTs to enlarge the results from the first phase. Results from the survey and the semi-structured interviews led to three main findings: 1) confirmation from the participants regarding the active use of free tools, public data, and open-source intelligence (OSINT) tools in national CSIRTs, 2) the perceived usefulness of free tools and public data to support incident responses in national CSIRTs, and 3) a lack of systematic procedures in guiding the use of free tools and public data across the participating national CSIRTs (for example, one aspect is on how such tools and data should be evaluated for quality and usability). The finding on the lack of systematic procedures for evaluating free tools and public data calls for further research and development to better understand current tools and data evaluation practices in national CSIRTs. Such understanding shall inform researchers on developing systematic procedures for evaluating free tools and public data in national CSIRTs' operations.

An empirical study using several focus group discussions was conducted to understand better current tools and data evaluation practices in real-world operations of national CSIRTs. The findings from the focus group study confirmed that the evaluation practices are ad hoc and informal. Systematic procedures that leverage industry standards, such as criteria for evaluating free tools and public data, are unavailable in the participating national CSIRTs' operations. This finding informed the construction of candidate criteria for evaluating free tools and public data from the focus group discussions. Nevertheless, the validity of the candidate criteria for usefulness, deployment and applicability is uncertain. This calls for studies to validate the candidate criteria before implementation in real-world operational practices of national CSIRTs for evaluating free tools and public data.

A validation study using semi-structured interviews was conducted to gain insights into how staff members of national CSIRTs perceive the usefulness and deployment of the candidate criteria in national CSIRTs. This is followed by a more objective validation by applying the criteria to evaluate two candidate tools and one sample data source for applicability in practice. This was performed by converting each criterion into one or more relevant metrics, such as ``measuring the time taken by a tool to produce results''. Significantly, results from both validation approaches were consistent, leading to the following findings: 1) the candidate criteria were perceived as practically useful for evaluating free tools and public data in the operations of national CSIRTs; 2) the candidate criteria were perceived as ready for deployment in national CSIRTs and 3) the criteria is applicable in practice to evaluate free tools and public data. It is envisaged that these criteria would help national CSIRTs and the broader security operations to select usable and good quality free tools and public data available on the Internet to support incident response. Subsequently, enhancing the current practices in evaluating free tools and public data in national CSIRTs.