Skip to main content
Kent Academic Repository

Latch: Enabling large-scale automated testing on constrained systems

Lauwaerts, Tom, Marr, Stefan, Scholliers, Christophe (2024) Latch: Enabling large-scale automated testing on constrained systems. Science of Computer Programming, 238 . Article Number 103157. ISSN 0167-6423. E-ISSN 1872-7964. (doi:10.1016/j.scico.2024.103157) (KAR id:106218)

PDF Author's Accepted Manuscript
Language: English


Download this file
(PDF/955kB)
[thumbnail of paper.pdf]
Preview
Request a format suitable for use with assistive technology e.g. a screenreader
PDF Publisher pdf
Language: English

Restricted to Repository staff only
Contact us about this Publication
[thumbnail of 1-s2.0-S0167642324000807-main.pdf]
Official URL:
https://doi.org/10.1016/j.scico.2024.103157

Abstract

Testing is an essential part of the software development cycle. Unfortunately, testing on constrained devices is currently very challenging. First, the limited memory of constrained devices severely restricts the size of test suites. Second, the limited processing power causes test suites to execute slowly, preventing a fast feedback loop. Third, when the constrained device becomes unresponsive, it is impossible to distinguish between the test failing or taking very long, forcing the developer to work with timeouts. Unfortunately, timeouts can cause tests to be flaky, i.e., have unpredictable outcomes independent of code changes. Given these problems, most IoT developers rely on laborious manual testing.

In this paper, we propose the novel testing framework Latch (Large-scale Automated Testing on Constrained Hardware) to overcome the three main challenges of running large test suites on constrained hardware, as well as automate manual testing scenarios through a novel testing methodology based on debugger-like operations—we call this new testing approach managed testing.

The core idea of Latch is to enable testing on constrained devices without those devices maintaining the whole test suite in memory. Therefore, programmers script and run tests on a workstation which then step-wise instructs the constrained device to execute each test, thereby overcoming the memory constraints. Our testing framework further allows developers to mark tests as depending on other tests. This way, Latch can skip tests that depend on previously failing tests resulting in a faster feedback loop. Finally, Latch addresses the issue of timeouts and flaky tests by including an analysis mode that provides feedback on timeouts and the flakiness of tests.

To illustrate the expressiveness of Latch, we present testing scenarios representing unit testing, integration testing, and end-to-end testing. We evaluate the performance of Latch by testing a virtual machine against the WebAssembly specification, with a large test suite consisting of 10,213 tests running on an ESP32 microcontroller. Our experience shows that the testing framework is expressive, reliable and reasonably fast, making it suitable to run large test suites on constrained devices. Furthermore, the debugger-like operations enable to closely mimic manual testing.

Item Type: Article
DOI/Identification number: 10.1016/j.scico.2024.103157
Additional information: For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.
Uncontrolled keywords: automated testing, embedded devices, flaky tests
Subjects: Q Science > QA Mathematics (inc Computing science)
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Funders: Research Foundation - Flanders (https://ror.org/03qtxy027)
Engineering and Physical Sciences Research Council (https://ror.org/0439y7842)
Royal Society (https://ror.org/03wnrjx87)
Depositing User: Stefan Marr
Date Deposited: 10 Jun 2024 14:24 UTC
Last Modified: 26 Jun 2024 11:13 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/106218 (The current URI for this page, for reference purposes)

University of Kent Author Information

  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.