Lauwaerts, Tom, Marr, Stefan, Scholliers, Christophe (2024) Latch: Enabling large-scale automated testing on constrained systems. Science of Computer Programming, 238 . Article Number 103157. ISSN 0167-6423. E-ISSN 1872-7964. (doi:10.1016/j.scico.2024.103157) (KAR id:106218)
PDF
Author's Accepted Manuscript
Language: English
This work is licensed under a Creative Commons Attribution 4.0 International License.
|
|
Download this file (PDF/955kB) |
Preview |
Request a format suitable for use with assistive technology e.g. a screenreader | |
PDF
Publisher pdf
Language: English Restricted to Repository staff only |
|
Contact us about this Publication
|
|
Official URL: https://doi.org/10.1016/j.scico.2024.103157 |
Abstract
Testing is an essential part of the software development cycle. Unfortunately, testing on constrained devices is currently very challenging. First, the limited memory of constrained devices severely restricts the size of test suites. Second, the limited processing power causes test suites to execute slowly, preventing a fast feedback loop. Third, when the constrained device becomes unresponsive, it is impossible to distinguish between the test failing or taking very long, forcing the developer to work with timeouts. Unfortunately, timeouts can cause tests to be flaky, i.e., have unpredictable outcomes independent of code changes. Given these problems, most IoT developers rely on laborious manual testing.
In this paper, we propose the novel testing framework Latch (Large-scale Automated Testing on Constrained Hardware) to overcome the three main challenges of running large test suites on constrained hardware, as well as automate manual testing scenarios through a novel testing methodology based on debugger-like operations—we call this new testing approach managed testing.
The core idea of Latch is to enable testing on constrained devices without those devices maintaining the whole test suite in memory. Therefore, programmers script and run tests on a workstation which then step-wise instructs the constrained device to execute each test, thereby overcoming the memory constraints. Our testing framework further allows developers to mark tests as depending on other tests. This way, Latch can skip tests that depend on previously failing tests resulting in a faster feedback loop. Finally, Latch addresses the issue of timeouts and flaky tests by including an analysis mode that provides feedback on timeouts and the flakiness of tests.
To illustrate the expressiveness of Latch, we present testing scenarios representing unit testing, integration testing, and end-to-end testing. We evaluate the performance of Latch by testing a virtual machine against the WebAssembly specification, with a large test suite consisting of 10,213 tests running on an ESP32 microcontroller. Our experience shows that the testing framework is expressive, reliable and reasonably fast, making it suitable to run large test suites on constrained devices. Furthermore, the debugger-like operations enable to closely mimic manual testing.
Item Type: | Article |
---|---|
DOI/Identification number: | 10.1016/j.scico.2024.103157 |
Additional information: | For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. |
Uncontrolled keywords: | automated testing, embedded devices, flaky tests |
Subjects: | Q Science > QA Mathematics (inc Computing science) |
Divisions: | Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing |
Funders: |
Research Foundation - Flanders (https://ror.org/03qtxy027)
Engineering and Physical Sciences Research Council (https://ror.org/0439y7842) Royal Society (https://ror.org/03wnrjx87) |
Depositing User: | Stefan Marr |
Date Deposited: | 10 Jun 2024 14:24 UTC |
Last Modified: | 26 Jun 2024 11:13 UTC |
Resource URI: | https://kar.kent.ac.uk/id/eprint/106218 (The current URI for this page, for reference purposes) |
- Link to SensusAccess
- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV
- Depositors only (login required):