OR-Benchmark: An Open and Reconfigurable Digital Watermarking Benchmarking Framework

Benchmarking digital watermarking algorithms is not an easy task because different applications of digital watermarking often have very different sets of requirements and trade-offs between conflicting requirements. While there have been some general-purpose digital watermarking benchmarking systems available, they normally do not support complicated benchmarking tasks and cannot be easily reconfigured to work with different watermarking algorithms and testing conditions. In this paper, we propose OR-Benchmark, an open and highly reconfigurable general-purpose digital watermarking benchmarking framework, which has the following two key features: 1) all the interfaces are public and general enough to support all watermarking applications and benchmarking tasks we can think of; 2) end users can easily extend the functionalities and freely configure what watermarking algorithms are tested, what system components are used, how the benchmarking process runs, and what results should be produced. We implemented a prototype of this framework as a MATLAB software package and demonstrate how it can be used in three typical use cases. The first two use cases show how easily we can define benchmarking profiles for some robust image watermarking algorithms. The third use case shows how OR-Benchmark can be configured to benchmark some image watermarking algorithms for content authentication and self-restoration, which cannot be easily supported by other digital watermarking benchmarking systems.