PPAndroid-Benchmarker: Benchmarking Privacy Protection Systems on Android Devices

Mobile devices are ubiquitous in today's digital world. While people enjoy the convenience brought by mobile devices, it has been proven that many mobile apps leak personal information without user consent or even awareness. That can occur due to many reasons, such as careless programming errors, intention of developers to collect private information, infection of innocent apps by malware, etc. Thus, the research community has proposed many methods and systems to detect privacy leakage and prevent such detected leakage on mobile devices. This is a to do note at margin While it is obviously essential to evaluate the accuracy and effectiveness of privacy protection systems, we are not aware of any automated system that can benchmark performance of privacy protection systems on Android devices. In this paper, we report PPAndroid-Benchmarker, the first system of this kind, which can fairly benchmark any privacy protection systems dynamically (i.e., in run time) or statically. PPAndroid-Benchmarker has been released as an open-source tool and we believe that it will help the research community, developers and even end users to analyze, improve, and choose privacy protection systems on Android devices. We applied PPAndroid-Benchmarker in dynamic mode to 165 Android apps with some privacy protection features, selected from variant app markets and the research community, and showed effectiveness of the tool. We also illustrate two components of PPAndroid-Benchmarker on the design level, which are Automatic Test Apps Generator for benchmarking static analysis based tools and Reconfigurability Engine that allows any instance of PPAndroid-Benchmarker to be reconfigured including but not limited to adding and removing information sources and sinks. Furthermore, we give some insights about current status of mobile privacy protection and prevention in app markets based upon our analysis.