Skip to main content

NetAgg: Using Middleboxes for Application-specific On-path Aggregation in Data Centres

Mai, Luo and Rupprecht, Lukas and Abdul, Alim and Costa, Paolo and Migliavacca, Matteo and Pietzuch, Peter and Wolf, Alexander L. (2014) NetAgg: Using Middleboxes for Application-specific On-path Aggregation in Data Centres. In: Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies. CoNEXT International Conference on Emerging Networking Experiments and Technologies . ACM, New York, USA, pp. 249-262. ISBN 978-1-4503-3279-8. (doi:10.1145/2674005.2674996) (The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided)

The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided. (Contact us about this Publication)
Official URL
http://dx.doi.org/10.1145/2674005.2674996

Abstract

Data centre applications for batch processing (e.g. map/reduce frameworks) and online services (e.g. search engines) scale by distributing data and computation across many servers. They typically follow a partition/aggregation pattern: tasks are first partitioned across servers that process data locally, and then those partial results are aggregated. This data aggregation step, however, shifts the performance bottleneck to the network, which typically struggles to support many-to-few, high-bandwidth traffic between servers.

Instead of performing data aggregation at edge servers, we show that it can be done more efficiently along network paths. We describe NETAGG, a software platform that supports on-path aggregation for network-bound partition/aggregation applications. NETAGG exploits a middlebox-like design, in which dedicated servers (agg boxes) are connected by high-bandwidth links to network switches. Agg boxes execute aggregation functions provided by applications, which alleviates network hotspots because only a fraction of the incoming traffic is forwarded at each hop. NETAGG requires only minimal application changes: it uses shim layers on edge servers to redirect application traffic transparently to the agg boxes. Our experimental results show that NETAGG improves substantially the throughput of two sample applications, the Solr distributed search engine and the Hadoop batch processing framework. Its design allows for incremental deployment in existing data centres and incurs only a modest investment cost.

Item Type: Book section
DOI/Identification number: 10.1145/2674005.2674996
Subjects: Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming,
Divisions: Faculties > Sciences > School of Computing > Data Science
Depositing User: Matteo Migliavacca
Date Deposited: 09 Jul 2015 11:38 UTC
Last Modified: 24 Jan 2020 12:11 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/49289 (The current URI for this page, for reference purposes)
Migliavacca, Matteo: https://orcid.org/0000-0002-5684-4865
  • Depositors only (login required):