File Download

  • Find it @ UNIST can give you direct access to the published full text of this article. (UNISTARs only)

Views & Downloads

Detailed Information

Cited time in webofscience Cited time in scopus
Metadata Downloads

Cache Affinity-aware In-memory Caching Management for Hadoop

Author(s)
Kwak, Jaewon
Advisor
Choi, Young-Ri
Issued Date
2017-02
URI
https://scholarworks.unist.ac.kr/handle/201301/72094 http://unist.dcollection.net/jsp/common/DcLoOrgPer.jsp?sItemId=000002332694
Abstract
In this paper, we investigate techniques to effectively manage HDFS in-memory caching for Hadoop.
We first revisit the current implementation of Hadoop with HDFS in-memory caching to understand its limitation on the effective usage of in-memory caching.
For various representative MapReduce applications, we also evaluate a degree of benefit each application can get from in-memory caching, i.e. cache affinity.
We then propose an adaptive cache local scheduling algorithm that adaptively computes how long a MapReduce job waits to be scheduled on a cache local node to be proportional to the percentage of cached input data for the job.
In addition, we propose a block goodness aware cache replacement algorithm that determines which block is cached and evicted based on the accessed rate and the cache affinity of applications.
Using various workloads consisting of multiple MapReduce applications, we conduct extensive experimental study to demonstrate the effects of the proposed in-memory orchestration techniques. Our experimental results show that our enhanced Hadoop in-memory caching scheme improves the performance of the MapReduce workloads.
Publisher
Ulsan National Institute of Science and Technology (UNIST)
Degree
Master
Major
Department of Computer Engineering

qrcode

Items in Repository are protected by copyright, with all rights reserved, unless otherwise indicated.