HConfig: Resource Adaptive Fast Bulk Loading in HBase

HConfig: Resource Adaptive Fast Bulk Loading in HBase

NoSQL (Not only SQL) data stores become a vital component in many big data computing platforms due to its inherent horizontal scalability. HBase is an open-source distributed NoSQL store that is widely used by many Internet enterprises to handle their big data computing applications (e.g. Facebook handles millions of messages each day with HBase). Optimizations that can enhance the performance of HBase are of paramount interests for big data applications that use HBase or Big Table like key-value stores. In this paper we study the problems inherent in misconfiguration of HBase clusters, including scenarios where the HBase default configurations can lead to poor performance. We develop HConfig, a semiautomated configuration manager for optimizing HBase system performance from multiple dimensions. Due to the space constraint, this paper will focus on how to improve the performance of HBase data loader using HConfig. Through this case study we will highlight the importance of resource adaptive and workload aware auto-configuration management and the design principles of HConfig. Our experiments show that the HConfig enhanced bulk loading can significantly improve the performance of HBase bulk loading jobs compared to the HBase default configuration, and achieve 2~3.7x speedup in throughput under different client threads while maintaining linear horizontal scalability.

EXISTING SYSTEM

  • Default configuration may provide poor resource utilization of HBase cluster for some test cases.

PROPOSED SYSTEM

  • To study the problems inherent in misconfiguration of HBase clusters, including scenarios where the HBase default configurations may lead to poor performance.
  • HConfig is proposed, a semi-automated configuration manager for optimizing HBase system performance from multiple dimensions.
  • Proposed system study how to improve the HBase bulk loading performance by HConfig.
  • Resource adaptive and workload aware autoconfiguration management and the design principles of HConfig is done.

Advantages

  • HConfig enhanced bulk loading can significantly improve the performance of HBase bulk loading jobs compared to the HBase default configuration, and achieve 2~3.7x speedup in throughput under different client threads while maintaining linear horizontal scalability .

SOFTWARE SPECIFICATION

Programming Language  : JDK 1.5 or higher, Hadoop

Database  : HBase

Leave a Reply