https://blg.robot-house.us/tags/hbase/Recent content in Hbase on BlgHugo -- gohugo.ioen-usFri, 26 Feb 2021 07:46:00 +0000https://blg.robot-house.us/posts/opentsdb-retrospective/Fri, 26 Feb 2021 07:46:00 +0000https://blg.robot-house.us/posts/opentsdb-retrospective/Over a year ago, I wrote about the success I’d had at $JOB with OpenTSDB. As with all success, this particular success meant we’d be punished…with more metrics! As our need for an effective TSDB grew, OpenTSDB started to creak. We’re now in the position where we are PoCing alternative metrics storage platforms to address our growing needs. The major problems: The community feels stale and possibly dying. There hasn’t been a release since 2018.https://blg.robot-house.us/posts/upgrading-hbase-away-from-ambari/Wed, 11 Mar 2020 01:39:54 +0000https://blg.robot-house.us/posts/upgrading-hbase-away-from-ambari/Hortonworks and Cloudera have merged: https://www.cloudera.com/about/news-and-blogs/press-releases/2019-01-03-cloudera-and-hortonworks-complete-planned-merger.html. At $JOB, we knew this would mean our Hortonworks install may change or be deprecated. At the same time, the HBase 2.0.2 install we had been using was starting to have some trouble. We had a few problems: Master procedure WAL files were not being properly cleared up. This meant that a master failover would take > 5 minutes to complete while the master read each old procedure WAL for the last ~10 months.https://blg.robot-house.us/posts/repairing-region-overlaps-hbase/Fri, 02 Aug 2019 12:14:00 +0000https://blg.robot-house.us/posts/repairing-region-overlaps-hbase/There is potential, when working with faulty hardware, that your HBase cluster can crash in weird ways, causing the hbase:meta region to report two different regions with the same start or end key. In HBase < 2.0, the hbck tool could fix these overlapping regions, but since that tool is disabled in HBase >= 2.0, how do we fix the problem? Bringing it up means I’ve experienced it, right? Anyway…https://blg.robot-house.us/posts/merging-regions-in-hbase/Mon, 10 Jun 2019 18:46:00 +0000https://blg.robot-house.us/posts/merging-regions-in-hbase/So, like with any job using physical hardware, getting new machines for your growing Hadoop cluster takes time. And sometimes, the amount of ingest starts putting pressure on components before new hosts can be added. Today, our HBase hosts are edging dangerously close to >=200 regions per host. The HBase Book suggests 200+ regions per host is bad. Okay, so we’re edging closer and closer to worse and worse performance as we add data (and regions naturally split) without adding hosts.https://blg.robot-house.us/posts/learning-about-opentsdb/Tue, 19 Feb 2019 18:46:00 +0000https://blg.robot-house.us/posts/learning-about-opentsdb/At $JOB, we needed a highly scalable metric storage system and we had a nice little Hadoop cluster I’d set up that could handle some IOPs. So we decided to test OpenTSDB and Druid. Druid was certainly appealing: They claimed OLAP powers Ambari had a built-in install There was a Grafana plugin The problem with Druid was simple: we couldn’t successfully query data once we got it in. Both the Grafana plugin and CLI tools could not produce readable data after weeks of effort.https://blg.robot-house.us/posts/graphite-and-hbase/Fri, 16 May 2014 10:23:00 +0000https://blg.robot-house.us/posts/graphite-and-hbase/So this is exciting… We run Graphite at work for (relatively) real-time metric collection of our environment. That’s cool. It’s a powerful tool that allows great insight into what is actually happening in our environment. However… Graphite relies (by default) on a flat-file, rrd-like database called whisper. That’s fine when you’re not getting that many metrics (or when you can spring for SSDs to write your data), but when you’re getting over 2.