Upgrading HBase Away From Ambari

Hortonworks and Cloudera have merged: https://www.cloudera.com/about/news-and-blogs/press-releases/2019-01-03-cloudera-and-hortonworks-complete-planned-merger.html. At $JOB, we knew this would mean our Hortonworks install may change or be deprecated. At the same time, the HBase 2.0.2 install we had been using was starting to have some trouble.

We had a few problems:

  1. Master procedure WAL files were not being properly cleared up. This meant that a master failover would take > 5 minutes to complete while the master read each old procedure WAL for the last ~10 months. (This was likely the fault of corrupted procedureWAL files and not the HBase Master process itself)
  2. Regionservers were climbing to around 170 regions per node, which starts to exhibit problems. (Compactions not completing, general performance degredation, etc.)
  • The HBase Book (link) goes into a little detail about this here:
  1. HBase 2.0.2 doesn’t properly work with HBCK2, which means either re-compiling the software without that restriction (dangerous), or not having repair tools available in a crisis.

I saw that recently, Cloudera had released Ambari 2.7.4/HDP 3.1.5. These two releases promised one big upgrade for us:

  • HBase would move from 2.0.2 to 2.1.5

We quickly gave up on this path when we discovered an Ambari/HDP upgrade would now require a Cloudera subscription. We could purchase a subscription, but getting approval would take far longer than we wanted to wait for the upgrade to occur.

So what could we do?

It turns out that Apache releases binary builds of HBase (https://hbase.apache.org/downloads.html)! And they’re relatively easy to install: just un-tar and add config! Great! We already had a working HBase config (from our Ambari deployment) and now we had upgraded packages? I quickly started testing a 2.1.6 build in our staging environment, but ran into trouble at startup because HBase kept assuming that the filesystem I had it looking at (HDFS) couldn’t handle certain HDFS commands…Strange…

Upgrading to the latest stable (2.2.3) seemed to resolve our initial deployment problems with regionservers, but we couldn’t successfully start a master instance. We could even add mutiple regionservers per host easily, and regions would successfully move between hosts (including the hbase:meta region) when a node was removed or added. But we couldn’t get the master processes to recognize that we were using a filesystem that supported hsync operations. We can’t really upgrade part of a cluster without ensuring the rest of the cluster is updated, can we? So ensuring we can upgrade the masters to 2.2.3 successfully was important.

After a lot of digging, we finally discovered the problem. Apache builds of HBase are executed with Hadoop 2.x jars. We’re running Hadoop 3.1.1. This library mis-match was causing problems with processes properly communicating. Because the master performs more aggressive checks at startup, it failed when loading the bad jars, where the regionservers didn’t fail, but may have run into issues as we pushed them (our staging environment doesn’t exactly process the same level of data as production). We would have to build HBase 2.2.3 with Hadoop 3.1.1 libraries and deploy that instead of simply downloading a pre-built tarball.

The basic build looks like this:

wget http://apache.mirrors.pair.com/hbase/2.2.3/hbase-2.2.3-src.tar.gz
tar xzf hbase-2.2.3-src.tar.gz
cd hbase-2.2.3
mvn -DskipTests -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.1 clean install
mvn -DskipTests -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.1 package assembly:single

After building the upgraded HBase binaries, deployment was, like with the pre-built binaries, actually quite simple. Copy tarball to directory and execute the start command:

bin/hbase-daemon.sh --config <config dir> foreground_start regionserver
bin/hbase-daemon.sh --config <config dir> foreground_start master

So now we can start a regionserver on a host without Ambari. And also start a HBase Master process on a host without Ambari. Everything successfully fails to other instances properly. The hbase:meta table loads properly. We can query OpenTSDB data from the test cluster successfully (and throughout the upgrade process, in fact).

There were a few challenges left:

  1. We needed to manage service state properly
  2. Regionservers in production still had a large number of regions on them
  3. client commands (e.g. hbase shell) wouldn’t execute properly

Solving #1 is simple. We just wire up some systemd units for regionservers and masters:

Regionserver Unit:

[Unit]
Description=HBase Regionserver Port 16030
After=network-online.target

[Service]
Type=simple
User=hbase
LimitNOFILE=32000
LimitNPROC=16000
ExecStart=/opt/hbase/bin/hbase-daemon.sh --config /etc/hbase/regionserver/16030/ foreground_start regionserver
ExecStop=/opt/hbase/bin/hbase-daemon.sh --config /etc/hbase/regionserver/16030/ stop regionserver
Restart=no

[Install]
WantedBy=default.target

Master Unit:

[Unit]
Description=HBase Master
After=network-online.target

[Service]
Type=simple
User=hbase
LimitNOFILE=32000
LimitNPROC=16000
ExecStart=/opt/hbase/bin/hbase-daemon.sh --config /etc/hbase/master/ foreground_start master
ExecStop=/opt/hbase/bin/hbase-daemon.sh --config /etc/hbase/master/ stop master
Restart=no

[Install]
WantedBy=default.target

Solving 3 is also fairly easy. Turned out an environment variable isn’t getting set quite right in the initially deployed hbase-config.sh that comes with the hbase-client tarball our build produces. Fortunately, fixing that is sed command away:

sed -i 's|HBASE_CONF_DIR=\"${HBASE_CONF_DIR:-$HBASE_HOME/conf}\"|HBASE_CONF_DIR=\"${HBASE_CONF_DIR:-/etc/hbase/master}\"|g' hbase-client/bin/hbase-config.sh

The big question was #2. My co-worker wondered why we couldn’t run multiple regionservers on a single host. And while I couldn’t find any arguments against the situation, I also couldn’t find any examples of multiple HBase Regionservers working on a single node. But running multiple instances on a single node would significantly reduce the region count on a single instance (double the number of regionservers without adding more hardware? Cheap and efficient!)

So we tried starting two regionservers on one host. Unsurprisingly, it worked fine. While there are no indications for or against this behavior I’ve been able to find online, I’m here to tell you: YOU CAN RUN MULTIPLE REGIONSERVERS ON A SINGLE HOST. There are, like with all things, trade-offs:

  1. We’re currently running the regionservers with smaller heaps to avoid over-running memory on our hosts.
  2. This does increase disk I/O on hosts (for obvious reasons), so you may see higher disk latencies (particularly when regionserver processes flush to disk).

The biggest take-away I have from this experience (other than how impressed I am with HBase’s resilience in failure scenarios…we broke everything quite a few times in our test instance and never lost any data) is how incredibly useful the HBase Book is. This is the type of documentation you always hope for with a large system like HBase. There are tuning guides, upgrade guides, and a general wealth of information on how to use HBase in a myriad of situations.