Performance and capacity guide

Documentation home · Configuration · Limitations

Measure the current driver on your workload. This guide makes no claim that native RPC is automatically faster than HappyBase, or that larger buffers reduce RPC count.

Understand the execution path

Operation	Current behavior	Measurement implication
Put / Get	Per-row request	Latency and connection behavior matter
BatchPut / BatchGet	Sequential per-row calls	API convenience, not a multi-action speedup
BufferedMutator	In-memory grouping, then individual calls	Measure flush time and failures, not enqueue speed alone
Scanner iteration	Uses next_batch(1)	Per-row consumption can cost more requests
next_batch(n)	Requests chunks of rows	Tune chunk size together with row size
scan_page()	New scanner for each page	Convenient stateless pagination with setup overhead

Batch and buffering are not bulk loading. Increasing a pool setting does not make sequential Python calls concurrent. Pool limits, cache behavior, socket reuse, and thread-safety need observation rather than assumptions.

A reproducible measurement

Use a dedicated table, record row/value size and key distribution, and measure a successful end-to-end operation. The following contextual example assumes table has cf and is disposable:

from time import perf_counter
from uuid import uuid4

from hbasedriver.operations.get import Get
from hbasedriver.operations.put import Put

prefix = f"bench:{uuid4().hex}:".encode("ascii")
keys = [prefix + f"{number:06d}".encode("ascii") for number in range(100)]
payload = b"x" * 256

started = perf_counter()
for key in keys:
    table.put(Put(key).add_column(b"cf", b"value", payload))
elapsed = perf_counter() - started
print({"successful_writes": len(keys), "seconds": elapsed,
       "rows_per_second": len(keys) / elapsed})

for key in keys:
    row = table.get(Get(key))
    assert row is not None and row.get(b"cf", b"value") == payload

This writes benchmark data and does not delete it. Run it only in a table you will explicitly remove afterward. A failed call aborts the measurement; do not report the planned count as completed throughput.

For buffered writes, place the timer before enqueueing and stop it after flush/close returns. If flush fails, report partial/unknown outcomes and reconcile the input. Record separate enqueue latency and durable-operation completion latency if you need both.

Tune reads

1. Start with the smallest useful row-key interval.
2. Select necessary families and apply suitable server-side filters.
3. Use next_batch(n) for bounded chunks; compare realistic sizes such as 20, 100, and 500.
4. Measure bytes returned, peak memory, request latency, and rows processed.
5. Close scanners promptly, including on early exit or processing failure.

Do not use Scan.set_limit() as a global safety cap. Scan.add_column() currently has a serialization bug; family selection plus a qualifier filter is the documented alternative. Do not request untested partial-row settings as a performance shortcut.

Tune writes and keys

• Choose row keys with actual access patterns in mind. Monotonically increasing hot keys can concentrate writes; salting distributes keys but complicates scans.
• Keep counters separate from UTF-8 numeric strings.
• Bound upstream queues; BufferedMutator is not a persistent retry queue.
• Avoid repeatedly constructing clients inside a per-row loop, while also validating the current connection cleanup/sharing limitations for long-lived services.
• Do not share Table/Client objects across threads without targeted correctness and resource tests.

Changing a column-family setting in Python is not proof that the server adopted it. Verify compression, TTL, versions, and block settings through the actual HBase schema before attributing a performance change to them.

Compare configurations fairly

Record:

• repository commit/package version, Python/protobuf versions, HBase version;
• CPU, memory, number of RegionServers, Region layout, storage, and network;
• payload distribution, row-key distribution, row width, and number of versions;
• cold/warm cache state, warmup duration, total run duration, concurrency;
• completed operations, p50/p95/p99 latency, errors, unknown outcomes, and resource usage.

Repeat runs and retain raw measurements. The bundled Docker cluster shares local resources and storage; it is useful for correctness checks, not a substitute for production sizing. Do not extrapolate a short smoke test into a throughput guarantee.

Diagnose saturation

If CPU is low but latency is high, inspect network round trips and server logs. If memory grows, check materialized scans, application queues, and incomplete resource cleanup. If one RegionServer is busy, inspect key distribution and Region placement. If failures rise with concurrency, reduce load and investigate connection/cache safety before enlarging pools.

Use read-only cluster diagnostics from administration. Balancer changes, Region splits, and schema changes are operational actions, not routine client benchmark steps.