Eblob backend
Eblob is a low-level local append-overwrite (configurable) storage system. It combines fast read/write performance with configuration simplicity and rich feature set. Its main goal was to provide bullet-proof low-level backend for Elliptics distributed storage (this is default backend now).
Eblob is an append-only low-level IO library, which saves data in blob files. Elliptics uses it as one of its low-level IO backends.
Supported features:
- Fast append-only updates which do not require disk seeks
- Compact index to populate lookup information from disk
- Multi-threaded index reading during startup
- O(1) data location lookup time (for in-memory indexes)
- Ability to lock in-memory lookup index (hash table) to eliminate memory swap
- Readahead games with data and index blobs for maximum performance
- Multiple blob files support (tested with single blob file on block device too)
- Optional sha512 on-disk checksumming
- 2-stage write: prepare (which reserves the space) and commit (which calculates checksum and update in-memory and on-disk indexes). One can (re)write data using pwrite() in between without locks
- Usuall 1-stage write interface
- Fexible configuration of hash table size, flags, alignment
- Defragmentation tool: entries to be deleted are only marked as removed, eblob_check will iterate over specified blob files and actually remove those blocks
- Off-line blob consistency checker: eblob_check can verify checksums for all records which have them enabled
- Run-time sync support — dedicated thread runs fsync in background on all files on timed base
- Google’s Snappy compression support
- Sorted data and indexes on disk – ideal for column creation, iteration, subkeys and range requests
- In-memory index compression (upto 60%) ~64 bytes per key in RAM
Performance:
Features: