Unishox2 Text Compression for MeshCore

[gjelsoe]

Unishox2 Text Compression for MeshCore

Feature proposal — USE_LZW

Note on naming: The build flag USE_LZW and related identifiers do not refer to the LZW compression algorithm. LZW is used here as a general term for the compression feature. The actual compression algorithm used is Unishox2, which is specifically designed for short strings and significantly outperforms LZW in this context.

Memory Footprint

Tested on Heltec V3 running Companion Radio BLE firmware v1.14.0:

	RAM	Flash
Without LZW	168,964 bytes (51.6%)	1,254,453 bytes (37.5%)
With LZW	169,348 bytes (51.7%)	1,263,233 bytes (37.8%)
Difference	+384 bytes	+8,780 bytes

The overhead is minimal — 384 bytes of additional RAM and less than 9 KB of additional Flash. The Flash increase covers the Unishox2 library and the compression/decompression code. The RAM increase is primarily the stack buffers used during compression and decompression.

Overview

This proposal adds optional Unishox2 text compression to MeshCore's private messaging layer. When both the sender and recipient are running compression-capable firmware, messages are compressed before encryption, reducing payload size by 25–40% for typical natural language messages. Nodes without compression support are fully unaffected — the feature is strictly opt-in and backward compatible.

Why Unishox2

Standard compression algorithms such as LZW or DEFLATE require hundreds of bytes of input before producing any meaningful reduction in size. MeshCore messages are capped at 160 bytes, making these algorithms unsuitable. Unishox2 is specifically designed for short Unicode strings and delivers positive compression ratios starting from around 40 characters. It handles ASCII, accented characters, and emoji (multi-byte UTF-8) correctly, and its C implementation (unishox2.c / unishox2.h) is small enough to run comfortably on ESP32, nRF52, RP2040, and STM32 targets.

Measured Compression Results

Example results observed during development:

Message	Original	Compressed	Saving
"The quick brown fox jumps over the lazy dog"	43 bytes	30 bytes	30%
"The phrase is commonly used for touch-typing..."	85 bytes	52 bytes	39%
Message with emoji content	95 bytes	69 bytes	27%

Messages below 40 characters, or messages with high entropy content such as base64 strings or coordinates, are sent uncompressed automatically.

Implementation

Build flag

The entire feature is gated behind a single build flag. No code paths are affected in builds without it:

build_flags = -DUSE_LZW

New files

src/helpers/MeshCompression.h / .cpp
Wrapper around Unishox2 with compression heuristics. The heuristics reject messages that are too short, have high character entropy, or a low ratio of alphabetic characters. Multi-byte UTF-8 sequences (emoji etc.) are handled correctly by counting Unicode codepoints rather than raw bytes.

lib/unishox2/unishox2.c / unishox2.h
The Unishox2 library by Siara Creations (Apache 2.0). Placed in the /lib directory so PlatformIO compiles it automatically without any changes to build_src_filter. Source: https://github.com/siara-cc/Unishox2

Modified files

src/helpers/TxtDataHelpers.h
Adds TXT_FLAG_COMPRESSED (0x80) — bit 7 of payload byte[4]. This bit is currently unused by the existing attempt number and TXT_TYPE fields.

src/helpers/AdvertDataHelpers.h
Adds ADV_CAP_LZW (0x0001) advertised in FEAT2. FEAT2 is currently unused.

src/helpers/BaseChatMesh.cpp
Four small #ifdef USE_LZW blocks are added:

1. createSelfAdvert() — advertises ADV_CAP_LZW in FEAT2 so other nodes can detect capability.
2. populateContactFromAdvert() and onAdvertRecv() update block — sets or clears CONTACT_FLAG_SUPPORTS_LZW on a contact whenever an advertisement is received, ensuring the flag stays current as nodes update firmware.
3. composeMsgPacket() — compresses the message if the recipient has CONTACT_FLAG_SUPPORTS_LZW set. A 2-byte length prefix is prepended to the compressed data so the receiver can determine the exact compressed length independently of AES block padding.
4. onPeerDataRecv() — decompresses the message if TXT_FLAG_COMPRESSED is set in byte[4]. The ACK hash is calculated on the raw compressed payload using the 2-byte length prefix, matching the sender's hash exactly.

Payload Format

Uncompressed message (unchanged from original):

[ timestamp:4 ][ flags:1 ][ text:N ]

Compressed message:

[ timestamp:4 ][ flags:1 | 0x80 ][ comp_len:2 ][ compressed_text:N ]

The 2-byte length prefix is necessary because AES-128 pads the encrypted payload to the nearest 16-byte block boundary. Without the prefix, the receiver cannot determine where the compressed data ends and padding begins, causing the ACK hash to mismatch.

Capability Negotiation

Nodes announce compression support by setting ADV_CAP_LZW (0x0001) in the FEAT2 field of their advertisement packet. When a node receives an advertisement from a peer, it sets or clears CONTACT_FLAG_SUPPORTS_LZW (bit 4 of ContactInfo::flags) accordingly. A node will only send compressed messages to contacts that have this flag set.

Compression activates automatically after both nodes have exchanged advertisements — no manual configuration is required. If a node downgrades to non-LZW firmware, its advertisement will no longer carry ADV_CAP_LZW, and the flag will be cleared on the peer at the next advertisement.

Important: After downgrading from LZW firmware to standard firmware, the node must send a new advertisement so that peers can detect the capability change and stop sending compressed messages. Until a new advertisement is received, peers that previously marked this node as LZW-capable may continue sending compressed messages which the downgraded node cannot decompress.

Backward Compatibility

Nodes without USE_LZW are completely unaffected. A compression-capable node will never send compressed data to a node that has not advertised ADV_CAP_LZW. Messages from non-LZW nodes are received and ACK'd normally by LZW-capable nodes. Mixed networks of LZW and non-LZW nodes operate without any issues.

Note on ContactInfo::flags Bit Allocation

ContactInfo::flags is a uint8_t managed directly by the companion app via CMD_ADD_UPDATE_CONTACT over the serial protocol — it is not derived from FEAT1 or FEAT2 in the advertisement packet. The companion app uses the following bits:

Bit 0 (0x01) — Favourite flag (protects contact from overwrite)
Bit 1 (0x02) — TELEM_PERM_BASE        (read as contact.flags >> 1 & 0x01)
Bit 2 (0x04) — TELEM_PERM_LOCATION    (read as contact.flags >> 1 & 0x02)
Bit 3 (0x08) — TELEM_PERM_ENVIRONMENT (read as contact.flags >> 1 & 0x04)
Bit 4 (0x10) — CONTACT_FLAG_SUPPORTS_LZW  ← this proposal
Bit 5 (0x20) — unused
Bit 6 (0x40) — unused
Bit 7 (0x80) — unused

FEAT1 and FEAT2 in the advertisement packet are separate fields used only as a transport mechanism — their values are not stored directly in ContactInfo::flags. This proposal uses FEAT2 to advertise LZW capability and stores the result in bit 4 of ContactInfo::flags, which is currently unused by the companion app.

Future Work

If a stream cipher such as ChaCha20-Poly1305 or Ascon is adopted in a future MeshCore release, the 2-byte length prefix will no longer be necessary since stream ciphers produce no block padding. The compression layer itself requires no changes — it remains independent of the encryption layer. Combining Unishox2 compression with a stream cipher would yield consistent 25–40% airtime savings on every message over 40 characters.

The following table compares encrypted payload sizes across four configurations. Stream cipher overhead is 8 bytes (4-byte counter + 4-byte tag as proposed in PR #1450 and PR #1677). AES overhead is 2 bytes HMAC plus block padding to the nearest 16 bytes.

Message	AES only	AES + LZW	Stream only	Stream + LZW	Saving vs AES
43 bytes (short text)	50 bytes	34 bytes	51 bytes	40 bytes	20%
85 bytes (medium text)	98 bytes	66 bytes	93 bytes	62 bytes	37%
95 bytes (text with emoji)	98 bytes	82 bytes	103 bytes	79 bytes	19%
130 bytes (long text)	146 bytes	82 bytes	138 bytes	88 bytes	40%

The combination of Stream + LZW consistently outperforms AES alone and eliminates the unpredictability caused by AES block padding.

Two open PRs are strong candidates for this combination:

• PR Cryptography overhaul for V2 #1450 — Ascon-128 AEAD encryption
• PR Add ChaChaPoly AEAD-4 encryption with nonce persistence #1677 — ChaCha20-Poly1305 AEAD encryption

Both eliminate AES block padding and would make the compression savings fully predictable on every message.

Credits

This implementation was developed with assistance from Claude (Anthropic).

Source Code

The full implementation is available at:
https://github.com/gjelsoe/MeshCore/tree/LZW-Messages