Librdkafka.jl

Julia wrapper for librdkafka for producing and consuming Apache Kafka messages from Julia. Library based on modern-cpp-kafka.

Platform	x86_64	aarch64	Notes
Linux (glibc)	✓	(planned)	Mainstream Linux distributions (x86_64 supported)
Linux (musl)	(planned)	(planned)	Alpine / minimal containers
macOS	(planned)	✓	Apple Silicon supported, Intel planned
FreeBSD	(planned)	(planned)	Community interest
Windows	(planned)	(planned)	Future support

Prebuilt binaries are published automatically for supported platforms. If you need to build the wrapper from source, run:

cmake -S deps/src -B deps/src/build && cmake --build deps/src/build

Installation

If you haven't installed our local registry yet, do that first:

] registry add https://github.com/bhftbootcamp/Green.git

To install Librdkafka, simply use the Julia package manager:

] add Librdkafka

Usage

Producer

using Librdkafka

topic = "julia-demo"

p = KafkaProducer("localhost:9092")

try
    i = 1
    while true
        msg = "hello #$i from julia"
        produce(p, topic, 0, "key", msg)
        @info "Sent" msg
        i += 1
        sleep(1)
    end
finally
    close(p) # unreachable here, Ctrl+C to stop
end

Consumer

using Librdkafka

topic = "julia-demo"

c = KafkaConsumer(
    "localhost:9092";
    group_id = "julia-demo-consumer",
    config = Dict(
        AUTO_OFFSET_RESET => "earliest",
        ENABLE_AUTO_COMMIT => "false",
    ),
)

subscribe!(c, [topic])

try
    while true
        for r in poll(c; timeout_ms=1000)
            @info "Got" key=r.key value=r.value offset=r.offset
            commit_record(c, r)
        end
    end
finally
    close(c) # unreachable here, Ctrl+C to stop
end

Useful Links

• librdkafka – Official library repository.
• librdkafka_jll – Julia binary wrapper for librdkafka.
• modern-cpp-kafka – a layer of C++ wrapper based on librdkafka.
• doc kafka-clients – configuration refers to the various settings and parameters that can be adjusted to optimize the performance, reliability, and security of a Kafka cluster and its clients.

Contributing

Contributions to Librdkafka are welcome! If you encounter a bug, have a feature request, or would like to contribute code, please open an issue or a pull request on GitHub.