# Feedback wanted: i added native streams (kafka-style logs + pub/sub) to the aerospike wire protocol. is my approach sane?

**URL:** <https://discuss.aerospike.com/t/feedback-wanted-i-added-native-streams-kafka-style-logs-pub-sub-to-the-aerospike-wire-protocol-is-my-approach-sane/12865>\
**Category:** General Discussion\
**Created:** [June 1, 2026, 7:07pm UTC](https://discuss.aerospike.com/t/feedback-wanted-i-added-native-streams-kafka-style-logs-pub-sub-to-the-aerospike-wire-protocol-is-my-approach-sane/12865 "2026-06-01T19:07:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![madmax007](https://avatars.discourse-cdn.com/v4/letter/m/77aa72/32.png) [@madmax007](https://discuss.aerospike.com/u/madmax007)\
**Post date:** [June 1, 2026, 7:07pm UTC](https://discuss.aerospike.com/t/feedback-wanted-i-added-native-streams-kafka-style-logs-pub-sub-to-the-aerospike-wire-protocol-is-my-approach-sane/12865/1 "2026-06-01T19:07:35Z")

</div>

Hello all,

I have been running aerospike CE for a while and kept hitting the same wall. the moment i needed event-driven stuff on top of aerospike data i had to bolt kafka next to it. two systems, two failure domains, replication lag i could never get rid of. so i tried an experiment. what if the log and the broker just lived inside the aerospike server itself?

the result is aerostream. it’s a self-contained server module that adds kafka-style stream primitives straight to the wire protocol. durable partitioned logs, consumer groups with committed offsets, replay/seek, and ephemeral pub/sub, all over the existing port 3000.

it’s based on CE 8.1.2.1 and it works end to end today. i’ve got a node client doing produce, consume, replay and pub/sub against it. but before i take it further i’d really like a sanity check from people who know the internals, because a few parts feel like i’m reaching into the engine in ways the architecture might not intend.

how it’s wired (i kept the core diff tiny on purpose):

- 8 new as\_proto message types (10-17), dispatched from service.c. everything else lives in a new as/src/modules/aerostream/. the patch to existing files is about 40 lines.

- produce: hash a partition key, assign an offset with as\_faa\_uint64 on an in-memory counter, then write a record (key {stream}:{partition}:{offset}) via an IOPS internal transaction (as\_transaction\_init\_iops plus as\_service\_enqueue\_internal).

- consume: a persistent server-push session. i spawn a dedicated thread per partition that holds the client fd open and pushes STREAM\_RECORD messages. records get read with as\_partition\_reserve, as\_record\_get, as\_storage\_rd\_load\_bins from those threads.

- offsets: committed offsets and per-stream config are stored as normal records. offset counters get reconstructed on restart by an exponential + binary search over the log keys.

- retention: per-record TTL plus nsup, no compaction process.

where i’d most love feedback:

1. reading records from non-service threads. the push loop does as\_partition\_reserve, as\_record\_get, as\_storage\_rd\_load\_bins outside the normal transaction path. is that safe around partition migrations, rebalance, and record locking? this is the part i trust the least.

2. IOPS for internal writes. is as\_transaction\_init\_iops plus as\_service\_enqueue\_internal the right way to drive internal writes from a module, or is there a lighter or more correct path?

3. holding a client fd in a dedicated push thread, outside the service/epoll pool. does that fight the connection lifecycle, or is it workable?

4. claiming proto type values 10-17. any conflicts, reserved ranges, or guidance i should know about?

5. multi-node reality. single node works, but i haven’t validated behavior under a real cluster with migrations, or how this should play with XDR.

repos:

- main project (design doc, full wire protocol spec, node client, examples): [https://github.com/aerostream-aerospike/aerostream](https://github.com/aerostream-aerospike/aerostream)

- server fork with the module: [https://github.com/aerostream-aerospike/aerospike-server](https://github.com/aerostream-aerospike/aerospike-server)

i’m not trying to upstream anything. this is more a “does this approach make sense, or am i about to get bitten by something the engine guarantees that i’m quietly violating?” kind of post. any pointers, war stories, or “don’t do that, here’s why” are very welcome.

Cheers 🙂