The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ Why do creates, updates and deletes get sent to different Kafka topics with the Aerospike Outbound Kafka Connector
Detail
When using the Aerospike Outbound Kafka Connector to ship changes from an Aerospike database to Kafka and routing by set, deletes are sent to a different Kafka topic than creates and updates. Why is this?
Answer
If sets are being used to route between different Kafka topics then deletes will not be sent to that topic as deletes are shipped without a set by XDR. As they cannot be routed, the deletes are sent to the default Kafka topic.
In the example below, creates and updates belonging to the set user_profile_test in namespace customer_user_profile_test will be sent to Kafka topic user_profile_test whereas the deletes will be sent to default.
service:
port: 8080
producer-props:
bootstrap.servers:
- kafka-development.mycloud.com:9092
logging:
file: /var/log/aerospike-kafka-outbound/aerospike-kafka-outbound.log
format:
mode: json
routing:
mode: static
destination: default # <<< this is a name of a topic >>>
namespaces:
customer_universal_user_profile:
sets:
user_profile_test:
routing:
mode: static
destination: user_profile_test
format:
mode: flat-json
metadata-key: metadata
Notes
Keywords
KAFKA OUTBOUND TOPIC DELETE SET
Timestamp
August 2020