FAQ Why do creates, updates and deletes get sent to different Kafka topics with the Aerospike Outbound Kafka Connector

FAQ Why do creates, updates and deletes get sent to different Kafka topics with the Aerospike Outbound Kafka Connector

Detail

When using the Aerospike Outbound Kafka Connector to ship changes from an Aerospike database to Kafka and routing by set, deletes are sent to a different Kafka topic than creates and updates. Why is this?

Answer

If sets are being used to route between different Kafka topics then deletes will not be sent to that topic as deletes are shipped without a set by XDR. As they cannot be routed, the deletes are sent to the default Kafka topic.

In the example below, creates and updates belonging to the set user_profile_test in namespace customer_user_profile_test will be sent to Kafka topic user_profile_test whereas the deletes will be sent to default.

service:
  port: 8080
producer-props:
  bootstrap.servers:
    - kafka-development.mycloud.com:9092
logging:
  file: /var/log/aerospike-kafka-outbound/aerospike-kafka-outbound.log
format:
  mode: json
routing: 
  mode: static
  destination: default # <<< this is a name of a topic >>>
namespaces:
  customer_universal_user_profile:
    sets:
      user_profile_test:
        routing:
          mode: static
          destination: user_profile_test
        format:
          mode: flat-json
          metadata-key: metadata

Notes

Keywords

KAFKA OUTBOUND TOPIC DELETE SET

Timestamp

August 2020

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.