How To Skip A Set At Pulsar Outbound Connector

How To Skip A Set At Pulsar Outbound Connector

Context

When XDR is not able to ship a specific set successfully to a destination cluster due to an issue on the destination cluster, the set is retried continuously on the XDR side. Typically this happens when leveraging a connector where different sets can be shipped to different routes or backends.

In this case, XDR blocks all transactions behind the set being retried, and the XDR lag increases. You may want to skip the specific set causing the shipping failure to allow other sets to progress.

If the backend is Pulsar, use the Pulsar Outbound Connector configuration to specify which sets should be shipped to the Pulsar backend. You do this with a simple set-based inclusion filter, which lists only the desired sets to be shipped.

This can be also used in a scenario when only some specific sets of a namespace need to be shipped to the Pulsar backend.

Skip a set from XDR retry path with Pulsar Outbound Connector Configuration

Make following changes in the Pulsar Outbound Connector Configuration :

  • There is no default routing set
  • There is no routing set at the ‘namespace’ level
  • The routing is set only at the ‘sets’ level for a given set

Sample /etc/aerospike-pulsar-outbound/aerospike-pulsar-outbound.yml changes

# one of json, flat-json, message-pack, or avro
format:
  mode: json

# a list of transformations and mappings.
#bin-transforms:
#  map:
#   yellow: red
#  transforms: # will be done in order
#   - uppercase

#routing:
  #mode: static
  #destination: default

namespaces:
  test:
    sets:
      premium:
        routing:
          mode: static
          destination: premium
        bin-transforms:
          map:
            gold: platinum

After these configuration changes and a restart of Connector service, only the ‘premium’ set is published to the Pulsar backend. All other sets are skipped with a warning.

Notes

  • As of Pulsar Outbound Connector version 2.1.0, there is no set-based exclusion filter.
  • An exclusion filter can be done using XDR Expression Filters at the Aerospike server level.

Applies To

All Pulsar Outbound Connector versions as of this writing.

Keywords

SKIP SET RETRY PATH PULSAR OUTBOUND CONNECTOR

Timestamp

October 2021

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.