How To Skip A Set At Pulsar Outbound Connector

Knowledge Base Connectors
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

How To Skip A Set At Pulsar Outbound Connector

Context

When XDR is not able to ship a specific set successfully to a destination cluster due to an issue on the destination cluster, the set is retried continuously on the XDR side. Typically this happens when leveraging a connector where different sets can be shipped to different routes or backends.

In this case, XDR blocks all transactions behind the set being retried, and the XDR lag increases. You may want to skip the specific set causing the shipping failure to allow other sets to progress.

If the backend is Pulsar, use the Pulsar Outbound Connector configuration to specify which sets should be shipped to the Pulsar backend. You do this with a simple set-based inclusion filter, which lists only the desired sets to be shipped.

This can be also used in a scenario when only some specific sets of a namespace need to be shipped to the Pulsar backend.

Skip a set from XDR retry path with Pulsar Outbound Connector Configuration

Make following changes in the Pulsar Outbound Connector Configuration :

  • There is no default routing set
  • There is no routing set at the ‘namespace’ level
  • The routing is set only at the ‘sets’ level for a given set

Sample /etc/aerospike-pulsar-outbound/aerospike-pulsar-outbound.yml changes

# one of json, flat-json, message-pack, or avro
format:
  mode: json

# a list of transformations and mappings.
#bin-transforms:
#  map:
#   yellow: red
#  transforms: # will be done in order
#   - uppercase

#routing:
  #mode: static
  #destination: default

namespaces:
  test:
    sets:
      premium:
        routing:
          mode: static
          destination: premium
        bin-transforms:
          map:
            gold: platinum

After these configuration changes and a restart of Connector service, only the ‘premium’ set is published to the Pulsar backend. All other sets are skipped with a warning.

Notes

  • As of Pulsar Outbound Connector version 2.1.0, there is no set-based exclusion filter.
  • An exclusion filter can be done using XDR Expression Filters at the Aerospike server level.

Applies To

All Pulsar Outbound Connector versions as of this writing.

Keywords

SKIP SET RETRY PATH PULSAR OUTBOUND CONNECTOR

Timestamp

October 2021