Kafka-aerospike connector issue


#1

Hi I am tried to start kafka to aerospike connector but there is error

when i ran this command

$ connect-standalone config/connect-standalone.properties config/aerospike-connector.properties

[root@rrm3 kafka_2.11-0.9.0.1]# connect-standalone config/connect-standalone.properties config/aerospike-connector.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-elasticsearch/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-hdfs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-replicator/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-s3/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-storage-common/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2017-07-15 05:14:55,291] INFO StandaloneConfig values:
        access.control.allow.methods =
        access.control.allow.origin =
        bootstrap.servers = [localhost:9092]
        internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
        internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
        key.converter = class org.apache.kafka.connect.json.JsonConverter
        offset.flush.interval.ms = 10000
        offset.flush.timeout.ms = 5000
        offset.storage.file.filename = /tmp/connect.offsets
        rest.advertised.host.name = null
        rest.advertised.port = null
        rest.host.name = null
        rest.port = 8083
        task.shutdown.graceful.timeout.ms = 5000
        value.converter = class org.apache.kafka.connect.json.JsonConverter
 (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:180)
[2017-07-15 05:14:55,464] INFO Logging initialized @1052ms (org.eclipse.jetty.util.log:186)
[2017-07-15 05:14:55,820] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:50)
[2017-07-15 05:14:55,821] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:72)
[2017-07-15 05:14:55,821] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:119)
[2017-07-15 05:14:55,821] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:60)
[2017-07-15 05:14:55,824] INFO Worker started (org.apache.kafka.connect.runtime.Worker:124)
[2017-07-15 05:14:55,825] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:74)
[2017-07-15 05:14:55,825] INFO Starting REST server (org.apache.kafka.connect.runtime.rest.RestServer:98)
[2017-07-15 05:14:55,989] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327)
Jul 15, 2017 5:14:57 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2017-07-15 05:14:57,062] INFO Started o.e.j.s.ServletContextHandler@1d9bec4d{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)
[2017-07-15 05:14:57,078] INFO Started ServerConnector@9d157ff{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)
[2017-07-15 05:14:57,080] INFO Started @2668ms (org.eclipse.jetty.server.Server:379)
[2017-07-15 05:14:57,081] INFO REST server listening at http://172.31.1.42:8083/, advertising URL http://172.31.1.42:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150)
[2017-07-15 05:14:57,082] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)
[2017-07-15 05:14:57,082] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:99)
java.io.FileNotFoundException: config/aerospike-connector.properties (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:488)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:83)
[2017-07-15 05:14:57,089] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)
[2017-07-15 05:14:57,089] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)
[2017-07-15 05:14:57,112] INFO Stopped ServerConnector@9d157ff{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)
[2017-07-15 05:14:57,134] INFO Stopped o.e.j.s.ServletContextHandler@1d9bec4d{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)
[2017-07-15 05:14:57,136] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:165)
[2017-07-15 05:14:57,136] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:78)
[2017-07-15 05:14:57,136] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:131)
[2017-07-15 05:14:57,137] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:68)
[2017-07-15 05:14:57,137] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:151)
[2017-07-15 05:15:14,972] INFO Reflections took 19055 ms to scan 579 urls, producing 13713 keys and 89742 values  (org.reflections.Reflections:229)
[2017-07-15 05:15:15,006] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:88)
[2017-07-15 05:15:15,007] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)

#2

@arulkumar,

This seems to be a mistake in the documentation for the connector. The name of the config file is actually connect-aerospike-sink.properties. So you should start the connector in stand-alone mode like this:

connect-standalone config/connect-standalone.properties config/config/connect-aerospike-sink.properties

I will fix the documentation as well.

Cheers, Jan


#3

Hi ,Kafka-aerospike connectivity issue

I have follwed same property as you given for this issue

$connect-standalone config/connect-standalone.properties config/connect-aerospike-sink.properties

i got following error

[root@rrm3 kafka_2.11-0.9.0.1]# connect-standalone config/connect-standalone.properties config/connect-aerospike-sink.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-elasticsearch/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-hdfs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-replicator/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-s3/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka-connect-storage-common/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/confluent/confluent-3.2.0/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2017-07-18 07:39:59,519] INFO StandaloneConfig values:
        access.control.allow.methods =
        access.control.allow.origin =
        bootstrap.servers = [localhost:9092]
        internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
        internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
        key.converter = class org.apache.kafka.connect.json.JsonConverter
        offset.flush.interval.ms = 10000
        offset.flush.timeout.ms = 5000
        offset.storage.file.filename = /tmp/connect.offsets
        rest.advertised.host.name = null
        rest.advertised.port = null
        rest.host.name = null
        rest.port = 8083
        task.shutdown.graceful.timeout.ms = 5000
        value.converter = class org.apache.kafka.connect.json.JsonConverter
 (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:180)
[2017-07-18 07:39:59,689] INFO Logging initialized @871ms (org.eclipse.jetty.util.log:186)
[2017-07-18 07:40:00,031] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:50)
[2017-07-18 07:40:00,032] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:72)
[2017-07-18 07:40:00,032] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:119)
[2017-07-18 07:40:00,032] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:60)
[2017-07-18 07:40:00,035] INFO Worker started (org.apache.kafka.connect.runtime.Worker:124)
[2017-07-18 07:40:00,036] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:74)
[2017-07-18 07:40:00,036] INFO Starting REST server (org.apache.kafka.connect.runtime.rest.RestServer:98)
[2017-07-18 07:40:00,196] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327)
Jul 18, 2017 7:40:01 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.
[2017-07-18 07:40:01,287] INFO Started o.e.j.s.ServletContextHandler@1d9bec4d{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)
[2017-07-18 07:40:01,303] INFO Started ServerConnector@9d157ff{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)
[2017-07-18 07:40:01,305] INFO Started @2489ms (org.eclipse.jetty.server.Server:379)
[2017-07-18 07:40:01,307] INFO REST server listening at http://172.31.1.42:8083/, advertising URL http://172.31.1.42:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150)
[2017-07-18 07:40:01,307] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)
[2017-07-18 07:40:19,924] INFO Reflections took 19802 ms to scan 579 urls, producing 13713 keys and 89742 values  (org.reflections.Reflections:229)
[2017-07-18 07:40:20,548] INFO Reflections took 19207 ms to scan 579 urls, producing 13713 keys and 89742 values  (org.reflections.Reflections:229)
[2017-07-18 07:40:20,551] ERROR Failed to create job for config/connect-aerospike-sink.properties (org.apache.kafka.connect.cli.ConnectStandalone:88)
[2017-07-18 07:40:20,552] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:99)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.aerospike.kafka.connect.sink.AerospikeSinkConnector, available connectors are: org.apache.kafka.connect.sink.SinkConnector, org.apache.kafka.connect.tools.VerifiableSourceConnector, org.apache.kafka.connect.tools.MockSourceConnector, org.apache.kafka.connect.tools.SchemaSourceConnector, io.confluent.connect.elasticsearch.ElasticsearchSinkConnector, io.confluent.connect.storage.tools.SchemaSourceConnector, io.confluent.connect.jdbc.JdbcSinkConnector, io.confluent.connect.hdfs.tools.SchemaSourceConnector, org.apache.kafka.connect.tools.MockConnector, io.confluent.connect.s3.S3SinkConnector, org.apache.kafka.connect.tools.VerifiableSinkConnector, io.confluent.connect.jdbc.JdbcSourceConnector, org.apache.kafka.connect.tools.MockSinkConnector, org.apache.kafka.connect.file.FileStreamSourceConnector, io.confluent.connect.hdfs.HdfsSinkConnector, org.apache.kafka.connect.file.FileStreamSinkConnector, org.apache.kafka.connect.source.SourceConnector, io.confluent.connect.replicator.ReplicatorSourceConnector
        at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:80)
        at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:67)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:96)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.aerospike.kafka.connect.sink.AerospikeSinkConnector, available connectors are: org.apache.kafka.connect.sink.SinkConnector, org.apache.kafka.connect.tools.VerifiableSourceConnector, org.apache.kafka.connect.tools.MockSourceConnector, org.apache.kafka.connect.tools.SchemaSourceConnector, io.confluent.connect.elasticsearch.ElasticsearchSinkConnector, io.confluent.connect.storage.tools.SchemaSourceConnector, io.confluent.connect.jdbc.JdbcSinkConnector, io.confluent.connect.hdfs.tools.SchemaSourceConnector, org.apache.kafka.connect.tools.MockConnector, io.confluent.connect.s3.S3SinkConnector, org.apache.kafka.connect.tools.VerifiableSinkConnector, io.confluent.connect.jdbc.JdbcSourceConnector, org.apache.kafka.connect.tools.MockSinkConnector, org.apache.kafka.connect.file.FileStreamSourceConnector, io.confluent.connect.hdfs.HdfsSinkConnector, org.apache.kafka.connect.file.FileStreamSinkConnector, org.apache.kafka.connect.source.SourceConnector, io.confluent.connect.replicator.ReplicatorSourceConnector
        at org.apache.kafka.connect.runtime.ConnectorFactory.getConnectorClass(ConnectorFactory.java:84)
        at org.apache.kafka.connect.runtime.ConnectorFactory.newConnector(ConnectorFactory.java:38)
        at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:334)
        at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:233)
        at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:159)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:93)
[2017-07-18 07:40:20,554] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)
[2017-07-18 07:40:20,554] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)
[2017-07-18 07:40:20,558] INFO Stopped ServerConnector@9d157ff{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)
[2017-07-18 07:40:20,565] INFO Stopped o.e.j.s.ServletContextHandler@1d9bec4d{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)
[2017-07-18 07:40:20,567] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:165)
[2017-07-18 07:40:20,567] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:78)
[2017-07-18 07:40:20,567] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:131)
[2017-07-18 07:40:20,568] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:68)
[2017-07-18 07:40:20,568] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:151)
[2017-07-18 07:40:20,568] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:88)
[2017-07-18 07:40:20,568] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)
[root@rrm3 kafka_2.11-0.9.0.1]#  ERROR Failed to create job for config/connect-aerospike-sink.properties (org.apache.kafka.connect.cli.ConnectStandalone:88)

#4

When it seems to be a bug that kind of information is useful on the GitHub repo, in this case aerospike/aerospike-kafka-connector.

Please open a new issue at the repo, with this information. It would be good if you added information about your environment (OS, etc).