FAQ - Should parameters be set on the fly with the Aerospike Spark Connector?

FAQ - Should parameters be set on the fly with the Aerospike Spark Connector?

Detail

There are certain Aerospike parameters that can be passed to a Spark session at any stage of session lifecycle, as long as it happens before using an Aerospike specific call. Examples of these parameters would be aerospike.seedhost, aerospike.port and aerospike.namespace. Should these be treated as parameters that can be set on the fly?

Answer

While it is possible to set these parameters during runtime, the pool of clients is instantiated from the settings and keyed by the hosts. When these settings are changed the pool will keep filling with more clients, which may not be desirable. For this reason, setting these parameters on the fly, while possible, is not recommended.

Keywords

SPARK CONNECTOR PARAMETER RUNTIME

Timestamp

November 2019

I fail to see how these parameters can be dynamic. These are all initialization parameters.

     val sqlContext = spark.sqlContext
     spark.conf.set("aerospike.seedhost", dbHost)
     spark.conf.set("aerospike.port", dbPort)
     spark.conf.set("aerospike.namespace",namespace)
     spark.conf.set("aerospike.set", dbSet)
     spark.conf.set("aerospike.keyPath", "/etc/aerospike/features.conf")
     spark.conf.set("aerospike.user", dbConnection)
     spark.conf.set("aerospike.password", dbPassword)
     spark.conf.set("aerospike.sendKey", true)

Also references has to be made to the features.conf (assuming this is enterprise edition) file that is specific to every aerospike instance and that has to exist in every host that is running Spark executable. Otherwise Spark job will fail.

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.