The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ Is there a way to get detailed debug information from the Spark connector?
Detail
When using the Aerospike Connector for Spark to move data between Aerospike and Spark, is there a way to get detailed debugging information on what is going on?
Answer
Yes, when pyspark within a console window it is possible to set the logging level to debug using the following statement:
sc.setLogLevel('DEBUG')
This will generate detailed debugging information on any subsequent statements that are executed. This can be useful when looking to see what data is being loaded where.
Notes
- Full documentation on changing log level in Spark.
- Reference documentation for the Aerospike Spark Connector.
Keywords
SPARK CONNECTOR DEBUG LOG LEVEL
Timestamp
July 2020