The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
Error -8 when running Spark job
Problem Description
When running a load from Spark via the Aerospike Connector for Spark the load fails with the following error.
20/08/05 13:21:53 ERROR DefaultSource: Key:xxxx:yyyy:00000999-3333-4444-5555-zzzz Error:Error -8,1,1000,1000,0,inDoubt,BB982BC5AFA8C00 10.10.0.1 3000: java.io.EOFException mode: Ignore code:-8
Explanation
This error indicates that there is a server resource error, specifically a lack of scan threads to process the Spark job.
Solution
To resolve this there are two potential options. The job can be throttled from the Spark side or the number of scan threads can be increased within Aerospike. Increasing the scan threads within Aerospike is done via the single-scan-threads and scan-threads-limit parameters. Example commands are shown below. It is advisable to use asadm so that the command executes on all nodes in the cluster.
Setting single-scan-threads to 128 (maximum value).
Admin> asinfo -v "set-config:context=namespace;id=namespaceName;single-scan-threads=128"
Setting scan-threads-limit to 1024 (maximum value).
Admin> asinfo -v "set-config:context=service;scan-threads-limit=64"
ok
Notes
- The scan parameters mentioned will affect all scans run on the system not just those initiated by running Spark code via the Aerospike Connector for Spark.
- Parameters given above are relevant to Aerospike 4.9 / Connect for Spark 2.0 and higher.
Keywords
SPARK ERROR -8 EOF EXCEPTION
Timestamp
August 2020