Error -8 when running Spark job
When running a load from Spark via the Aerospike Connector for Spark the load fails with the following error.
20/08/05 13:21:53 ERROR DefaultSource: Key:xxxx:yyyy:00000999-3333-4444-5555-zzzz Error:Error -8,1,1000,1000,0,inDoubt,BB982BC5AFA8C00 10.10.0.1 3000: java.io.EOFException mode: Ignore code:-8
This error indicates that there is a server resource error, specifically a lack of
scan threads to process the Spark job.
To resolve this there are two potential options. The job can be throttled from the Spark side or the number of
scan threads can be increased within Aerospike. Increasing the
scan threads within Aerospike is done via the
scan-threads-limit parameters. Example commands are shown below. It is advisable to use
asadm so that the command executes on all nodes in the cluster.
single-scan-threads to 128 (maximum value).
Admin> asinfo -v "set-config:context=namespace;id=namespaceName;single-scan-threads=128"
scan-threads-limit to 1024 (maximum value).
Admin> asinfo -v "set-config:context=service;scan-threads-limit=64" ok
- The scan parameters mentioned will affect all scans run on the system not just those initiated by running Spark code via the Aerospike Connector for Spark.
- Parameters given above are relevant to Aerospike 4.9 / Connect for Spark 2.0 and higher.
SPARK ERROR -8 EOF EXCEPTION