Error -8 when running Spark job

Error -8 when running Spark job

Problem Description

When running a load from Spark via the Aerospike Connector for Spark the load fails with the following error.

20/08/05 13:21:53 ERROR DefaultSource: Key:xxxx:yyyy:00000999-3333-4444-5555-zzzz Error:Error -8,1,1000,1000,0,inDoubt,BB982BC5AFA8C00 10.10.0.1 3000: java.io.EOFException  mode: Ignore  code:-8 

Explanation

This error indicates that there is a server resource error, specifically a lack of scan threads to process the Spark job.

Solution

To resolve this there are two potential options. The job can be throttled from the Spark side or the number of scan threads can be increased within Aerospike. Increasing the scan threads within Aerospike is done via the single-scan-threads and scan-threads-limit parameters. Example commands are shown below. It is advisable to use asadm so that the command executes on all nodes in the cluster.

Setting single-scan-threads to 128 (maximum value).

Admin> asinfo -v "set-config:context=namespace;id=namespaceName;single-scan-threads=128"

Setting scan-threads-limit to 1024 (maximum value).

Admin> asinfo -v "set-config:context=service;scan-threads-limit=64"
ok

Notes

  • The scan parameters mentioned will affect all scans run on the system not just those initiated by running Spark code via the Aerospike Connector for Spark.
  • Parameters given above are relevant to Aerospike 4.9 / Connect for Spark 2.0 and higher.

Keywords

SPARK ERROR -8 EOF EXCEPTION

Timestamp

August 2020

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.