Crash Aerospike with ValuesByList

Hi,

We have crash Aerospike CE 5.1.0.3-el7 than use operation in scala-client:

      _ <- recordOperations(key, NonEmptyList.one(ListOperation.removeByValueList(bin, toDelete, ListReturnType.NONE)))

Error in client - “Error -8,1,30000,0,0,BB9B2333C008C02 10.232.51.23 3000: java.io.EOFException

Error in Aerospike logs - “CRITICAL (particle): (cdt.c:2652) offset_index_find_items() sort failed after index filled” Our config:

service {
        user aerospike
        group aerospike
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 2
        proto-fd-max 10000
        log-local-time false
        debug-allocations all
}

logging {
        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any debug
        }
}

network {
        service {
                address any
                port 3000
        }

        heartbeat {
                mode mesh
                address any
                port 3002
                mesh-seed-address-port 10.10.0.23 3002
                interval 150
                timeout 10
        }
        fabric {
                port 3001
        }
        info {
                port 3003
        }
}

namespace test {
       replication-factor 2
       memory-size 2G
       default-ttl 30d 
       storage-engine memory
       default-ttl 120d
       nsup-period 1d
}

Could you provide the stack trace associated with the CRITICAL log?

What is the contents of toDelete? Anything may help, how many elements it has, etc.

Sep 18 2020 09:37:41 GMT+0300: CRITICAL (particle): (cdt.c:2652) offset_index_find_items() sort failed after index filled
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (signal.c:213) SIGUSR1 received, aborting Aerospike Community Edition build 5.1.0.3 os el7
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:605) stacktrace: registers: rax 0000000000000000 rbx 000000000000000a rcx ffffffffffffffff rdx 000000000000000a rsi 0000000000001d41 rdi 0000000000001d0b rbp 00007f6d2c06e190 rsp 00007f6d2c06dfb8 r8 64656c6c69662078 r9 28736d6574695f64 r10 00007f6d2c06da20 r11 0000000000000202 r12 00007f6d2c06e2a0 r13 0000000000000002 r14 0000000000000000 r15 00007f6d2c06e460 rip 00007f6dce4aa49b
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:618) stacktrace: found 19 frames: 0x5475d6 0x4a1ce4 0x7f6dce4aa5d0 0x7f6dce4aa49b 0x546dac 0x451544 0x47b0b7 0x484376 0x44aae7 0x52e40c 0x52e7cc 0x52f521 0x52fac2 0x530401 0x4d6715 0x4b067a 0x535bc7 0x7f6dce4a2dd5 0x7f6dcd121ead offset 0x0
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 0: /usr/bin/asd(cf_log_stack_trace+0xd4) [0x5475d6]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 1: /usr/bin/asd(as_sig_handle_usr1+0x55) [0x4a1ce4]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 2: /lib64/libpthread.so.0(+0xf5d0) [0x7f6dce4aa5d0]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 3: /lib64/libpthread.so.0(raise+0x2b) [0x7f6dce4aa49b]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 4: /usr/bin/asd(cf_log_write_no_return+0x86) [0x546dac]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 5: /usr/bin/asd(cdt_idx_mask_get_content_sz+0) [0x451544]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 6: /usr/bin/asd() [0x47b0b7]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 7: /usr/bin/asd(cdt_process_state_packed_list_modify_optype+0x1847) [0x484376]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 8: /usr/bin/asd(as_bin_cdt_packed_modify+0x1d1) [0x44aae7]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 9: /usr/bin/asd(write_master_bin_ops_loop+0xa25) [0x52e40c]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 10: /usr/bin/asd(write_master_bin_ops+0xd7) [0x52e7cc]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 11: /usr/bin/asd(write_master_ssd+0x178) [0x52f521]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 12: /usr/bin/asd(write_master+0x2fb) [0x52fac2]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 13: /usr/bin/asd(as_write_start+0xf7) [0x530401]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 14: /usr/bin/asd(as_tsvc_process_transaction+0x825) [0x4d6715]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 15: /usr/bin/asd() [0x4b067a]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 16: /usr/bin/asd() [0x535bc7]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 17: /lib64/libpthread.so.0(+0x7dd5) [0x7f6dce4a2dd5]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 18: /lib64/libc.so.6(clone+0x6d) [0x7f6dcd121ead]
toDelete = List(2.2F, 4.4F).map(Value.get).asJava

That is quite small. What is the contents of the list bin?

it’s testing data: values = List(2.2F, 3.3F, 4.4F).map(Value.get).asJava

I’ve used your bin data, input data and settings, but cannot reproduce this crash. Is this a consistent crash, and/or do you happen to have a core file?

We use java-client, maybe problem is in java-sdk.

Maybe, but this needs some reproducible code, not just one line. Are you able to write a small program that can consistently cause the crash? If not, do you have a core file from a recent crash?

 package ru.mycompany.testnamespave

import cats.data.NonEmptyList
import cats.effect.IO
import com.aerospike.client.cdt.{ListOperation, ListReturnType}
import com.aerospike.client.{Bin, Value}
import org.aero.writes.PartialEncoder
import org.aero.writes.WriteOps.{BinValueMagnet, WriteParamDef}
import org.aero.{AeroClient, AeroOps, Schema}

import scala.jdk.CollectionConverters._

object AeroRomperApp extends AeroOps[IO] with AdditionalWriteOps[IO] {
  val hosts: List[String] = List("localhost")
  val port = 3000
  val namespace = "testnamespace"

  val set = "id_aero_set"
  val bin = "some_other_bin"
  val key = "test_id"
  implicit val schema = Schema(namespace, set)
  val encoderFloat: PartialEncoder[Float] = Value.get(_)
  implicit val wpd = WriteParamDef.forWBin[Float](encoderFloat)
  val initValueMagnet = new BinValueMagnet {
    type Out = List[Bin]
    override def apply(): List[Bin] = wpd.apply(bin ->> 1.1F)
  }

  def main(args: Array[String]): Unit = {
    implicit val aeroClient: AeroClient[IO] = AeroClient(hosts, port)
    (for {
      _ <- IO.pure(println("started"))
      values = List(2.2F, 3.3F, 4.4F).map(Value.get).asJava
      toDelete = List(2.2F, 4.4F).map(Value.get).asJava
      _ <- recordOperations(key, NonEmptyList.one(ListOperation.appendItems(bin, values)))
      _ <- IO.pure(println("finished append"))
      
      _ <- recordOperations(key, NonEmptyList.one(ListOperation.removeByValueList(bin, toDelete, ListReturnType.NONE)))
      _ <- IO.pure(println("finished delete"))
      _ <- IO.pure(println("OK"))
    } yield {}).unsafeRunSync()
  }
}

We are about to release a hotfix for an issue we recently found that may potentially help this and would suggest trying it out (hope that hotfix release comes out later today).

Thenk you, problem resolved.

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.