Hi,
We have crash Aerospike CE 5.1.0.3-el7 than use operation in scala-client:
_ <- recordOperations(key, NonEmptyList.one(ListOperation.removeByValueList(bin, toDelete, ListReturnType.NONE)))
Error in client - “Error -8,1,30000,0,0,BB9B2333C008C02 10.232.51.23 3000: java.io.EOFException”
Error in Aerospike logs - “CRITICAL (particle): (cdt.c:2652) offset_index_find_items() sort failed after index filled”
Our config:
service {
user aerospike
group aerospike
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
service-threads 2
proto-fd-max 10000
log-local-time false
debug-allocations all
}
logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
context any debug
}
}
network {
service {
address any
port 3000
}
heartbeat {
mode mesh
address any
port 3002
mesh-seed-address-port 10.10.0.23 3002
interval 150
timeout 10
}
fabric {
port 3001
}
info {
port 3003
}
}
namespace test {
replication-factor 2
memory-size 2G
default-ttl 30d
storage-engine memory
default-ttl 120d
nsup-period 1d
}
kporter
September 18, 2020, 5:41pm
2
Could you provide the stack trace associated with the CRITICAL log?
xorphox
September 18, 2020, 5:46pm
3
What is the contents of toDelete? Anything may help, how many elements it has, etc.
Sep 18 2020 09:37:41 GMT+0300: CRITICAL (particle): (cdt.c:2652) offset_index_find_items() sort failed after index filled
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (signal.c:213) SIGUSR1 received, aborting Aerospike Community Edition build 5.1.0.3 os el7
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:605) stacktrace: registers: rax 0000000000000000 rbx 000000000000000a rcx ffffffffffffffff rdx 000000000000000a rsi 0000000000001d41 rdi 0000000000001d0b rbp 00007f6d2c06e190 rsp 00007f6d2c06dfb8 r8 64656c6c69662078 r9 28736d6574695f64 r10 00007f6d2c06da20 r11 0000000000000202 r12 00007f6d2c06e2a0 r13 0000000000000002 r14 0000000000000000 r15 00007f6d2c06e460 rip 00007f6dce4aa49b
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:618) stacktrace: found 19 frames: 0x5475d6 0x4a1ce4 0x7f6dce4aa5d0 0x7f6dce4aa49b 0x546dac 0x451544 0x47b0b7 0x484376 0x44aae7 0x52e40c 0x52e7cc 0x52f521 0x52fac2 0x530401 0x4d6715 0x4b067a 0x535bc7 0x7f6dce4a2dd5 0x7f6dcd121ead offset 0x0
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 0: /usr/bin/asd(cf_log_stack_trace+0xd4) [0x5475d6]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 1: /usr/bin/asd(as_sig_handle_usr1+0x55) [0x4a1ce4]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 2: /lib64/libpthread.so.0(+0xf5d0) [0x7f6dce4aa5d0]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 3: /lib64/libpthread.so.0(raise+0x2b) [0x7f6dce4aa49b]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 4: /usr/bin/asd(cf_log_write_no_return+0x86) [0x546dac]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 5: /usr/bin/asd(cdt_idx_mask_get_content_sz+0) [0x451544]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 6: /usr/bin/asd() [0x47b0b7]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 7: /usr/bin/asd(cdt_process_state_packed_list_modify_optype+0x1847) [0x484376]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 8: /usr/bin/asd(as_bin_cdt_packed_modify+0x1d1) [0x44aae7]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 9: /usr/bin/asd(write_master_bin_ops_loop+0xa25) [0x52e40c]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 10: /usr/bin/asd(write_master_bin_ops+0xd7) [0x52e7cc]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 11: /usr/bin/asd(write_master_ssd+0x178) [0x52f521]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 12: /usr/bin/asd(write_master+0x2fb) [0x52fac2]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 13: /usr/bin/asd(as_write_start+0xf7) [0x530401]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 14: /usr/bin/asd(as_tsvc_process_transaction+0x825) [0x4d6715]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 15: /usr/bin/asd() [0x4b067a]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 16: /usr/bin/asd() [0x535bc7]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 17: /lib64/libpthread.so.0(+0x7dd5) [0x7f6dce4a2dd5]
Sep 18 2020 09:37:41 GMT+0300: WARNING (as): (log.c:628) stacktrace: frame 18: /lib64/libc.so.6(clone+0x6d) [0x7f6dcd121ead]
toDelete = List(2.2F, 4.4F).map(Value.get).asJava
xorphox
September 21, 2020, 5:44pm
6
That is quite small. What is the contents of the list bin?
it’s testing data:
values = List(2.2F, 3.3F, 4.4F).map(Value.get).asJava
xorphox
September 22, 2020, 8:49pm
8
I’ve used your bin data, input data and settings, but cannot reproduce this crash.
Is this a consistent crash, and/or do you happen to have a core file?
We use java-client, maybe problem is in java-sdk.
rbotzer
September 23, 2020, 8:50pm
10
Maybe, but this needs some reproducible code, not just one line. Are you able to write a small program that can consistently cause the crash? If not, do you have a core file from a recent crash?
package ru.mycompany.testnamespave
import cats.data.NonEmptyList
import cats.effect.IO
import com.aerospike.client.cdt.{ListOperation, ListReturnType}
import com.aerospike.client.{Bin, Value}
import org.aero.writes.PartialEncoder
import org.aero.writes.WriteOps.{BinValueMagnet, WriteParamDef}
import org.aero.{AeroClient, AeroOps, Schema}
import scala.jdk.CollectionConverters._
object AeroRomperApp extends AeroOps[IO] with AdditionalWriteOps[IO] {
val hosts: List[String] = List("localhost")
val port = 3000
val namespace = "testnamespace"
val set = "id_aero_set"
val bin = "some_other_bin"
val key = "test_id"
implicit val schema = Schema(namespace, set)
val encoderFloat: PartialEncoder[Float] = Value.get(_)
implicit val wpd = WriteParamDef.forWBin[Float](encoderFloat)
val initValueMagnet = new BinValueMagnet {
type Out = List[Bin]
override def apply(): List[Bin] = wpd.apply(bin ->> 1.1F)
}
def main(args: Array[String]): Unit = {
implicit val aeroClient: AeroClient[IO] = AeroClient(hosts, port)
(for {
_ <- IO.pure(println("started"))
values = List(2.2F, 3.3F, 4.4F).map(Value.get).asJava
toDelete = List(2.2F, 4.4F).map(Value.get).asJava
_ <- recordOperations(key, NonEmptyList.one(ListOperation.appendItems(bin, values)))
_ <- IO.pure(println("finished append"))
_ <- recordOperations(key, NonEmptyList.one(ListOperation.removeByValueList(bin, toDelete, ListReturnType.NONE)))
_ <- IO.pure(println("finished delete"))
_ <- IO.pure(println("OK"))
} yield {}).unsafeRunSync()
}
}
meher
October 14, 2020, 3:41pm
12
We are about to release a hotfix for an issue we recently found that may potentially help this and would suggest trying it out (hope that hotfix release comes out later today).
Thenk you, problem resolved.
system
January 12, 2021, 1:51pm
14
This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.