Error code 11 returned when i add a new namespace -- set failed, 11


#1

When i add a new namespace in /etc/aerospike/aerospike.conf and run cli set command, i got the error

set failed, 11

and i go to check the log, i got this:

Nov 18 2015 06:47:16 GMT: WARNING (tsvc): (thr_tsvc.c::382) rejecting client transaction - initial partition balance unresolved

here is my conf file content:

# Aerospike database configuration file.

service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
service-threads 4
transaction-queues 4
transaction-threads-per-queue 4
proto-fd-max 15000
}

logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
        context any info
}
}

network {
service {
        address any
        port 3000
}

heartbeat {
        mode multicast
        address 239.1.99.222
        port 9918

        # To use unicast-mesh heartbeats, remove the 3 lines above, and see
        # aerospike_mesh.conf for alternative.

        interval 150
        timeout 10
}

fabric {
        port 3001
}

info {
        port 3003
}
}

namespace test {
replication-factor 2
memory-size 3G
default-ttl 30d # 30 days, use 0 to never expire/evict.

storage-engine memory
}

namespace bar {
replication-factor 2
memory-size 3G
default-ttl 30d # 30 days, use 0 to never expire/evict.

storage-engine memory

# To use file storage backing, comment out the line above and use the
# following lines instead.
#       storage-engine device {
#               file /opt/aerospike/data/bar.dat
#               filesize 16G
#               data-in-memory true # Store data in memory in addition to file.
#       }
}

namespace BPO {
replication-factor 2
memory-size 2G
default-ttl 30d # 30 days, use 0 to never expire/evict.

storage-engine memory
}

aerospike version: 3.6.4

OS: Linux ubuntu-luck 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Any help ? thanks


#2

Hm, how many nodes are in this cluster?

This issue should be transient, basically this message means the client discovered and began making requests of this node before it was fully initialized.

CLI doesn’t use a smart client which means it will send requests to any node regardless of the current partition table state. It is much less likely that a regular client would reach this state.


#3

Here is my asinfo, it looks like I only have one node in this cluster.

    1 :  node
 BB94DCBA012BCF8

    2 :  statistics
 cluster_size=1;cluster_key=79DE4AE71322EDEF;cluster_integrity=true;objects=0;sub-records=0;total-bytes-disk=0;used-bytes-disk=0;free-pct-disk=0;total-bytes-memory=8589934592;used-bytes-memory=37120;data-used-bytes-memory=0;index-used-bytes-memory=0;sindex-used-bytes-memory=37120;free-pct-memory=99;stat_read_reqs=0;stat_read_reqs_xdr=0;stat_read_success=0;stat_read_errs_notfound=0;stat_read_errs_other=0;stat_write_reqs=0;stat_write_reqs_xdr=0;stat_write_success=0;stat_write_errs=0;stat_xdr_pipe_writes=0;stat_xdr_pipe_miss=0;stat_delete_success=0;stat_rw_timeout=0;udf_read_reqs=0;udf_read_success=0;udf_read_errs_other=0;udf_write_reqs=0;udf_write_success=0;udf_write_err_others=0;udf_delete_reqs=0;udf_delete_success=0;udf_delete_err_others=0;udf_lua_errs=0;udf_scan_rec_reqs=0;udf_query_rec_reqs=0;udf_replica_writes=0;stat_proxy_reqs=0;stat_proxy_reqs_xdr=0;stat_proxy_success=0;stat_proxy_errs=0;stat_ldt_proxy=0;stat_cluster_key_trans_to_proxy_retry=0;stat_cluster_key_transaction_reenqueue=0;stat_slow_trans_queue_push=0;stat_slow_trans_queue_pop=0;stat_slow_trans_queue_batch_pop=0;stat_cluster_key_regular_processed=0;stat_cluster_key_prole_retry=0;stat_cluster_key_err_ack_dup_trans_reenqueue=0;stat_cluster_key_partition_transaction_queue_count=0;stat_cluster_key_err_ack_rw_trans_reenqueue=0;stat_expired_objects=0;stat_evicted_objects=0;stat_deleted_set_objects=0;stat_evicted_objects_time=0;stat_zero_bin_records=0;stat_nsup_deletes_not_shipped=0;err_tsvc_requests=2;err_tsvc_requests_timeout=0;err_out_of_space=0;err_duplicate_proxy_request=0;err_rw_request_not_found=0;err_rw_pending_limit=0;err_rw_cant_put_unique=0;fabric_msgs_sent=300;fabric_msgs_rcvd=150;paxos_principal=BB98A46C3270008;migrate_msgs_sent=0;migrate_msgs_recv=0;migrate_progress_send=0;migrate_progress_recv=0;migrate_num_incoming_accepted=0;migrate_num_incoming_refused=0;queue=0;transactions=710;reaped_fds=0;scans_active=0;basic_scans_succeeded=0;basic_scans_failed=0;aggr_scans_succeeded=0;aggr_scans_failed=0;udf_bg_scans_succeeded=0;udf_bg_scans_failed=0;batch_index_initiate=0;batch_index_queue=0:0,0:0,0:0,0:0;batch_index_complete=0;batch_index_timeout=0;batch_index_errors=0;batch_index_unused_buffers=0;batch_index_huge_buffers=0;batch_index_created_buffers=0;batch_index_destroyed_buffers=0;batch_initiate=0;batch_queue=0;batch_tree_count=0;batch_timeout=0;batch_errors=0;info_queue=0;delete_queue=0;proxy_in_progress=0;proxy_initiate=0;proxy_action=0;proxy_retry=0;proxy_retry_q_full=0;proxy_unproxy=0;proxy_retry_same_dest=0;proxy_retry_new_dest=0;write_master=0;write_prole=0;read_dup_prole=0;rw_err_dup_internal=0;rw_err_dup_cluster_key=0;rw_err_dup_send=0;rw_err_write_internal=0;rw_err_write_cluster_key=0;rw_err_write_send=0;rw_err_ack_internal=0;rw_err_ack_nomatch=0;rw_err_ack_badnode=0;client_connections=11;waiting_transactions=0;tree_count=0;record_refs=0;record_locks=0;migrate_tx_objs=0;migrate_rx_objs=0;ongoing_write_reqs=0;err_storage_queue_full=0;partition_actual=0;partition_replica=0;partition_desync=0;partition_absent=12288;partition_object_count=0;partition_ref_count=12288;system_free_mem_pct=96;sindex_ucgarbage_found=0;sindex_gc_locktimedout=0;sindex_gc_inactivity_dur=706989;sindex_gc_activity_dur=11;sindex_gc_list_creation_time=8;sindex_gc_list_deletion_time=0;sindex_gc_objects_validated=0;sindex_gc_garbage_found=0;sindex_gc_garbage_cleaned=0;system_swapping=false;err_replica_null_node=0;err_replica_non_null_node=0;err_sync_copy_null_master=0;storage_defrag_corrupt_record=0;err_write_fail_prole_unknown=0;err_write_fail_prole_generation=0;err_write_fail_unknown=0;err_write_fail_key_exists=0;err_write_fail_generation=0;err_write_fail_generation_xdr=0;err_write_fail_bin_exists=0;err_write_fail_parameter=0;err_write_fail_incompatible_type=0;err_write_fail_noxdr=0;err_write_fail_prole_delete=0;err_write_fail_not_found=0;err_write_fail_key_mismatch=0;err_write_fail_record_too_big=0;err_write_fail_bin_name=0;err_write_fail_bin_not_found=0;err_write_fail_forbidden=0;stat_duplicate_operation=0;uptime=750;stat_write_errs_notfound=0;stat_write_errs_other=0;heartbeat_received_self=4937;heartbeat_received_foreign=4935;query_reqs=0;query_success=0;query_fail=0;query_abort=0;query_avg_rec_count=0;query_short_running=0;query_long_running=0;query_short_queue_full=0;query_long_queue_full=0;query_short_reqs=0;query_long_reqs=0;query_agg=0;query_agg_success=0;query_agg_err=0;query_agg_abort=0;query_agg_avg_rec_count=0;query_lookups=0;query_lookup_success=0;query_lookup_err=0;query_lookup_abort=0;query_lookup_avg_rec_count=0

    3 :  features
 float;batch-index;replicas-all;replicas-master;replicas-prole;udf

    4 :  cluster-generation
 0

    5 :  partition-generation
 0

    6 :  edition
 Aerospike Community Edition

    7 :  version
 Aerospike Community Edition build 3.6.4

    8 :  build
 3.6.4

    9 :  services
 
    10 :  services-alumni

Also I used the PHP client to set bins to the namesapce, still got no luck. Here is my PHP code:

<?php
$config = [
    "hosts" => [
        ["addr" => "127.0.0.1", "port" => 3000]
    ]
];
$db = new Aerospike($config);

if (!$db->isConnected()) {
  echo "Failed to connect to the Aerospike server [{$db->errorno()}]: {$db->error()}\n";
  exit(1);
}

$key = $db->initKey("test", "characters", 3);
$bins = [
  "Occupation" => ["Bender", "Criminal", "Iron Chef"],
  "Alma Mater" => "Bending State University"];

$status=$db->put($key,$bins);

if ($status == Aerospike::OK) {
    echo "Record written.\n";
} else {
    echo "[{$db->errorno()}] ".$db->error().PHP_EOL;
}

this will output “[11] AEROSPIKE_ERR_CLUSTER”.

So any further advice ?

thanks


#4

Hm, we seem to think we are getting heartbeats from a foreign source (ie another node). This is very strange, the server logs should be interesting.

This could be the issue, if you will need to add the namespace to all node and restart the cluster.


#5

Sorry, I’m new to use aerospike. Is it the only file i need to edit to add the namespace in /etc/aerospike/aerospike.conf ?

If it is so, I have restarted the cluster already. but no luck.


#6

Thank goodness!! It works finally. I do the following steps to make it work.

  1. rm aerospike.conf
  2. cp aerospike_ssd.conf aerospike.conf
  3. add new namespace to the new conf file

It works now. But I still can’t understand why the default conf file does not work correctly.

so weird. Now my conf content is here:

# Aerospike database configuration file for deployments using raw storage.

service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
service-threads 8
transaction-queues 8
transaction-threads-per-queue 8
proto-fd-max 15000
}

logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
        context any info
}
}

network {
service {
        address any
        port 3000
}

heartbeat {
        mode multicast
        address 239.1.99.222
        port 9918

        # To use unicast-mesh heartbeats, remove the 3 lines above, and see
        # aerospike_mesh.conf for alternative.

        interval 150
        timeout 10
}

fabric {
        port 3001
}

info {
        port 3003
}
}

namespace test {
replication-factor 2
memory-size 4G
default-ttl 30d # 30 days, use 0 to never expire/evict.
storage-engine memory
}

namespace Paul {
replication-factor 2
memory-size 4G
default-ttl 30d # 30 days, use 0 to never expire/evict.
storage-engine memory
}

#7

Congrats! :smiley:

Not sure what was going on with the default config? It is working locally :confused:.