# Scan and duplicated records (AER-3648)

**URL:** <https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274>\
**Category:** Query & Indexing\
**Created:** [May 14, 2015, 9:35am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274 "2015-05-14T09:35:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 14, 2015, 9:35am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/1 "2015-05-14T09:35:45Z")

</div>

HI all,

I’m using Aerospike 3.5.4

Cluster is composed of 3 nodes (replication factor 2).

Using Java Client (the one of the online course) I inserted 3 records in “Users” SET and 3 records in “Tweets” SET.

When Java Client tries to scan all tweets of all users it gets duplicated records!

The same result if a use as client a python script or AQL.

In all 3 cases clients connect to the Cluster specifying the IP Address of one of the 3 nodes.

If clients specify, in connect phase, Ip Addresses of all the 3 Nodes, scan run correctly and duplicated records are not returned.

Am I missing something?

Many thanks in advance. Moreno

---

<div class="post-metadata">

**Author:** ![pratyyy](https://avatars.discourse-cdn.com/v4/letter/p/5f9b8f/32.png) [@pratyyy](https://discuss.aerospike.com/u/pratyyy)\
**Post date:** [May 15, 2015, 6:24am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/2 "2015-05-15T06:24:24Z")

</div>

Hi moreno,

Can you share the following information with us ?

1. Is the namespace same for both the sets ?
2. AQL command and its output.

Thanks

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 15, 2015, 6:38am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/3 "2015-05-15T06:38:57Z")

</div>

Thank you pratyyy!

Yes, namespace is the same for both the sets.

Aerospike.conf:

namespace test { replication-factor 2 memory-size 4G default-ttl 30d storage-engine memory }

SETS are “users” and “tweets”.

AQL output when I connect to localhost:

aql\> select \* from test.users ±-----------±-----------±---------±-------±--------------±-----------±------------±-------+ | key | username | password | region | lasttweeted | tweetcount | interests | gender | ±-----------±-----------±---------±-------±--------------±-----------±------------±-------+ | “pietro” | “pietro” | “qwerty” | “n” | 1430984773607 | 1 | [“inter”] | “m” | | “pietro” | “pietro” | “qwerty” | “n” | 1430984773607 | 1 | [“inter”] | “m” | | “pietro” | “pietro” | “qwerty” | “n” | 1430984773607 | 1 | [“inter”] | “m” | | “giuseppe” | “giuseppe” | “qwerty” | “s” | 1430984928958 | 1 | [“sicilia”] | “f” | | “moreno” | “moreno” | “qwerty” | “n” | 1430984519751 | 1 | [“juve”] | “m” | | “giuseppe” | “giuseppe” | “qwerty” | “s” | 1430984928958 | 1 | [“sicilia”] | “f” | | “moreno” | “moreno” | “qwerty” | “n” | 1430984519751 | 1 | [“juve”] | “m” | ±-----------±-----------±---------±-------±--------------±-----------±------------±-------+

7 rows in set (1.392 secs)

(7 rows when I inserted 3)

If I connect to another host in the cluster:

aql -h 10.100.0.4

aql\> select \* from test.users ±-----------±-----------±---------±-------±-------±--------------±-----------±------------+ | key | username | password | gender | region | lasttweeted | tweetcount | interests | ±-----------±-----------±---------±-------±-------±--------------±-----------±------------+ | “giuseppe” | “giuseppe” | “qwerty” | “f” | “s” | 1430984928958 | 1 | [“sicilia”] | | “moreno” | “moreno” | “qwerty” | “m” | “n” | 1430984519751 | 1 | [“juve”] | | “pietro” | “pietro” | “qwerty” | “m” | “n” | 1430984773607 | 1 | [“inter”] | | “pietro” | “pietro” | “qwerty” | “m” | “n” | 1430984773607 | 1 | [“inter”] | | “pietro” | “pietro” | “qwerty” | “m” | “n” | 1430984773607 | 1 | [“inter”] | ±-----------±-----------±---------±-------±-------±--------------±-----------±------------+ 5 rows in set (1.563 secs)

The same when I use a python client:

./users.10.100.0.3.py

(‘test’, ‘users’, u’pietro’, bytearray(b’\xc8\_\xf85\xd1\xfe\x00&\xfc\xe1\x0ctO,\x83\xf17!\xe3\xcf’)) {‘gen’: 13, ‘ttl’: 2589144} {‘username’: u’pietro’}

(‘test’, ‘users’, u’giuseppe’, bytearray(b’\xbf\x90\x01\xdf\x8c=\xf1\xfe\xda\xfa\nL\xaf\xec\nQ\xa4xE\x97’)) {‘gen’: 4, ‘ttl’: 2528163} {‘username’: u’giuseppe’}

(‘test’, ‘users’, u’moreno’, bytearray(b’t]87X\xb2\x82\x01\xf9\xc7P\xbcW\x81\xffKO\xd4\xd0\x18’)) {‘gen’: 2, ‘ttl’: 1901527} {‘username’: u’moreno’}

(‘test’, ‘users’, u’giuseppe’, bytearray(b’\xbf\x90\x01\xdf\x8c=\xf1\xfe\xda\xfa\nL\xaf\xec\nQ\xa4xE\x97’)) {‘gen’: 4, ‘ttl’: 2528163} {‘username’: u’giuseppe’}

(‘test’, ‘users’, u’moreno’, bytearray(b’t]87X\xb2\x82\x01\xf9\xc7P\xbcW\x81\xffKO\xd4\xd0\x18’)) {‘gen’: 2, ‘ttl’: 1901527} {‘username’: u’moreno’}

5 records.

If I use a client python that connect to hosts 10.100.0.3 and 10.100.0.4 the result is OK:

./users.10.100.0.3e4.py

(Here are the first rows of python script:

# Configure the client

config = { ‘hosts’: [(‘10.100.0.3’, 3000), (‘10.100.0.4’, 3000)] }

and its execution:

(‘test’, ‘users’, u’pietro’, bytearray(b’\xc8\_\xf85\xd1\xfe\x00&\xfc\xe1\x0ctO,\x83\xf17!\xe3\xcf’)) {‘gen’: 13, ‘ttl’: 2589074} {‘username’: u’pietro’}

(‘test’, ‘users’, u’giuseppe’, bytearray(b’\xbf\x90\x01\xdf\x8c=\xf1\xfe\xda\xfa\nL\xaf\xec\nQ\xa4xE\x97’)) {‘gen’: 4, ‘ttl’: 2528094} {‘username’: u’giuseppe’}

(‘test’, ‘users’, u’moreno’, bytearray(b’t]87X\xb2\x82\x01\xf9\xc7P\xbcW\x81\xffKO\xd4\xd0\x18’)) {‘gen’: 2, ‘ttl’: 1901458} {‘username’: u’moreno’}

3 records.

The hosts have Linux Centos as their operating system:

CentOS release 6.5 (Final)

uname -a

Linux couchbase-1 2.6.32-431.el6.x86\_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86\_64 x86\_64 x86\_64 GNU/Linux

---

<div class="post-metadata">

**Author:** ![pratyyy](https://avatars.discourse-cdn.com/v4/letter/p/5f9b8f/32.png) [@pratyyy](https://discuss.aerospike.com/u/pratyyy)\
**Post date:** [May 15, 2015, 9:13am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/5 "2015-05-15T09:13:17Z")

</div>

Thanks for the info.

Can you please also send us the output of command

_asinfo -v “statistics”_

Thanks

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 15, 2015, 9:54am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/6 "2015-05-15T09:54:21Z")

</div>

ASINFO on node 1:

cluster\_size=3 cluster\_key=A5A28076E110C33B cluster\_integrity=true objects=2010382 sub-records=0 total-bytes-disk=0 used-bytes-disk=0 free-pct-disk=0 total-bytes-memory=8589934592 used-bytes-memory=282140675 data-used-bytes-memory=127340916 index-used-bytes-memory=128664448 sindex-used-bytes-memory=26135311 free-pct-memory=96 stat\_read\_reqs=3 stat\_read\_reqs\_xdr=0 stat\_read\_success=0 stat\_read\_errs\_notfound=3 stat\_read\_errs\_other=0 stat\_write\_reqs=4 stat\_write\_reqs\_xdr=0 stat\_write\_success=3 stat\_write\_errs=1 stat\_xdr\_pipe\_writes=0 stat\_xdr\_pipe\_miss=0 stat\_delete\_success=8 stat\_rw\_timeout=0 udf\_read\_reqs=0 udf\_read\_success=0 udf\_read\_errs\_other=0 udf\_write\_reqs=0 udf\_write\_success=0 udf\_write\_err\_others=0 udf\_delete\_reqs=0 udf\_delete\_success=0 udf\_delete\_err\_others=0 udf\_lua\_errs=0 udf\_scan\_rec\_reqs=0 udf\_query\_rec\_reqs=0 udf\_replica\_writes=0 stat\_proxy\_reqs=0 stat\_proxy\_reqs\_xdr=0 stat\_proxy\_success=0 stat\_proxy\_errs=0 stat\_ldt\_proxy=0 stat\_cluster\_key\_trans\_to\_proxy\_retry=0 stat\_cluster\_key\_transaction\_reenqueue=0 stat\_slow\_trans\_queue\_push=0 stat\_slow\_trans\_queue\_pop=0 stat\_slow\_trans\_queue\_batch\_pop=0 stat\_cluster\_key\_regular\_processed=0 stat\_cluster\_key\_prole\_retry=0 stat\_cluster\_key\_err\_ack\_dup\_trans\_reenqueue=0 stat\_cluster\_key\_partition\_transaction\_queue\_count=0 stat\_cluster\_key\_err\_ack\_rw\_trans\_reenqueue=0 stat\_expired\_objects=0 stat\_evicted\_objects=0 stat\_deleted\_set\_objects=0 stat\_evicted\_set\_objects=0 stat\_evicted\_objects\_time=0 stat\_zero\_bin\_records=0 stat\_nsup\_deletes\_not\_shipped=0 err\_tsvc\_requests=1 err\_out\_of\_space=0 err\_duplicate\_proxy\_request=0 err\_rw\_request\_not\_found=0 err\_rw\_pending\_limit=0 err\_rw\_cant\_put\_unique=0 fabric\_msgs\_sent=10263439 fabric\_msgs\_rcvd=10263420 paxos\_principal=BB9CF35A5565000 migrate\_msgs\_sent=10221978 migrate\_msgs\_recv=10263301 migrate\_progress\_send=0 migrate\_progress\_recv=0 migrate\_num\_incoming\_accepted=16464 migrate\_num\_incoming\_refused=0 queue=0 transactions=697914 reaped\_fds=12 tscan\_initiate=840 tscan\_pending=0 tscan\_succeeded=836 tscan\_aborted=0 batch\_initiate=0 batch\_queue=0 batch\_tree\_count=0 batch\_timeout=0 batch\_errors=0 info\_queue=0 delete\_queue=0 proxy\_in\_progress=0 proxy\_initiate=0 proxy\_action=0 proxy\_retry=0 proxy\_retry\_q\_full=0 proxy\_unproxy=0 proxy\_retry\_same\_dest=0 proxy\_retry\_new\_dest=0 write\_master=4 write\_prole=28 read\_dup\_prole=0 rw\_err\_dup\_internal=0 rw\_err\_dup\_cluster\_key=0 rw\_err\_dup\_send=0 rw\_err\_write\_internal=0 rw\_err\_write\_cluster\_key=0 rw\_err\_write\_send=0 rw\_err\_ack\_internal=0 rw\_err\_ack\_nomatch=0 rw\_err\_ack\_badnode=0 client\_connections=2 waiting\_transactions=0 tree\_count=0 record\_refs=2010382 record\_locks=0 migrate\_tx\_objs=0 migrate\_rx\_objs=0 ongoing\_write\_reqs=0 err\_storage\_queue\_full=0 partition\_actual=2728 partition\_replica=2760 partition\_desync=0 partition\_absent=2704 partition\_object\_count=2010382 partition\_ref\_count=8192 system\_free\_mem\_pct=78 sindex\_ucgarbage\_found=0 sindex\_gc\_locktimedout=0 sindex\_gc\_inactivity\_dur=758318989 sindex\_gc\_activity\_dur=1205011 sindex\_gc\_list\_creation\_time=1200847 sindex\_gc\_list\_deletion\_time=1035 sindex\_gc\_objects\_validated=721678292 sindex\_gc\_garbage\_found=347617 sindex\_gc\_garbage\_cleaned=347617 system\_swapping=false err\_replica\_null\_node=0 err\_replica\_non\_null\_node=0 err\_sync\_copy\_null\_node=0 err\_sync\_copy\_null\_master=0 storage\_defrag\_corrupt\_record=0 err\_write\_fail\_prole\_unknown=0 err\_write\_fail\_prole\_generation=0 err\_write\_fail\_unknown=0 err\_write\_fail\_key\_exists=0 err\_write\_fail\_generation=0 err\_write\_fail\_generation\_xdr=0 err\_write\_fail\_bin\_exists=0 err\_write\_fail\_parameter=0 err\_write\_fail\_incompatible\_type=0 err\_write\_fail\_noxdr=0 err\_write\_fail\_prole\_delete=0 err\_write\_fail\_not\_found=0 err\_write\_fail\_key\_mismatch=0 err\_write\_fail\_record\_too\_big=0 err\_write\_fail\_bin\_name=0 err\_write\_fail\_bin\_not\_found=0 err\_write\_fail\_forbidden=0 stat\_duplicate\_operation=0 uptime=761715 stat\_write\_errs\_notfound=1 stat\_write\_errs\_other=0 heartbeat\_received\_self=5051021 heartbeat\_received\_foreign=9711102 query\_reqs=25 query\_success=15 query\_fail=10 query\_abort=0 query\_avg\_rec\_count=0 query\_short\_queue\_full=0 query\_long\_queue\_full=0 query\_short\_running=15 query\_long\_running=0 query\_tracked=0 query\_agg=0 query\_agg\_success=0 query\_agg\_err=0 query\_agg\_abort=0 query\_agg\_avg\_rec\_count=0 query\_lookups=15 query\_lookup\_success=15 query\_lookup\_err=0 query\_lookup\_abort=0 query\_lookup\_avg\_rec\_count=0

* * *

ASINFO on node 2:

cluster\_size=3 cluster\_key=A5A28076E110C33B cluster\_integrity=true objects=1984282 sub-records=0 total-bytes-disk=0 used-bytes-disk=0 free-pct-disk=0 total-bytes-memory=8589934592 used-bytes-memory=278934760 data-used-bytes-memory=125643464 index-used-bytes-memory=126994048 sindex-used-bytes-memory=26297248 free-pct-memory=96 stat\_read\_reqs=120 stat\_read\_reqs\_xdr=0 stat\_read\_success=105 stat\_read\_errs\_notfound=15 stat\_read\_errs\_other=0 stat\_write\_reqs=1492600 stat\_write\_reqs\_xdr=0 stat\_write\_success=1492595 stat\_write\_errs=5 stat\_xdr\_pipe\_writes=0 stat\_xdr\_pipe\_miss=0 stat\_delete\_success=44 stat\_rw\_timeout=0 udf\_read\_reqs=4 udf\_read\_success=2 udf\_read\_errs\_other=2 udf\_write\_reqs=14 udf\_write\_success=14 udf\_write\_err\_others=0 udf\_delete\_reqs=0 udf\_delete\_success=0 udf\_delete\_err\_others=0 udf\_lua\_errs=0 udf\_scan\_rec\_reqs=4 udf\_query\_rec\_reqs=4 udf\_replica\_writes=0 stat\_proxy\_reqs=0 stat\_proxy\_reqs\_xdr=0 stat\_proxy\_success=0 stat\_proxy\_errs=0 stat\_ldt\_proxy=0 stat\_cluster\_key\_trans\_to\_proxy\_retry=0 stat\_cluster\_key\_transaction\_reenqueue=0 stat\_slow\_trans\_queue\_push=0 stat\_slow\_trans\_queue\_pop=0 stat\_slow\_trans\_queue\_batch\_pop=0 stat\_cluster\_key\_regular\_processed=0 stat\_cluster\_key\_prole\_retry=0 stat\_cluster\_key\_err\_ack\_dup\_trans\_reenqueue=0 stat\_cluster\_key\_partition\_transaction\_queue\_count=0 stat\_cluster\_key\_err\_ack\_rw\_trans\_reenqueue=0 stat\_expired\_objects=0 stat\_evicted\_objects=0 stat\_deleted\_set\_objects=0 stat\_evicted\_set\_objects=0 stat\_evicted\_objects\_time=0 stat\_zero\_bin\_records=0 stat\_nsup\_deletes\_not\_shipped=0 err\_tsvc\_requests=5 err\_out\_of\_space=0 err\_duplicate\_proxy\_request=0 err\_rw\_request\_not\_found=0 err\_rw\_pending\_limit=0 err\_rw\_cant\_put\_unique=0 fabric\_msgs\_sent=12659592 fabric\_msgs\_rcvd=12659550 paxos\_principal=BB9CF35A5565000 migrate\_msgs\_sent=9605908 migrate\_msgs\_recv=9659115 migrate\_progress\_send=0 migrate\_progress\_recv=0 migrate\_num\_incoming\_accepted=24452 migrate\_num\_incoming\_refused=0 queue=0 transactions=2889093 reaped\_fds=94 tscan\_initiate=781 tscan\_pending=0 tscan\_succeeded=789 tscan\_aborted=0 batch\_initiate=0 batch\_queue=0 batch\_tree\_count=0 batch\_timeout=0 batch\_errors=0 info\_queue=0 delete\_queue=0 proxy\_in\_progress=0 proxy\_initiate=0 proxy\_action=0 proxy\_retry=0 proxy\_retry\_q\_full=0 proxy\_unproxy=0 proxy\_retry\_same\_dest=0 proxy\_retry\_new\_dest=0 write\_master=1492600 write\_prole=1507713 read\_dup\_prole=0 rw\_err\_dup\_internal=0 rw\_err\_dup\_cluster\_key=0 rw\_err\_dup\_send=0 rw\_err\_write\_internal=0 rw\_err\_write\_cluster\_key=0 rw\_err\_write\_send=0 rw\_err\_ack\_internal=0 rw\_err\_ack\_nomatch=0 rw\_err\_ack\_badnode=0 client\_connections=4 waiting\_transactions=0 tree\_count=0 record\_refs=1984282 record\_locks=0 migrate\_tx\_objs=0 migrate\_rx\_objs=0 ongoing\_write\_reqs=0 err\_storage\_queue\_full=0 partition\_actual=2708 partition\_replica=2712 partition\_desync=0 partition\_absent=2772 partition\_object\_count=1984282 partition\_ref\_count=8192 system\_free\_mem\_pct=71 sindex\_ucgarbage\_found=0 sindex\_gc\_locktimedout=0 sindex\_gc\_inactivity\_dur=2129846129 sindex\_gc\_activity\_dur=3499873 sindex\_gc\_list\_creation\_time=3489177 sindex\_gc\_list\_deletion\_time=2072 sindex\_gc\_objects\_validated=2053448283 sindex\_gc\_garbage\_found=696100 sindex\_gc\_garbage\_cleaned=696100 system\_swapping=false err\_replica\_null\_node=0 err\_replica\_non\_null\_node=0 err\_sync\_copy\_null\_node=0 err\_sync\_copy\_null\_master=0 storage\_defrag\_corrupt\_record=0 err\_write\_fail\_prole\_unknown=0 err\_write\_fail\_prole\_generation=0 err\_write\_fail\_unknown=0 err\_write\_fail\_key\_exists=4 err\_write\_fail\_generation=0 err\_write\_fail\_generation\_xdr=0 err\_write\_fail\_bin\_exists=0 err\_write\_fail\_parameter=0 err\_write\_fail\_incompatible\_type=0 err\_write\_fail\_noxdr=0 err\_write\_fail\_prole\_delete=0 err\_write\_fail\_not\_found=0 err\_write\_fail\_key\_mismatch=0 err\_write\_fail\_record\_too\_big=0 err\_write\_fail\_bin\_name=0 err\_write\_fail\_bin\_not\_found=0 err\_write\_fail\_forbidden=0 stat\_duplicate\_operation=0 uptime=2152798 stat\_write\_errs\_notfound=1 stat\_write\_errs\_other=4 heartbeat\_received\_self=14275817 heartbeat\_received\_foreign=18934583 query\_reqs=101 query\_success=60 query\_fail=31 query\_abort=10 query\_avg\_rec\_count=1259 query\_short\_queue\_full=0 query\_long\_queue\_full=0 query\_short\_running=46 query\_long\_running=24 query\_tracked=14 query\_agg=25 query\_agg\_success=15 query\_agg\_err=0 query\_agg\_abort=10 query\_agg\_avg\_rec\_count=3523 query\_lookups=45 query\_lookup\_success=45 query\_lookup\_err=0 query\_lookup\_abort=0 query\_lookup\_avg\_rec\_count=0

* * *

ASINFO on node 3:

cluster\_size=3 cluster\_key=A5A28076E110C33B cluster\_integrity=true objects=2005344 sub-records=0 total-bytes-disk=0 used-bytes-disk=0 free-pct-disk=0 total-bytes-memory=8589934592 used-bytes-memory=277329629 data-used-bytes-memory=127016768 index-used-bytes-memory=128342016 sindex-used-bytes-memory=21970845 free-pct-memory=96 stat\_read\_reqs=0 stat\_read\_reqs\_xdr=0 stat\_read\_success=0 stat\_read\_errs\_notfound=0 stat\_read\_errs\_other=0 stat\_write\_reqs=0 stat\_write\_reqs\_xdr=0 stat\_write\_success=0 stat\_write\_errs=0 stat\_xdr\_pipe\_writes=0 stat\_xdr\_pipe\_miss=0 stat\_delete\_success=0 stat\_rw\_timeout=0 udf\_read\_reqs=0 udf\_read\_success=0 udf\_read\_errs\_other=0 udf\_write\_reqs=0 udf\_write\_success=0 udf\_write\_err\_others=0 udf\_delete\_reqs=0 udf\_delete\_success=0 udf\_delete\_err\_others=0 udf\_lua\_errs=0 udf\_scan\_rec\_reqs=0 udf\_query\_rec\_reqs=0 udf\_replica\_writes=0 stat\_proxy\_reqs=0 stat\_proxy\_reqs\_xdr=0 stat\_proxy\_success=0 stat\_proxy\_errs=0 stat\_ldt\_proxy=0 stat\_cluster\_key\_trans\_to\_proxy\_retry=0 stat\_cluster\_key\_transaction\_reenqueue=0 stat\_slow\_trans\_queue\_push=0 stat\_slow\_trans\_queue\_pop=0 stat\_slow\_trans\_queue\_batch\_pop=0 stat\_cluster\_key\_regular\_processed=0 stat\_cluster\_key\_prole\_retry=0 stat\_cluster\_key\_err\_ack\_dup\_trans\_reenqueue=0 stat\_cluster\_key\_partition\_transaction\_queue\_count=0 stat\_cluster\_key\_err\_ack\_rw\_trans\_reenqueue=0 stat\_expired\_objects=0 stat\_evicted\_objects=0 stat\_deleted\_set\_objects=0 stat\_evicted\_set\_objects=0 stat\_evicted\_objects\_time=0 stat\_zero\_bin\_records=0 stat\_nsup\_deletes\_not\_shipped=0 err\_tsvc\_requests=0 err\_out\_of\_space=0 err\_duplicate\_proxy\_request=0 err\_rw\_request\_not\_found=0 err\_rw\_pending\_limit=0 err\_rw\_cant\_put\_unique=0 fabric\_msgs\_sent=3042378 fabric\_msgs\_rcvd=3042365 paxos\_principal=BB9CF35A5565000 migrate\_msgs\_sent=3028681 migrate\_msgs\_recv=3042350 migrate\_progress\_send=0 migrate\_progress\_recv=0 migrate\_num\_incoming\_accepted=5476 migrate\_num\_incoming\_refused=0 queue=0 transactions=56866 reaped\_fds=0 tscan\_initiate=18 tscan\_pending=0 tscan\_succeeded=18 tscan\_aborted=0 batch\_initiate=0 batch\_queue=0 batch\_tree\_count=0 batch\_timeout=0 batch\_errors=0 info\_queue=0 delete\_queue=0 proxy\_in\_progress=0 proxy\_initiate=0 proxy\_action=0 proxy\_retry=0 proxy\_retry\_q\_full=0 proxy\_unproxy=0 proxy\_retry\_same\_dest=0 proxy\_retry\_new\_dest=0 write\_master=0 write\_prole=0 read\_dup\_prole=0 rw\_err\_dup\_internal=0 rw\_err\_dup\_cluster\_key=0 rw\_err\_dup\_send=0 rw\_err\_write\_internal=0 rw\_err\_write\_cluster\_key=0 rw\_err\_write\_send=0 rw\_err\_ack\_internal=0 rw\_err\_ack\_nomatch=0 rw\_err\_ack\_badnode=0 client\_connections=3 waiting\_transactions=0 tree\_count=0 record\_refs=2005344 record\_locks=0 migrate\_tx\_objs=0 migrate\_rx\_objs=0 ongoing\_write\_reqs=0 err\_storage\_queue\_full=0 partition\_actual=2756 partition\_replica=2720 partition\_desync=0 partition\_absent=2716 partition\_object\_count=2005344 partition\_ref\_count=8192 system\_free\_mem\_pct=80 sindex\_ucgarbage\_found=0 sindex\_gc\_locktimedout=0 sindex\_gc\_inactivity\_dur=10808110 sindex\_gc\_activity\_dur=15890 sindex\_gc\_list\_creation\_time=15866 sindex\_gc\_list\_deletion\_time=2 sindex\_gc\_objects\_validated=9360170 sindex\_gc\_garbage\_found=0 sindex\_gc\_garbage\_cleaned=0 system\_swapping=false err\_replica\_null\_node=0 err\_replica\_non\_null\_node=0 err\_sync\_copy\_null\_node=0 err\_sync\_copy\_null\_master=0 storage\_defrag\_corrupt\_record=0 err\_write\_fail\_prole\_unknown=0 err\_write\_fail\_prole\_generation=0 err\_write\_fail\_unknown=0 err\_write\_fail\_key\_exists=0 err\_write\_fail\_generation=0 err\_write\_fail\_generation\_xdr=0 err\_write\_fail\_bin\_exists=0 err\_write\_fail\_parameter=0 err\_write\_fail\_incompatible\_type=0 err\_write\_fail\_noxdr=0 err\_write\_fail\_prole\_delete=0 err\_write\_fail\_not\_found=0 err\_write\_fail\_key\_mismatch=0 err\_write\_fail\_record\_too\_big=0 err\_write\_fail\_bin\_name=0 err\_write\_fail\_bin\_not\_found=0 err\_write\_fail\_forbidden=0 stat\_duplicate\_operation=0 uptime=10861 stat\_write\_errs\_notfound=0 stat\_write\_errs\_other=0 heartbeat\_received\_self=72117 heartbeat\_received\_foreign=144224 query\_reqs=0 query\_success=0 query\_fail=0 query\_abort=0 query\_avg\_rec\_count=0 query\_short\_queue\_full=0 query\_long\_queue\_full=0 query\_short\_running=0 query\_long\_running=0 query\_tracked=0 query\_agg=0 query\_agg\_success=0 query\_agg\_err=0 query\_agg\_abort=0 query\_agg\_avg\_rec\_count=0 query\_lookups=0 query\_lookup\_success=0 query\_lookup\_err=0 query\_lookup\_abort=0 query\_lookup\_avg\_rec\_count=0

Thanks again! Moreno

---

<div class="post-metadata">

**Author:** ![pratyyy](https://avatars.discourse-cdn.com/v4/letter/p/5f9b8f/32.png) [@pratyyy](https://discuss.aerospike.com/u/pratyyy)\
**Post date:** [May 15, 2015, 10:27am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/7 "2015-05-15T10:27:31Z")

</div>

Hi Moreno,

Since you have lot of data in the namespace, there is a possibility that you might have inserted some records in this set earlier. Scanning different nodes should not change the result.

Lets investigate this further. Can you please take backup of set in question ? You can use the following command-

asbackup -d \<dir\_name\> -n namespace\_name -s set\_name -h host\_name

Also can you share the result of following command on each host ?

asinfo -v “sets”

Thanks

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 15, 2015, 12:18pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/8 "2015-05-15T12:18:47Z")

</div>

asbackup fails on 3 nodes, output is:

asbackup -d /MORENO\_91/BACKUP -n test -s users -h 10.100.0.2

Backing up From: host 10.100.0.2 port 3000 namespace test set users bin\_list (null) to directory /MORENO\_91/BACKUP with scan\_pct 100 Nodes are repeated or different addresses of same node. Give proper input

* * *

asinfo -v “sets” node1:

ns\_name=test:set\_name=demo2:n\_objects=1340058:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default:set-dele te=false;ns\_name=test:set\_name=ccc:n\_objects=670322:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default:se t-delete=false;ns\_name=test:set\_name=tweets:n\_objects=1:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-defaul t:set-delete=false;ns\_name=test:set\_name=users:n\_objects=1:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-def ault:set-delete=false;

* * *

asinfo -v “sets” node2:

ns\_name=test:set\_name=ccc:n\_objects=661095:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default:set-delete= false;ns\_name=test:set\_name=demo2:n\_objects=1323181:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default:se t-delete=false;ns\_name=test:set\_name=users:n\_objects=3:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default :set-delete=false;ns\_name=test:set\_name=tweets:n\_objects=3:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-def ault:set-delete=false;

* * *

asinfo -v “sets” node3:

ns\_name=test:set\_name=demo2:n\_objects=1336759:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default:set-dele te=false;ns\_name=test:set\_name=ccc:n\_objects=668581:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-default:se t-delete=false;ns\_name=test:set\_name=tweets:n\_objects=2:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-defaul t:set-delete=false;ns\_name=test:set\_name=users:n\_objects=2:set-stop-write-count=0:set-evict-hwm-count=0:set-enable-xdr=use-def ault:set-delete=false;

---

<div class="post-metadata">

**Author:** ![pratyyy](https://avatars.discourse-cdn.com/v4/letter/p/5f9b8f/32.png) [@pratyyy](https://discuss.aerospike.com/u/pratyyy)\
**Post date:** [May 15, 2015, 1:37pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/9 "2015-05-15T13:37:46Z")

</div>

Hi Moreno,

Looking at the backup failure, it seems service list of the nodes is corrupted somehow at some node. This might trigger a scan twice at some node.

To confirm this -

Can you share the output of following command from each node with us ?

asinfo -v “services”

Also to be 100 % sure, can you share the config file of each node with us ?

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 15, 2015, 3:01pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/10 "2015-05-15T15:01:48Z")

</div>

Hi pratyyy,

* * *

Node 1

ip a

eth0: 138.132.43.90/24 eth1: 10.100.0.2/24 eth1.1: 20.100.0.2/24

asinfo -v “services”

10.100.0.3:3000;20.100.0.3:3000;138.132.43.91:3000;10.100.0.4:3000;138.132.43.92:3000

* * *

Node 2

ip a eth0: 138.132.43.91/24 eth1: 10.100.0.3/24 eth1.1: 20.100.0.3/24

asinfo -v “services”

10.100.0.2:3000;20.100.0.2:3000;138.132.43.90:3000;10.100.0.4:3000;138.132.43.92:3000

* * *

Node 3

ip a

eth0: 138.132.43.92/24 eth1: 10.100.0.4/24 (no eth1.1)

asinfo -v “services”

10.100.0.3:3000;20.100.0.3:3000;138.132.43.91:3000;10.100.0.2:3000;20.100.0.2:3000;138.132.43.90:3000

* * *

/etc/aerospike/aerospike.conf (is the same, identical, on the 3 nodes)

(sorry, copy and paste of the file is not easily readable)

service { user root group root paxos-single-replica-limit 1 pidfile /var/run/aerospike/asd.pid service-threads 4 transaction-queues 4 transaction-threads-per-queue 4 proto-fd-max 15000 }

logging { file /var/log/aerospike/aerospike.log { context any info } }

network { service { address any port 3000 }

```
    heartbeat {
            mode multicast
            address 239.1.99.222
            port 9918
            interval 150
            timeout 10
    }

    fabric {
            port 3001
    }

    info {
            port 3003
    }

```

}

namespace test { replication-factor 2 memory-size 4G default-ttl 30d storage-engine memory }

namespace bar { replication-factor 2 memory-size 4G default-ttl 30d storage-engine memory }

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 18, 2015, 9:11am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/11 "2015-05-18T09:11:03Z")

</div>

Hi pratyyy, do you have any news?

Do you need other info?

Thanks Moreno

---

<div class="post-metadata">

**Author:** ![jyoti](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jyoti/32/240_2.png) [@jyoti](https://discuss.aerospike.com/u/jyoti)\
**Post date:** [May 18, 2015, 10:53am UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/12 "2015-05-18T10:53:32Z")

</div>

Hi Moreno, Extremely sorry for late reply. From your as-info result we come to know that you have multiple nic addresses of same machine. Which creates same node-id for multiple nics. We will enhance our scan api to handle duplicate node ids efficiently.

You can resolve your issue by specifying external address of each node in service section of your config file. e.g.

```
service {
    address any
    port 3000
    access-address 10.100.0.03
}

```

By adding access-address, each node will expose only one ip for client connection.

Let us know whether above solution is working for you or you need more help?

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 18, 2015, 12:07pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/13 "2015-05-18T12:07:27Z")

</div>

Thank you jyoti (late reply? You and pratyyy are so collaborative with me :-)).

I changed aerospike.conf on 3 nodes according your suggestion.

Node 1:

Added row:

access-address 10.100.0.2

Node2:

Added row:

access-address 10.100.0.3

Node3:

Added row:

access-address 10.100.0.4

The output of asinfo -v “services” seems to be right:

Node1:

10.100.0.3:3000;10.100.0.4:3000

Node2:

10.100.0.2:3000;10.100.0.4:3000

Node3:

10.100.0.3:3000;10.100.0.2:3000

I restarted aerospike service in this way:

service aerospike stop

service aerospike start

Bad new is that now I see only 1 record for “Users” SET and 1 for “Tweets” SET.

Something wrong in this sequence of commands and aerospike.conf update?

Thanks again Moreno

---

<div class="post-metadata">

**Author:** ![jyoti](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jyoti/32/240_2.png) [@jyoti](https://discuss.aerospike.com/u/jyoti)\
**Post date:** [May 18, 2015, 12:10pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/14 "2015-05-18T12:10:58Z")

</div>

Can you run as backup now. And just give us the backup output.

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 18, 2015, 12:30pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/15 "2015-05-18T12:30:05Z")

</div>

All 3 backups were ok.

Output for Node1 (for example):

asbackup -d /MORENO\_90/BACKUP -n test -s users

Backing up From: host 127.0.0.1 port 3000 namespace test set users bin\_list (null) to directory /MORENO\_90/BACKUP with scan\_pct 100

Aerospike scan nodes: 3 nodes

Node\_name Objects Rep\_fact

BB96753A5565000 469350 2

BB9CF35A5565000 293504 2

BB9B1ECA5565000 329158 2

directory “/MORENO\_90/BACKUP” prepared for backup

starting backup for node BB96753A5565000

starting backup for node BB9CF35A5565000

starting backup for node BB9B1ECA5565000

May 18 2015 12:22:11 GMT: New file created /MORENO\_90/BACKUP/BB9CF35A5565000\_00000.asb

May 18 2015 12:22:11 GMT: New file created /MORENO\_90/BACKUP/BB96753A5565000\_00000.asb

May 18 2015 12:22:11 GMT: New file created /MORENO\_90/BACKUP/BB9B1ECA5565000\_00000.asb

Complete backup for node BB9B1ECA5565000 and total backed up from this node: 0

Complete backup for node BB96753A5565000 and total backed up from this node: 0

Complete backup for node BB9CF35A5565000 and total backed up from this node: 1

May 18 2015 12:22:12 GMT: backed up records 0%

May 18 2015 12:22:12 GMT: backed up records 100%

May 18 2015 12:22:12 GMT: Backup successfully completed.

May 18 2015 12:22:12 GMT: Total backed up records from all nodes 1

---

<div class="post-metadata">

**Author:** ![jyoti](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jyoti/32/240_2.png) [@jyoti](https://discuss.aerospike.com/u/jyoti)\
**Post date:** [May 18, 2015, 12:40pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/16 "2015-05-18T12:40:45Z")

</div>

From asbackup output it seems you have only one record in users set. Are you sure there are 3 records in users set? You can confirm this using following command in each node:

asinfo -v “sets”

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 18, 2015, 1:18pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/17 "2015-05-18T13:18:37Z")

</div>

asinfo -v “sets” reported 1 record for set\_name=Users

I didn’t deleted the other 2 users (and the other 2 tweets).

So I stopped Aerospike, replaced the right aerospike.conf with older version (where access-address was not specified).

After restarting it I had duplicated records.

When I stopped again Aerospike, replaced aerospike.conf with the right version (with “access-address” row) and restarted Aerospike, ALL Users and Tweets records were disappeared!

(asinfo -v “sets” has now empty output)

I don’y know if this last test is an interesting or a stupid test: perhaps the best thing is to delete all previous records and with the correct aerospike.conf restart from the beginning.

---

<div class="post-metadata">

**Author:** ![jyoti](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jyoti/32/240_2.png) [@jyoti](https://discuss.aerospike.com/u/jyoti)\
**Post date:** [May 18, 2015, 1:42pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/18 "2015-05-18T13:42:21Z")

</div>

Actually your data is in-memory thats why you lost your records. Sorry I could have explain you the steps in proper way. Whenever you do node restart do one by one and wait for migration to finish.

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 18, 2015, 1:51pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/19 "2015-05-18T13:51:43Z")

</div>

Ok jyoti.

No problem for data loss.

I’m only doing preliminary tests with Aerospike (the ICT company where I work is evaluating if it’s possible to use a NoSql DB, in cloud, in order to store real time application’s data).

---

<div class="post-metadata">

**Author:** ![jyoti](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/jyoti/32/240_2.png) [@jyoti](https://discuss.aerospike.com/u/jyoti)\
**Post date:** [May 18, 2015, 2:11pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/20 "2015-05-18T14:11:05Z")

</div>

We are able reproduce this issue when multiple NICs are there. It seems this is due to some client bug. We are fixing this. Thanks for the patience. Looking forward to have more queries from you.

Thanks, Jyoti

---

<div class="post-metadata">

**Author:** ![moreno](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@moreno](https://discuss.aerospike.com/u/moreno)\
**Post date:** [May 18, 2015, 2:22pm UTC](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274/21 "2015-05-18T14:22:24Z")

</div>

Ok jyoti.

Thanks for your patience!

After preliminary tests I’ll switch to stress test (with related questions 🙂 )

[Next page](https://discuss.aerospike.com/t/scan-and-duplicated-records-aer-3648/1274.md?page=2)