Sometimes we observe the situation that after adding an index it disappears after some time (few days). The server is not rebooted during this period. We have to add the index again.
What could it be?
Indexes is numeric. We used server v3.5.2 on Centos 6.6. We do not see any SMD, WARNING or any other errors in the logs.
Sequence of events in right: start cluster, create indexes, run workload (~ 30qps on write, schedule by hour read from indexes), after few days indexes missed.
At now - 4 nodes. Initially, there were 3 nodes. They have been updated from 3.4.1 to 3.5.2. And then was immediately added a fourth node. A few days after these actions, the problem arose. Downs nodes or other events did not occur.
In parallel, we have updated our second cluster on the same principle. There were no problems, although in the second cluster we have much more namespaces, indexes and workload.
Also how are you verifying the index has gone missing ??
Sorry, I do not quite understand the question. When requesting to index from client the data did not have (error “Error Code 11: Query failed because cluster is empty.”). AQL “show indexes” showed a lack of indexes.
Related to Error Code 11. Query returns 11 only if it cannot talk to any of nodes in the cluster …
Code Snippet
if (node == null) {
nodes = cluster.getNodes();
if (nodes.length == 0) {
throw new AerospikeException(ResultCode.SERVER_NOT_AVAILABLE, "Query failed because cluster is empty.");
}
}
See if you are able to see the cluster nodes from the node where Application is running
About the indexes going missing just like that suddenly, there is no such known issue. Can you check to make sure if it has not been accidently deleted. You would see following kind of message in the log if such a thing has happend
Mar 12 2015 09:00:34 GMT: INFO (info): (thr_info.c::6363) Secondary index deletion called for ns:test si:ind1
Mar 12 2015 09:00:34 GMT: INFO (info): (thr_info.c::6396) Index deletion request received for test:ind1 via SMD
Migration has no correlation to the index metadata. Whenever cluster is formed the metadata state is picked based on majority state and with index being intact if both the splits are of equal size and one with index and other without index
Should not cause index to go missing. Can you publish .smd files on all the nodes …