Help please related to java plugin eclipse, Executor class under aerospike.client.command package, waitTillComplete method


#1

Hi,

My Environment:

  • I am using java Aerospike-core-plugin-3.1.7 for development in eclipse
  • I am using a vangrnt box to run my server on the local host

What am I trying to do

I am trying to read all the records in a set using the scanAll() method of the AerospikeClient class

Questions

I havesome information and a few questions to ask. Please help

  1. I dont find the API docs for 3.1.7 on the net to relate to a few classes which I dont see in the latest api java docs. Ex : I dont see a package aerospike.client.command in the latest API and so I dont see an Executor class in this package
  2. So if I have to match the latest docs, I have to update my plugin to the latest one i,e 3.2.3 but I am not able to find it on the net and I am not able to update the existing plugin which I got from here
  3. So, pushing all the version issues aside, the execution control waits indefinitely in the waitTillComplete() method which is called from the execute() method of the Executor class and further this execute() method is called from the scanAll() method of AerospikeClient class. Please help me to find out why is my control waiting indefinitely.

Thanks


#2
  1. Executor is an internal class. Internal classes are not documented in the public API.

  2. The plugin version 3.1.7 is not the same as the client version 3.2.3. The latest plugin version is 3.1.7.

  3. What is your ScanPolicy.maxConcurrentNodes value?


#3

Oh ok

Thanks Brian and my maxCurrentNodes =0


#4

and it just started working now

I just restarted my system

but what could have been the problem

please help me understand this

cheers


#5

maxCurrentNodes=0 should work.


#6

okay

Thanks.

but it all started working fine after I restarted my system.

and now I am trying to scan all the records in the name space to determine “How many pages were hit a given number of times”

Input data file: This is a file that has hourly data for Wikipedia hits 200MB with 4 bins(1st bin is the language, 2nd bin is the title of the page retrieved, 3rd bin is the number of requests(hit count), 4th bin is the size of the content returned)

To do this:

  1. Inserted the records into the AS cluster by creating sets based on bin 1 i.e each set is dedicated for a language
  2. For each distinct page_title(2nd bin): filter all the records with this page title and then aggregate the 3rd bin to get the total hit count by passing the Result set stream to a simple Map: Reduce function for count

But this seems to take a lot of time to execute:

To be more specific with the algorithm:

  1. // Parse all the set_names into a HashSet

    	sets = parseForSets(answer);
    
  2. // for each set scan all the records

     	for (int index = 0; index < sets.size(); index++) {
     		client.scanAll(policy, "test", sets.get(index), this);
     	}
    
  3. // For each record scanned in a specific set

       @Override
       public void scanCallback(Key key, Record record) throws AerospikeException {
     // TODO Auto-generated method stub
    
     // Query for the total count of page_hits(3rd Bin) FROM ALL THE SETS i.e ENTIRE NAMESPACE if the page_title(2nd Bin) of 
     // the scanned record is not present in a local HashSet
     
     boolean added = pageTitlesSet.add(record.getString("page_title"));
     if (added == true) {
     	getCount(sets, clienT, record.getString("page_title"));
     }
    

    }

Looking at all the above, I have the following questions

  1. could the algorithm be improved in its design in any way?
  2. Is there a way to read/ filter on a specific bin value all the records in a name space without mentioning a set_name?

Guidance please

Regards


#7

To scan all sets in a namespace, just pass null for the set argument.


#8

But when I use "null " no records are scanned and the control skips to finally block

                        client.scanAll(policy, "test", null, this);

		} finally {
			client.close();
		}

The control doesnt enter into the scanCallback method!

As I debug, I observe that, false is returned from the

parseRecordResults(int receiveSize)

// If this is the end marker of the response, do not proceed further
			if ((info3 & Command.INFO3_LAST) == Command.INFO3_LAST) {
				return false;
			}

The values at the instance:

info3 = 1

Command.INFO3_LAST = 1


#9

Oh! I think all the records expired

because it doesnt show any result when I try to use “SHOW SETS” on an aql interface

Now I am inserting the records again and the “SHOW SETS” produces the expected table

BUT I used a “null” write policy while I am inserting(cient.put(…)) each record and I did not change any of the server defaults.

what could have happened

please throw some light


#10

Hi Brian,

I came back to the same problem again!

The control waits in the “waitTillComplete” method infinitely and this time the system restart doesnt work and the maxCurrentNodes = 0

any suggestions? please


#11

Some recommendation to help you with what to check -

  • Have you run the examples/src/com/aerospike/examples/AsyncScan.java?
  • Is that successful consistently?
  • Does your code follow the example?
  • Does it capture success/failure and do a notifyComplete()?

Good luck!


#12

I understand this with an AsyncClient but I am trying to do it with a syncClient

am I suppose to use the AsyncClient?

could you please advise me on when to use the two different clients

regards


#13

I am not able to run the AsyncScan.java!!!


#14

AsyncScan isn’t really relevant here. I suggest setting “ScanPolicy.concurrentNodes = false” for your synchronous scan.


#15

You do not need to run the AsyncClient. Best for you to stay using the syncClient.

Maybe you want to send your source code, and we can spot if there is anything incorrect with the source code.


#16

okay Brian,

but now the control doesn’t proceed beyond the read

while (pos < length) {
			int count = bis.read(dataBuffer, pos, length - pos);

#17

Please find the code below:

		// TODO Auto-generated method stub
		try {
			ScanPolicy policy = new ScanPolicy();
			policy.concurrentNodes = false;
			policy.priority = Priority.LOW;
			policy.includeBinData = true;
			pageTitlesSet = new HashSet<String>();
			pagesPerCount = new HashMap<Integer, Integer>();
			client.register(null, "udf/first.lua", "first.lua", Language.LUA);
			LuaConfig.SourceDirectory = "udf";
			client.scanAll(policy, "test", null, this);

		} finally {
			client.close();
		}

		System.out.println(pagesPerCount);

and as the class implements ScanCallback

	@Override
	public void scanCallback(Key key, Record record) throws AerospikeException {
		// TODO Auto-generated method stub
		// Query for the total count of page hits if the page_title(2nd Bin) of
		// the scanned record is not present in a local HashSet

		boolean added = pageTitlesSet.add(record.getString("page_title"));
		if (added == true) {
			getCount(sets, clienT, record.getString("page_title"));
		} else {
			System.out.println();
		}

	}

#18

The buferf of bis (BufferedInputStream) has all values equal to 0


#19

The server is not responding to the scan request. Are there any messages regarding the scan in the server log?

Also, it’s not clear why you are opening/closing the client just to run a scan. A single client instance should remain open for the duration of the program.

Do the scan examples in the examples directory work for you?


#20

The server is not responding to the scan request. Are there any messages regarding the scan in the server log?

There are no messages regarding the scan in the server log

Also, it’s not clear why you are opening/closing the client just to run a scan.

I am just trying to scan all the records and in the scanCallBack function I am trying to output the records whose bin 2 starts with a specific sequence of characters

A single client instance should remain open for the duration of the program.

Sure, I am maintaining only a single instance of the client and the program ends after the scanAll, so I close the client at the end

Do the scan examples in the examples directory work for you?

NO!!!, they are stuck just like this program

Please find below a screen shot