Help please related to java plugin eclipse, Executor class under aerospike.client.command package, waitTillComplete method

AJVee · June 28, 2016, 10:19pm

Hi,

My Environment:

I am using java Aerospike-core-plugin-3.1.7 for development in eclipse
I am using a vangrnt box to run my server on the local host

What am I trying to do

I am trying to read all the records in a set using the scanAll() method of the AerospikeClient class

Questions

I havesome information and a few questions to ask. Please help

I dont find the API docs for 3.1.7 on the net to relate to a few classes which I dont see in the latest api java docs. Ex : I dont see a package aerospike.client.command in the latest API and so I dont see an Executor class in this package
So if I have to match the latest docs, I have to update my plugin to the latest one i,e 3.2.3 but I am not able to find it on the net and I am not able to update the existing plugin which I got from here
So, pushing all the version issues aside, the execution control waits indefinitely in the waitTillComplete() method which is called from the execute() method of the Executor class and further this execute() method is called from the scanAll() method of AerospikeClient class. Please help me to find out why is my control waiting indefinitely.

Thanks

Brian · June 28, 2016, 11:37pm

Executor is an internal class. Internal classes are not documented in the public API.
The plugin version 3.1.7 is not the same as the client version 3.2.3. The latest plugin version is 3.1.7.
What is your ScanPolicy.maxConcurrentNodes value?

AJVee · June 29, 2016, 12:42pm

Oh ok

Thanks Brian and my maxCurrentNodes =0

AJVee · June 29, 2016, 1:58pm

and it just started working now

I just restarted my system

but what could have been the problem

please help me understand this

cheers

Brian · June 29, 2016, 4:54pm

maxCurrentNodes=0 should work.

AJVee · July 1, 2016, 8:21pm

okay

Thanks.

but it all started working fine after I restarted my system.

and now I am trying to scan all the records in the name space to determine “How many pages were hit a given number of times”

Input data file: This is a file that has hourly data for Wikipedia hits 200MB with 4 bins(1st bin is the language, 2nd bin is the title of the page retrieved, 3rd bin is the number of requests(hit count), 4th bin is the size of the content returned)

To do this:

Inserted the records into the AS cluster by creating sets based on bin 1 i.e each set is dedicated for a language
For each distinct page_title(2nd bin): filter all the records with this page title and then aggregate the 3rd bin to get the total hit count by passing the Result set stream to a simple Map: Reduce function for count

But this seems to take a lot of time to execute:

To be more specific with the algorithm:

// Parse all the set_names into a HashSet
```
	sets = parseForSets(answer);
```

// for each set scan all the records

 	for (int index = 0; index < sets.size(); index++) {
 		client.scanAll(policy, "test", sets.get(index), this);
 	}

// For each record scanned in a specific set

   @Override
   public void scanCallback(Key key, Record record) throws AerospikeException {
 // TODO Auto-generated method stub

 // Query for the total count of page_hits(3rd Bin) FROM ALL THE SETS i.e ENTIRE NAMESPACE if the page_title(2nd Bin) of 
 // the scanned record is not present in a local HashSet
 
 boolean added = pageTitlesSet.add(record.getString("page_title"));
 if (added == true) {
 	getCount(sets, clienT, record.getString("page_title"));
 }

}

Looking at all the above, I have the following questions

could the algorithm be improved in its design in any way?
Is there a way to read/ filter on a specific bin value all the records in a name space without mentioning a set_name?

Guidance please

Regards

Brian · July 2, 2016, 12:06am

To scan all sets in a namespace, just pass null for the set argument.

AJVee · July 4, 2016, 9:22pm

But when I use "null " no records are scanned and the control skips to finally block

                        client.scanAll(policy, "test", null, this);

		} finally {
			client.close();
		}

The control doesnt enter into the scanCallback method!

As I debug, I observe that, false is returned from the

parseRecordResults(int receiveSize)

// If this is the end marker of the response, do not proceed further
			if ((info3 & Command.INFO3_LAST) == Command.INFO3_LAST) {
				return false;
			}

The values at the instance:

info3 = 1

Command.INFO3_LAST = 1

AJVee · July 5, 2016, 3:47pm

Oh! I think all the records expired

because it doesnt show any result when I try to use “SHOW SETS” on an aql interface

Now I am inserting the records again and the “SHOW SETS” produces the expected table

BUT I used a “null” write policy while I am inserting(cient.put(…)) each record and I did not change any of the server defaults.

what could have happened

please throw some light

AJVee · July 5, 2016, 5:07pm

Hi Brian,

I came back to the same problem again!

The control waits in the “waitTillComplete” method infinitely and this time the system restart doesnt work and the maxCurrentNodes = 0

any suggestions? please

wchu · July 5, 2016, 10:00pm

Some recommendation to help you with what to check -

Have you run the examples/src/com/aerospike/examples/AsyncScan.java?
Is that successful consistently?
Does your code follow the example?
Does it capture success/failure and do a notifyComplete()?

Good luck!

AJVee · July 6, 2016, 6:16pm

I understand this with an AsyncClient but I am trying to do it with a syncClient

am I suppose to use the AsyncClient?

could you please advise me on when to use the two different clients

regards

AJVee · July 6, 2016, 7:32pm

I am not able to run the AsyncScan.java!!!

Brian · July 6, 2016, 7:49pm

AsyncScan isn’t really relevant here. I suggest setting “ScanPolicy.concurrentNodes = false” for your synchronous scan.

wchu · July 6, 2016, 8:39pm

You do not need to run the AsyncClient. Best for you to stay using the syncClient.

Maybe you want to send your source code, and we can spot if there is anything incorrect with the source code.

AJVee · July 6, 2016, 8:41pm

okay Brian,

but now the control doesn’t proceed beyond the read

while (pos < length) {
			int count = bis.read(dataBuffer, pos, length - pos);

AJVee · July 6, 2016, 8:44pm

Please find the code below:

		// TODO Auto-generated method stub
		try {
			ScanPolicy policy = new ScanPolicy();
			policy.concurrentNodes = false;
			policy.priority = Priority.LOW;
			policy.includeBinData = true;
			pageTitlesSet = new HashSet<String>();
			pagesPerCount = new HashMap<Integer, Integer>();
			client.register(null, "udf/first.lua", "first.lua", Language.LUA);
			LuaConfig.SourceDirectory = "udf";
			client.scanAll(policy, "test", null, this);

		} finally {
			client.close();
		}

		System.out.println(pagesPerCount);

and as the class implements ScanCallback

	@Override
	public void scanCallback(Key key, Record record) throws AerospikeException {
		// TODO Auto-generated method stub
		// Query for the total count of page hits if the page_title(2nd Bin) of
		// the scanned record is not present in a local HashSet

		boolean added = pageTitlesSet.add(record.getString("page_title"));
		if (added == true) {
			getCount(sets, clienT, record.getString("page_title"));
		} else {
			System.out.println();
		}

	}

AJVee · July 6, 2016, 9:07pm

The buferf of bis (BufferedInputStream) has all values equal to 0

Brian · July 6, 2016, 9:41pm

The server is not responding to the scan request. Are there any messages regarding the scan in the server log?

Also, it’s not clear why you are opening/closing the client just to run a scan. A single client instance should remain open for the duration of the program.

Do the scan examples in the examples directory work for you?

AJVee · July 6, 2016, 10:20pm

The server is not responding to the scan request. Are there any messages regarding the scan in the server log?

There are no messages regarding the scan in the server log

Also, it’s not clear why you are opening/closing the client just to run a scan.

I am just trying to scan all the records and in the scanCallBack function I am trying to output the records whose bin 2 starts with a specific sequence of characters

A single client instance should remain open for the duration of the program.

Sure, I am maintaining only a single instance of the client and the program ends after the scanAll, so I close the client at the end

Do the scan examples in the examples directory work for you?

NO!!!, they are stuck just like this program

Please find below a screen shot

Topic		Replies	Views
Executor class: waitTillComplete() infinite wait: control doesnt proceed further Java Client	0	1224	July 5, 2016
Aerospike Java Client Release 5.1.9 (November 11, 2021) Releases (Server, Client & Tools)	0	659	November 12, 2021
How to put Aerospike Async Client into wait till all submitted job completed? Java Client udf	1	1493	April 8, 2016
Experiencing pauses in data when running AerospikeClient.scanNode Java Client	5	2028	August 20, 2015
Client.get() timeout after a couple of requests Java Client	2	2159	July 11, 2016

Help please related to java plugin eclipse, Executor class under aerospike.client.command package, waitTillComplete method

Related topics