Newbie needs help!

1

I have successfully built and installed the Aerospike server and c client and ran a benchmark… everything seems to be working. Please see my steps below. Now I’m wondering what to do next!

Is there a chat room for Aerospike or is this forum as good as it gets? :slight_smile:

I’d like to be able ‘see how Aerospike is doing’ – e.g. how much RAM is it using, how much disk is it using, etc and preferably from the command line – so which tools / utilities can I do this with? I had a look at [1] but it’s not obvious how to build and install these tools. Are they on GitHub? Also, I heard about the Aerospike Management Console but also couldn’t find it on GitHub and why isn’t it listed on [1] too? How to build and instal it?

Why running the benchmark then I noticed that it doesn’t seem to go much faster after using 4 threads. I have 64 cores available. I’ve read that Aerospike can handle 1 M ops/second/box but I’m only getting about 123k/s/box via localhost. What to tune to make the performance go up?

I’m interested in writing a UDF function which calls my C code. Does anybody have any links to useful info / tutorials about that?

Thanks in advance, Simon

[1] Overview | Aerospike Documentation

$ git clone https://github.com/aerospike/aerospike-server.git
$ cd aerospike-server/
$ git submodule update --init
$ sudo yum install openssl openssl-devel openssl-static
$ sudo yum install lua lua-devel lua-static
$ sudo yum install python
$ make -j64

$ make init
$ make start

$ git clone https://github.com/aerospike/aerospike-client-c.git
$ cd aerospike-client-c/
$ sudo yum install openssl-devel glibc-devel autoconf automake libtool
$ sudo yum install lua-devel
$ sudo yum install gcc-c++ graphviz rpm-build
$ ./install_libev
$ git submodule update --init
$ make EVENT_LIB=libev
$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib make EVENT_LIB=libev AS_HOST=127.0.0.1 test

...
SUMMARY
...
185 tests: 185 passed, 0 failed

http://www.aerospike.com/docs/client/c/benchmarks

$ cd benchmarks/
$ make EVENT_LIB=libev
$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib make run
$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib ./target/benchmarks -h 127.0.0.1 -p 3000 -n test -s demoset -k 1000000 -o S:1400 -w RU,80 -z 4 -L 4,3
hosts:          127.0.0.1
port:           3000
user:           (null)
namespace:      test
set:            demoset
startKey:       1
keys/records:   1000000
bins:           1
object spec:    UTF8 string[1400]
random values:  false
workload:       read 80% write 20%
stop after:     0 transactions
threads:        4
max throughput: unlimited
read timeout:   0 ms
write timeout:  0 ms
max retries:    1
debug:          false
latency:        4 columns, shift exponent 3
shared memory:  false
read replica:   master
read consistency level: one
write commit level: all
Conn pools per node: 1
asynchronous mode:  off
2017-07-11 18:19:30 INFO Add node BB948E04A701B06 127.0.0.1:3000
2017-07-11 18:19:30 INFO Read/write using 1000000 records
2017-07-11 18:19:30 INFO Start 4 generator threads
2017-07-11 18:19:31 INFO write(tps=23696 timeouts=0 errors=0) read(tps=95020 timeouts=0 errors=0) total(tps=118716 timeouts=0 errors=0)
      <=1ms >1ms >8ms >64ms
write  100%   0%   0%    0%
read   100%   0%   0%    0%
2017-07-11 18:19:32 INFO write(tps=23895 timeouts=0 errors=0) read(tps=95110 timeouts=0 errors=0) total(tps=119005 timeouts=0 errors=0)
      <=1ms >1ms >8ms >64ms
write  100%   0%   0%    0%
read   100%   0%   0%    0%
2017-07-11 18:19:33 INFO write(tps=24572 timeouts=0 errors=0) read(tps=98757 timeouts=0 errors=0) total(tps=123329 timeouts=0 errors=0)
      <=1ms >1ms >8ms >64ms
write  100%   0%   0%    0%
read   100%   0%   0%    0%
^C
2

Asadm would be the tool to use from the command line.

For long term trends, I would suggest aerospike-collectd plugin.

AMC can be downloaded here:

What version of Aerospike and can you share your configuration?

3

Thanks!

I tried to install asadm but got this error. Any ideas?

$ git clone https://github.com/aerospike/aerospike-admin.git
$ cd aerospike-admin/
$ sudo ./asadm-deps/install.sh 
./asadm-deps/install.sh: line 50: cd: /home/ec2-user/simon/as/aerospike-admin/asadm-deps/UNKNOWN: No such file or directory

Regarding the config then I haven’t change it yet. It’s the one on GitHub. The config shown upon start is:

$ make start
Running the Aerospike Server locally...
./target/Linux-x86_64/bin/asd --config-file as/etc/aerospike_dev.conf
Jul 11 2017 21:50:52 GMT: INFO (as): (as.c:319) <><><><><><><><><><>  Aerospike Community Edition build 3.14.1.1  <><><><><><><><><><>
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) # Aerospike database developer configuration file.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) service {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   run-as-daemon false # To work with gdb, and make console logging visible.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   # The number of concurrent connections to the database is limited by
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   # proto-fd-max, and by the system's maximum number of open file descriptors.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   # See "man limits.conf" for how to set the system's "nofile" limit.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   proto-fd-max 1024
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   work-directory run/work
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   pidfile run/asd.pid
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) mod-lua {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   user-path run/work/usr/udf/lua
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   system-path run/work/sys/udf/lua
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) logging {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   # Log file must be an absolute path.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   file run/log/aerospike.log {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           context any info
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   console {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           context any info
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) network {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   service {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           address any
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           port 3000
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   heartbeat {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           mode multicast
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           multicast-group 239.1.99.222
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           port 9918
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           # To use unicast-mesh heartbeats, remove the 3 lines above, and see
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           # aerospike_mesh.conf for alternative.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           interval 150
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           timeout 10
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   fabric {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           port 3001
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   info {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)           port 3003
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) namespace test {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   replication-factor 2
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   memory-size 4G
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   default-ttl 30d # 30 days, use 0 to never expire/evict.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   storage-engine memory
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) namespace bar {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   replication-factor 2
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   memory-size 4G
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   default-ttl 30d # 30 days, use 0 to never expire/evict.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   storage-engine memory
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) 
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   # To use file storage backing, comment out the line above and use the
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539)   # following lines instead.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) # storage-engine device {
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) #         file /opt/aerospike/data/bar.dat
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) #         filesize 16G
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) #         data-in-memory true # Store data in memory in addition to file.
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) # }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3539) }
Jul 11 2017 21:50:52 GMT: INFO (config): (cfg.c:3559) system file descriptor limit: 1024, proto-fd-max: 1024
Jul 11 2017 21:50:52 GMT: INFO (hardware): (hardware.c:1785) detected 64 CPU(s), 32 core(s), 2 NUMA node(s)
Jul 11 2017 21:50:52 GMT: INFO (socket): (socket.c:2556) Node port 3001, node ID bb948e04a701b06
4

For both the server and the tools you have pre-built packages that you can download and install. That would be easier than building from source, unless you are modifying the code of the server or the tools. That’s probably what you want to do.

You have multiple things going on here, most of which have nothing to do with UDF. It would be best if you edited your question, and opened separate topics for the different questions.

Try to install from the packages (RPM most likely, depending on your OS). That should address your initial question.

1 Like
5

Thanks for the quick reply.

I like the idea of building from source especially because I couldn’t find an install guide for Amazon Linux. I have successfully built both server and c client. I wouldn’t want to rule out eventually modifying the code of the server. I really like it that I can just run the server in-place without installing it. I’d like now to find out about the running server and was recommended to install asadm but the install steps on the GitHub page do not work for me. Any ideas how to get it to work? I also just created an issue ticket on the GitHub page. Maybe that’s a better place to ask?

6

It’s on our site (and I was hoping you got those links by emails. Len should have updated you.)

Of course we have a deployment guide for Amazon EC2. It includes how to start with Amazon CloudFormation, but also how to install manually on Amazon Linux.

The dependencies for the aerospike-admin tool are:

python 2.6+ (< 3) and the python modules:

  • ply: >= 3.4
  • pexpect: >= 3.0
  • pyOpenSSL: 16.2.0

Standard open source procedure - if the dependency installation script isn’t working open an issue in the aerospike/aerospike-admin repo on Github, and install them by hand. Make sure to mention which version of Python you have, and which version of those modules in your issue. Otherwise there’s an RPM for that.

7

Thanks for the quick reply.

Yes, I saw [1] but I have an own AMI that I want to install Aerospike on. So the link I was looking for is the guide on how the Aerospike AMI is created and configured. Does that exist?

Yes, I installed the python modules using: sudo yum install python27-ply python27-pexpect python27-pyOpenSSL

But I still get the following error:

$ sudo ./asadm-deps/install.sh 
./asadm-deps/install.sh: line 50: cd: /home/ec2-user/simon/as/aerospike-admin/asadm-deps/UNKNOWN: No such file or directory

[1] Install on Amazon EC2 | Aerospike Documentation

8

4 posts were split to a new topic: Email stating community members are hiding my posts