Aerospike datastorage filesystem issue

Hi

I have deployed multi-node cluster in centos 7(azure), working fine after some time my storage location not found there is not data in my storage location and showing disk consumption 100% I m not able to realize . I was restart the machine after it was not started kindly find my storage configuration and disk structure

My storage loc:

storage engine device {
    device /dev/sdc1
    write-block-size 128k
}

find the var/log/messages

    0.690274] pci 0000:00:07.3: quirk: [io  0x0400-0x043f] claimed by PIIX4 ACPI
[    0.701737] pci 0000:00:08.0: [1414:5353] type 00 class 0x030000
[    0.702128] pci 0000:00:08.0: reg 0x10: [mem 0xf8000000-0xfbffffff]
[    0.714574] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 10 *11 12 14 15)
[    0.771288] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, disabled.
[    0.820134] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, disabled.
[    0.870262] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, disabled.
[    0.936165] ACPI: Enabled 1 GPEs in block 00 to 0F
[    0.954099] vgaarb: device added: PCI:0000:00:08.0,decodes=io+mem,owns=io+mem,locks=none
[    0.965006] vgaarb: loaded
[    0.984008] vgaarb: bridge control possible 0000:00:08.0
[    0.991152] SCSI subsystem initialized
[    1.000056] ACPI: bus type USB registered
[    1.018054] usbcore: registered new interface driver usbfs
[    1.027024] usbcore: registered new interface driver hub
[    1.038123] usbcore: registered new device driver usb
[    1.046106] PCI: Using ACPI for IRQ routing
[    1.054008] PCI: pci_cache_line_size set to 64 bytes
[    1.055045] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[    1.055049] e820: reserve RAM buffer [mem 0x1fff0000-0x1fffffff]
[    1.055221] NetLabel: Initializing
[    1.061004] NetLabel:  domain hash size = 128
[    1.071005] NetLabel:  protocols = UNLABELED CIPSOv4
[    1.077019] NetLabel:  unlabeled traffic allowed by default
[    1.086135] Switched to clocksource hyperv_clocksource
[    1.100161] pnp: PnP ACPI init
[    1.106236] ACPI: bus type PNP registered
[    1.112520] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
[    1.113048] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[    1.113567] pnp 00:02: Plug and Play ACPI device, IDs PNP0f03 PNP0f13 (active)
[    1.116403] pnp 00:03: [dma 0 disabled]
[    1.116441] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active)
[    1.118297] pnp 00:04: [dma 0 disabled]
[    1.118352] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
[    1.119533] pnp 00:05: [dma 2]
[    1.119575] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)
[    1.119626] system 00:06: [io  0x01e0-0x01ef] has been reserved
[    1.127296] system 00:06: [io  0x0160-0x016f] has been reserved
[    1.149062] system 00:06: [io  0x0278-0x027f] has been reserved
[    1.157779] system 00:06: [io  0x0378-0x037f] has been reserved
[    1.177362] system 00:06: [io  0x0678-0x067f] has been reserved
[    1.187482] system 00:06: [io  0x0778-0x077f] has been reserved
[    1.195949] system 00:06: [io  0x04d0-0x04d1] has been reserved
[    1.203188] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    1.203419] system 00:07: [io  0x0400-0x043f] has been reserved
[    1.213224] system 00:07: [io  0x0370-0x0371] has been reserved
[    1.220925] system 00:07: [io  0x0440-0x044f] has been reserved
[    1.229872] system 00:07: [mem 0xfec00000-0xfec00fff] could not be reserved
[    1.239150] system 00:07: [mem 0xfee00000-0xfee00fff] has been reserved
[    1.247404] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    1.247635] system 00:08: [mem 0x00000000-0x0009ffff] could not be reserved
[    1.258215] system 00:08: [mem 0x000c0000-0x000dffff] could not be reserved
[    1.272497] system 00:08: [mem 0x000e0000-0x000fffff] could not be reserved
[    1.280503] system 00:08: [mem 0x00100000-0x1fffffff] could not be reserved
[    1.288680] system 00:08: [mem 0xfffc0000-0xffffffff] has been reserved
[    1.296399] system 00:08: Plug and Play ACPI device, IDs PNP0c01 (active)
[    1.296687] pnp: PnP ACPI: found 9 devices
[    1.302912] ACPI: bus type PNP unregistered
[    1.328378] pci_bus 0000:00: resource 4 [mem 0xfe0000000-0xfffefffff window]
[    1.328384] pci_bus 0000:00: resource 5 [io  0x0000-0x0cf7 window]
[    1.328387] pci_bus 0000:00: resource 6 [io  0x0d00-0xffff window]
[    1.328389] pci_bus 0000:00: resource 7 [mem 0x000a0000-0x000bffff window]
[    1.328392] pci_bus 0000:00: resource 8 [mem 0x20000000-0xfffbffff window]
[    1.328458] NET: Registered protocol family 2
[    1.335996] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[    1.344798] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    1.353329] TCP: Hash tables configured (established 65536 bind 65536)
[    1.361977] TCP: reno registered
[    1.368820] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[    1.376160] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[    1.384683] NET: Registered protocol family 1
[    1.390695] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.398559] pci 0000:00:08.0: Boot video device
[    1.398605] PCI: CLS 0 bytes, default 64
[    1.398673] Unpacking initramfs...
[    1.784412] Freeing initrd memory: 17504k freed
[    1.798634] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.809043] software IO TLB [mem 0x1a7e6000-0x1e7e6000] (64MB) mapped at [ffff88001a7e6000-ffff88001e7e5fff]
[    1.823332] sha1_ssse3: Neither AVX nor AVX2 nor SSSE3/SHA-NI is available/usable.
[    1.833200] sha256_ssse3: Neither AVX nor SSSE3/SHA-NI is available/usable.
[    1.844437] futex hash table entries: 16384 (order: 8, 1048576 bytes)
[    1.854439] Initialise system trusted keyring
[    1.861276] audit: initializing netlink socket (disabled)
[    1.869590] type=2000 audit(1495626514.869:1): initialized
[    1.907913] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.917389] zpool: loaded
[    1.921987] zbud: loaded
[    1.926875] VFS: Disk quotas dquot_6.5.2
[    1.932826] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.940096] msgmni has been set to 13933
[    1.945668] Key type big_key registered
[    1.951011] SELinux:  Registering netfilter hooks
[    1.954610] NET: Registered protocol family 38
[    1.960505] Key type asymmetric registered
[    1.966035] Asymmetric key parser 'x509' registered
[    1.972088] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    1.980123] io scheduler noop registered
[    1.985781] io scheduler deadline registered (default)
[    1.992092] io scheduler cfq registered
[    1.997457] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    2.004059] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    2.011481] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    2.019340] ACPI: Power Button [PWRF]
[    2.024889] GHES: HEST is not enabled!
[    2.030314] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.066097] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.101438] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.111997] Non-volatile memory driver v1.3
[    2.123807] Linux agpgart interface v0.103
[    2.130408] crash memory driver: version 1.1
[    2.136884] rdac: device handler registered
[    2.143447] hp_sw: device handler registered
[    2.152158] emc: device handler registered
[    2.158503] alua: device handler registered
[    2.165086] libphy: Fixed MDIO Bus: probed
[    2.170865] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.178217] ehci-pci: EHCI PCI platform driver
[    2.184881] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.192059] ohci-pci: OHCI PCI platform driver
[    2.198889] uhci_hcd: USB Universal Host Controller Interface driver
[    2.206309] usbcore: registered new interface driver usbserial
[    2.215074] usbcore: registered new interface driver usbserial_generic
[    2.222364] usbserial: USB Serial support registered for generic
[    2.238019] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
[    2.254915] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.262070] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.268551] mousedev: PS/2 mouse device common for all mice
[    2.282800] rtc_cmos 00:00: RTC can wake from S4
[    2.316951] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
[    2.317761] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    2.341026] rtc_cmos 00:00: alarms up to one month, 114 bytes nvram
[    2.353812] cpuidle: using governor menu
[    2.359859] hidraw: raw HID events driver (C) Jiri Kosina
[    2.368178] usbcore: registered new interface driver usbhid
[    2.374661] usbhid: USB HID core driver
[    2.381892] drop_monitor: Initializing network drop monitor service
[    2.390360] TCP: cubic registered
[    2.395513] Initializing XFRM netlink socket
[    2.402729] NET: Registered protocol family 10
[    2.408976] NET: Registered protocol family 17
[    2.416326] microcode: CPU0: patch_level=0xffffffff
[    2.424023] microcode: CPU1: patch_level=0xffffffff
[    2.430675] microcode: CPU2: patch_level=0xffffffff
[    2.435916] microcode: CPU3: patch_level=0xffffffff
[    2.442382] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    2.454452] Loading compiled-in X.509 certificates
[    2.462016] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[    2.476746] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[    2.493040] Loaded X.509 cert 'CentOS Linux kernel signing key: 3fe1eb8b4f91d484cd55448454a024de5634e106'
[    2.503393] registered taskstats version 1
[    2.513291] Key type trusted registered
[    2.524445] Key type encrypted registered
[    2.537972] psmouse serio1: alps: Unknown ALPS touchpad: E7=12 00 64, EC=12 00 64
[    2.540308] IMA: No TPM chip found, activating TPM-bypass!
[    2.547882] rtc_cmos 00:00: setting system clock to 2017-05-24 11:48:37 UTC (1495626517)
[    2.569075] Freeing unused kernel memory: 1680k freed
[    2.580974] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[    2.597576] systemd[1]: Detected virtualization microsoft.
[    2.604164] systemd[1]: Detected architecture x86-64.
[    2.610359] systemd[1]: Running in initial RAM disk.
[    2.623937] systemd[1]: Set hostname to <RES-Aerospike10>.
[    2.682923] systemd[1]: Reached target Timers.
[    2.691767] systemd[1]: Starting Timers.
[    2.704851] systemd[1]: Created slice Root Slice.
[    2.711645] systemd[1]: Starting Root Slice.
[    2.728182] systemd[1]: Created slice System Slice.
[    2.748188] systemd[1]: Starting System Slice.
[    2.760388] systemd[1]: Reached target Slices.
[    2.768858] systemd[1]: Starting Slices.
[    2.784279] psmouse serio1: trackpoint: failed to get extended button data
[    2.797721] systemd[1]: Listening on Journal Socket.
[    2.816017] systemd[1]: Starting Journal Socket.
[    2.826578] systemd[1]: Starting Journal Service...
[    2.847936] systemd[1]: Listening on udev Kernel Socket.
[    2.857103] systemd[1]: Starting udev Kernel Socket.
[    2.871486] systemd[1]: Reached target Local File Systems.
[    2.878227] systemd[1]: Starting Local File Systems.
[    2.891620] systemd[1]: Listening on udev Control Socket.
[    2.901620] systemd[1]: Starting udev Control Socket.
[    2.933108] systemd[1]: Reached target Sockets.
[    2.945377] systemd[1]: Starting Sockets.
[    2.953369] systemd[1]: Starting Setup Virtual Console...
[    2.996956] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[    3.046682] systemd[1]: Reached target Swap.
[    3.053740] systemd[1]: Starting Swap.
[    3.061069] systemd[1]: Starting dracut cmdline hook...
[    3.082798] systemd[1]: Starting Apply Kernel Variables...
[    3.118734] systemd[1]: Started Journal Service.
[    3.316648] Floppy drive(s): fd0 is 1.44M
[    3.334692] FUJITSU Extended Socket Network Device Driver - version 1.1 - Copyright (c) 2015 FUJITSU LIMITED
[    3.345445] Switched to clocksource hyperv_clocksource_tsc_page
[    3.363938] FDC 0 is an 82078.
[    3.371007] libata version 3.00 loaded.
[    3.384187] ata_piix 0000:00:07.1: version 2.13
[    3.384477] ata_piix 0000:00:07.1: Hyper-V Virtual Machine detected, ATA device ignore set
[    3.393145] scsi host0: ata_piix
[    3.397710] scsi host1: ata_piix
[    3.397874] ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
[    3.397876] ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
[    3.455406] hv_vmbus: Hyper-V Host Build:9600-6.3-18-0.18190; Vmbus version:3.0
[    3.455408] hv_vmbus: CPU offlining is not supported by hypervisor
[    3.478105] hv_vmbus: Unknown GUID: f8e65716-3cb3-4a06-9a60-1889c5cccab5
[    3.484556] hv_vmbus: Unknown GUID: 3375baf4-9e15-4b30-b765-67acb10d607b
[    3.497946] hv_vmbus: registering driver hv_storvsc
[    3.498595] hv_vmbus: registering driver hyperv_fb
[    3.500015] hv_vmbus: Unknown GUID: 276aacf4-ac15-426c-98dd-7521ad3f01fe
[    3.518577] scsi host3: storvsc_host_t
[    3.525353] hv_vmbus: registering driver hyperv_keyboard
[    3.525832] hv_vmbus: registering driver hid_hyperv
[    3.540894] hv_vmbus: registering driver hv_netvsc
[    3.540934] scsi host2: storvsc_host_t
[    3.543072] scsi 2:0:0:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 4
[    3.543177] ata1.01: host indicates ignore ATA devices, ignored
[    3.544485] ata1.00: host indicates ignore ATA devices, ignored
[    3.557144] scsi host4: storvsc_host_t
[    3.565325] scsi 4:0:0:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 4
[    3.596169] scsi host5: storvsc_host_t
[    3.604227] scsi 5:0:1:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 4
[    3.620759] hyperv_fb: Screen resolution: 1152x864, Color depth: 32
[    3.637807] Console: switching to colour frame buffer device 144x54
[    3.640207] sd 2:0:0:0: [sda] 266338304 512-byte logical blocks: (136 GB/127 GiB)
[    3.640213] sd 4:0:0:0: [sdb] 629145600 512-byte logical blocks: (322 GB/300 GiB)
[    3.640662] sd 2:0:0:0: [sda] Write Protect is off
[    3.640665] sd 2:0:0:0: [sda] Mode Sense: 0f 00 10 00
[    3.641074] sd 5:0:1:0: [sdc] 597688320 512-byte logical blocks: (306 GB/285 GiB)
[    3.641076] sd 5:0:1:0: [sdc] 4096-byte physical blocks
[    3.641089] sd 4:0:0:0: [sdb] Write Protect is off
[    3.641093] sd 4:0:0:0: [sdb] Mode Sense: 0f 00 10 00
[    3.641149] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    3.641281] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    3.642407]  sda: sda1
[    3.643383] sd 2:0:0:0: [sda] Attached SCSI disk
[    3.650755] sd 5:0:1:0: [sdc] Write Protect is off
[    3.650758] sd 5:0:1:0: [sdc] Mode Sense: 0f 00 10 00
[    3.652179] sd 5:0:1:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    3.652611] input: Microsoft Vmbus HID-compliant Mouse as /devices/virtual/input/input2
[    3.652678] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    3.661150] hv_netvsc: hv_netvsc channel opened successfully
[    3.715654]  sdc: sdc1
[    3.719763] sd 5:0:1:0: [sdc] Attached SCSI disk
[    3.734274]  sdb: sdb1
[    3.738000] sd 4:0:0:0: [sdb] Attached SCSI disk
[    3.754079] hv_netvsc vmbus_14: Send section size: 6144, Section count:2560
[    3.762522] hv_netvsc vmbus_14: Device MAC 00:0d:3a:70:98:31 link state up
[    4.912375] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    6.836076] psmouse serio1: trackpoint: IBM TrackPoint firmware: 0x01, buttons: 0/0
[    6.850397] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/input/input3
[    6.862904] input: AT Translated Set 2 keyboard as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:08/VMBUS:01/vmbus_5/serio2/input/input4
[    7.385053] systemd-journald[102]: Received SIGTERM from PID 1 (systemd).
[   12.547739] type=1404 audit(1495626527.504:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
[   13.573789] SELinux: 2048 avtab hash slots, 104872 rules.
[   13.649895] SELinux: 2048 avtab hash slots, 104872 rules.
[   13.704475] SELinux:  8 users, 14 roles, 4980 types, 301 bools, 1 sens, 1024 cats
[   13.704481] SELinux:  91 classes, 104872 rules
[   13.711630] SELinux:  Completing initialization.
[   13.711636] SELinux:  Setting up existing superblocks.
[   13.731329] type=1403 audit(1495626528.688:3): policy loaded auid=4294967295 ses=4294967295
[   13.744518] systemd[1]: Successfully loaded SELinux policy in 1.241254s.
[   17.927889] ip_tables: (C) 2000-2006 Netfilter Core Team
[   17.933919] systemd[1]: Inserted module 'ip_tables'
[   17.967895] systemd[1]: Relabelled /dev and /run in 27.118ms.
[   26.290331] EXT4-fs (sda1): re-mounted. Opts: (null)
[   27.048802] systemd-journald[373]: Received request to flush runtime journal from PID 1
[   29.387549] hv_utils: Registering HyperV Utility Driver
[   29.391450] hv_vmbus: registering driver hv_util
[   29.395540] input: PC Speaker as /devices/platform/pcspkr/input/input5
[   29.476113] piix4_smbus 0000:00:07.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[   29.716259] sd 2:0:0:0: Attached scsi generic sg0 type 0
[   29.721849] sd 4:0:0:0: Attached scsi generic sg1 type 0
[   29.727280] sd 5:0:1:0: Attached scsi generic sg2 type 0
[   29.795187] attempt to access beyond end of device
[   29.800109] sda1: rw=0, want=266323528, limit=266323522
[   30.914378] device-mapper: uevent: version 1.0.3
[   30.922297] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[   33.346246] type=1305 audit(1495626549.022:4): audit_pid=474 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   37.013867] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   49.034379]  sdc: sdc1
[   49.154566]  sdc: sdc1
[   49.163459]  sdc: sdc1
[   52.760099] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)
[  364.992110] EXT4-fs error (device sdc1): ext4_map_blocks:560: inode #8: block 37257410: comm jbd2/sdc1-8: lblock 194 mapped to illegal pblock (length 1)
[  365.003428] jbd2_journal_bmap: journal block not found at offset 194 on sdc1-8
[  365.009244] Aborting journal on device sdc1-8.

You shouldn’t mount the devices that have been allocated to Aerospike. Aerospike doesn’t use a filesystem for these devices.

There is an option to use files instead of raw devices (see file). Bypassing the filesystem (using device) typically provides better performance.

@kporter apparently there are 2 anchors named #file so that brought him to the wrong place. @arulkumar just scroll down to file near storage details on that page

1 Like