Hi
I have deployed multi-node cluster in centos 7(azure), working fine after some time my storage location not found there is not data in my storage location and showing disk consumption 100% I m not able to realize . I was restart the machine after it was not started kindly find my storage configuration and disk structure
My storage loc:
storage engine device {
device /dev/sdc1
write-block-size 128k
}
find the var/log/messages
0.690274] pci 0000:00:07.3: quirk: [io 0x0400-0x043f] claimed by PIIX4 ACPI
[ 0.701737] pci 0000:00:08.0: [1414:5353] type 00 class 0x030000
[ 0.702128] pci 0000:00:08.0: reg 0x10: [mem 0xf8000000-0xfbffffff]
[ 0.714574] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 10 *11 12 14 15)
[ 0.771288] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, disabled.
[ 0.820134] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, disabled.
[ 0.870262] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, disabled.
[ 0.936165] ACPI: Enabled 1 GPEs in block 00 to 0F
[ 0.954099] vgaarb: device added: PCI:0000:00:08.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.965006] vgaarb: loaded
[ 0.984008] vgaarb: bridge control possible 0000:00:08.0
[ 0.991152] SCSI subsystem initialized
[ 1.000056] ACPI: bus type USB registered
[ 1.018054] usbcore: registered new interface driver usbfs
[ 1.027024] usbcore: registered new interface driver hub
[ 1.038123] usbcore: registered new device driver usb
[ 1.046106] PCI: Using ACPI for IRQ routing
[ 1.054008] PCI: pci_cache_line_size set to 64 bytes
[ 1.055045] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[ 1.055049] e820: reserve RAM buffer [mem 0x1fff0000-0x1fffffff]
[ 1.055221] NetLabel: Initializing
[ 1.061004] NetLabel: domain hash size = 128
[ 1.071005] NetLabel: protocols = UNLABELED CIPSOv4
[ 1.077019] NetLabel: unlabeled traffic allowed by default
[ 1.086135] Switched to clocksource hyperv_clocksource
[ 1.100161] pnp: PnP ACPI init
[ 1.106236] ACPI: bus type PNP registered
[ 1.112520] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 1.113048] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[ 1.113567] pnp 00:02: Plug and Play ACPI device, IDs PNP0f03 PNP0f13 (active)
[ 1.116403] pnp 00:03: [dma 0 disabled]
[ 1.116441] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active)
[ 1.118297] pnp 00:04: [dma 0 disabled]
[ 1.118352] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
[ 1.119533] pnp 00:05: [dma 2]
[ 1.119575] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)
[ 1.119626] system 00:06: [io 0x01e0-0x01ef] has been reserved
[ 1.127296] system 00:06: [io 0x0160-0x016f] has been reserved
[ 1.149062] system 00:06: [io 0x0278-0x027f] has been reserved
[ 1.157779] system 00:06: [io 0x0378-0x037f] has been reserved
[ 1.177362] system 00:06: [io 0x0678-0x067f] has been reserved
[ 1.187482] system 00:06: [io 0x0778-0x077f] has been reserved
[ 1.195949] system 00:06: [io 0x04d0-0x04d1] has been reserved
[ 1.203188] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.203419] system 00:07: [io 0x0400-0x043f] has been reserved
[ 1.213224] system 00:07: [io 0x0370-0x0371] has been reserved
[ 1.220925] system 00:07: [io 0x0440-0x044f] has been reserved
[ 1.229872] system 00:07: [mem 0xfec00000-0xfec00fff] could not be reserved
[ 1.239150] system 00:07: [mem 0xfee00000-0xfee00fff] has been reserved
[ 1.247404] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.247635] system 00:08: [mem 0x00000000-0x0009ffff] could not be reserved
[ 1.258215] system 00:08: [mem 0x000c0000-0x000dffff] could not be reserved
[ 1.272497] system 00:08: [mem 0x000e0000-0x000fffff] could not be reserved
[ 1.280503] system 00:08: [mem 0x00100000-0x1fffffff] could not be reserved
[ 1.288680] system 00:08: [mem 0xfffc0000-0xffffffff] has been reserved
[ 1.296399] system 00:08: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 1.296687] pnp: PnP ACPI: found 9 devices
[ 1.302912] ACPI: bus type PNP unregistered
[ 1.328378] pci_bus 0000:00: resource 4 [mem 0xfe0000000-0xfffefffff window]
[ 1.328384] pci_bus 0000:00: resource 5 [io 0x0000-0x0cf7 window]
[ 1.328387] pci_bus 0000:00: resource 6 [io 0x0d00-0xffff window]
[ 1.328389] pci_bus 0000:00: resource 7 [mem 0x000a0000-0x000bffff window]
[ 1.328392] pci_bus 0000:00: resource 8 [mem 0x20000000-0xfffbffff window]
[ 1.328458] NET: Registered protocol family 2
[ 1.335996] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[ 1.344798] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 1.353329] TCP: Hash tables configured (established 65536 bind 65536)
[ 1.361977] TCP: reno registered
[ 1.368820] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[ 1.376160] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[ 1.384683] NET: Registered protocol family 1
[ 1.390695] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 1.398559] pci 0000:00:08.0: Boot video device
[ 1.398605] PCI: CLS 0 bytes, default 64
[ 1.398673] Unpacking initramfs...
[ 1.784412] Freeing initrd memory: 17504k freed
[ 1.798634] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 1.809043] software IO TLB [mem 0x1a7e6000-0x1e7e6000] (64MB) mapped at [ffff88001a7e6000-ffff88001e7e5fff]
[ 1.823332] sha1_ssse3: Neither AVX nor AVX2 nor SSSE3/SHA-NI is available/usable.
[ 1.833200] sha256_ssse3: Neither AVX nor SSSE3/SHA-NI is available/usable.
[ 1.844437] futex hash table entries: 16384 (order: 8, 1048576 bytes)
[ 1.854439] Initialise system trusted keyring
[ 1.861276] audit: initializing netlink socket (disabled)
[ 1.869590] type=2000 audit(1495626514.869:1): initialized
[ 1.907913] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 1.917389] zpool: loaded
[ 1.921987] zbud: loaded
[ 1.926875] VFS: Disk quotas dquot_6.5.2
[ 1.932826] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 1.940096] msgmni has been set to 13933
[ 1.945668] Key type big_key registered
[ 1.951011] SELinux: Registering netfilter hooks
[ 1.954610] NET: Registered protocol family 38
[ 1.960505] Key type asymmetric registered
[ 1.966035] Asymmetric key parser 'x509' registered
[ 1.972088] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[ 1.980123] io scheduler noop registered
[ 1.985781] io scheduler deadline registered (default)
[ 1.992092] io scheduler cfq registered
[ 1.997457] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 2.004059] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 2.011481] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 2.019340] ACPI: Power Button [PWRF]
[ 2.024889] GHES: HEST is not enabled!
[ 2.030314] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.066097] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 2.101438] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 2.111997] Non-volatile memory driver v1.3
[ 2.123807] Linux agpgart interface v0.103
[ 2.130408] crash memory driver: version 1.1
[ 2.136884] rdac: device handler registered
[ 2.143447] hp_sw: device handler registered
[ 2.152158] emc: device handler registered
[ 2.158503] alua: device handler registered
[ 2.165086] libphy: Fixed MDIO Bus: probed
[ 2.170865] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 2.178217] ehci-pci: EHCI PCI platform driver
[ 2.184881] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 2.192059] ohci-pci: OHCI PCI platform driver
[ 2.198889] uhci_hcd: USB Universal Host Controller Interface driver
[ 2.206309] usbcore: registered new interface driver usbserial
[ 2.215074] usbcore: registered new interface driver usbserial_generic
[ 2.222364] usbserial: USB Serial support registered for generic
[ 2.238019] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
[ 2.254915] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 2.262070] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 2.268551] mousedev: PS/2 mouse device common for all mice
[ 2.282800] rtc_cmos 00:00: RTC can wake from S4
[ 2.316951] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
[ 2.317761] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[ 2.341026] rtc_cmos 00:00: alarms up to one month, 114 bytes nvram
[ 2.353812] cpuidle: using governor menu
[ 2.359859] hidraw: raw HID events driver (C) Jiri Kosina
[ 2.368178] usbcore: registered new interface driver usbhid
[ 2.374661] usbhid: USB HID core driver
[ 2.381892] drop_monitor: Initializing network drop monitor service
[ 2.390360] TCP: cubic registered
[ 2.395513] Initializing XFRM netlink socket
[ 2.402729] NET: Registered protocol family 10
[ 2.408976] NET: Registered protocol family 17
[ 2.416326] microcode: CPU0: patch_level=0xffffffff
[ 2.424023] microcode: CPU1: patch_level=0xffffffff
[ 2.430675] microcode: CPU2: patch_level=0xffffffff
[ 2.435916] microcode: CPU3: patch_level=0xffffffff
[ 2.442382] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[ 2.454452] Loading compiled-in X.509 certificates
[ 2.462016] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[ 2.476746] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[ 2.493040] Loaded X.509 cert 'CentOS Linux kernel signing key: 3fe1eb8b4f91d484cd55448454a024de5634e106'
[ 2.503393] registered taskstats version 1
[ 2.513291] Key type trusted registered
[ 2.524445] Key type encrypted registered
[ 2.537972] psmouse serio1: alps: Unknown ALPS touchpad: E7=12 00 64, EC=12 00 64
[ 2.540308] IMA: No TPM chip found, activating TPM-bypass!
[ 2.547882] rtc_cmos 00:00: setting system clock to 2017-05-24 11:48:37 UTC (1495626517)
[ 2.569075] Freeing unused kernel memory: 1680k freed
[ 2.580974] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[ 2.597576] systemd[1]: Detected virtualization microsoft.
[ 2.604164] systemd[1]: Detected architecture x86-64.
[ 2.610359] systemd[1]: Running in initial RAM disk.
[ 2.623937] systemd[1]: Set hostname to <RES-Aerospike10>.
[ 2.682923] systemd[1]: Reached target Timers.
[ 2.691767] systemd[1]: Starting Timers.
[ 2.704851] systemd[1]: Created slice Root Slice.
[ 2.711645] systemd[1]: Starting Root Slice.
[ 2.728182] systemd[1]: Created slice System Slice.
[ 2.748188] systemd[1]: Starting System Slice.
[ 2.760388] systemd[1]: Reached target Slices.
[ 2.768858] systemd[1]: Starting Slices.
[ 2.784279] psmouse serio1: trackpoint: failed to get extended button data
[ 2.797721] systemd[1]: Listening on Journal Socket.
[ 2.816017] systemd[1]: Starting Journal Socket.
[ 2.826578] systemd[1]: Starting Journal Service...
[ 2.847936] systemd[1]: Listening on udev Kernel Socket.
[ 2.857103] systemd[1]: Starting udev Kernel Socket.
[ 2.871486] systemd[1]: Reached target Local File Systems.
[ 2.878227] systemd[1]: Starting Local File Systems.
[ 2.891620] systemd[1]: Listening on udev Control Socket.
[ 2.901620] systemd[1]: Starting udev Control Socket.
[ 2.933108] systemd[1]: Reached target Sockets.
[ 2.945377] systemd[1]: Starting Sockets.
[ 2.953369] systemd[1]: Starting Setup Virtual Console...
[ 2.996956] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[ 3.046682] systemd[1]: Reached target Swap.
[ 3.053740] systemd[1]: Starting Swap.
[ 3.061069] systemd[1]: Starting dracut cmdline hook...
[ 3.082798] systemd[1]: Starting Apply Kernel Variables...
[ 3.118734] systemd[1]: Started Journal Service.
[ 3.316648] Floppy drive(s): fd0 is 1.44M
[ 3.334692] FUJITSU Extended Socket Network Device Driver - version 1.1 - Copyright (c) 2015 FUJITSU LIMITED
[ 3.345445] Switched to clocksource hyperv_clocksource_tsc_page
[ 3.363938] FDC 0 is an 82078.
[ 3.371007] libata version 3.00 loaded.
[ 3.384187] ata_piix 0000:00:07.1: version 2.13
[ 3.384477] ata_piix 0000:00:07.1: Hyper-V Virtual Machine detected, ATA device ignore set
[ 3.393145] scsi host0: ata_piix
[ 3.397710] scsi host1: ata_piix
[ 3.397874] ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
[ 3.397876] ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
[ 3.455406] hv_vmbus: Hyper-V Host Build:9600-6.3-18-0.18190; Vmbus version:3.0
[ 3.455408] hv_vmbus: CPU offlining is not supported by hypervisor
[ 3.478105] hv_vmbus: Unknown GUID: f8e65716-3cb3-4a06-9a60-1889c5cccab5
[ 3.484556] hv_vmbus: Unknown GUID: 3375baf4-9e15-4b30-b765-67acb10d607b
[ 3.497946] hv_vmbus: registering driver hv_storvsc
[ 3.498595] hv_vmbus: registering driver hyperv_fb
[ 3.500015] hv_vmbus: Unknown GUID: 276aacf4-ac15-426c-98dd-7521ad3f01fe
[ 3.518577] scsi host3: storvsc_host_t
[ 3.525353] hv_vmbus: registering driver hyperv_keyboard
[ 3.525832] hv_vmbus: registering driver hid_hyperv
[ 3.540894] hv_vmbus: registering driver hv_netvsc
[ 3.540934] scsi host2: storvsc_host_t
[ 3.543072] scsi 2:0:0:0: Direct-Access Msft Virtual Disk 1.0 PQ: 0 ANSI: 4
[ 3.543177] ata1.01: host indicates ignore ATA devices, ignored
[ 3.544485] ata1.00: host indicates ignore ATA devices, ignored
[ 3.557144] scsi host4: storvsc_host_t
[ 3.565325] scsi 4:0:0:0: Direct-Access Msft Virtual Disk 1.0 PQ: 0 ANSI: 4
[ 3.596169] scsi host5: storvsc_host_t
[ 3.604227] scsi 5:0:1:0: Direct-Access Msft Virtual Disk 1.0 PQ: 0 ANSI: 4
[ 3.620759] hyperv_fb: Screen resolution: 1152x864, Color depth: 32
[ 3.637807] Console: switching to colour frame buffer device 144x54
[ 3.640207] sd 2:0:0:0: [sda] 266338304 512-byte logical blocks: (136 GB/127 GiB)
[ 3.640213] sd 4:0:0:0: [sdb] 629145600 512-byte logical blocks: (322 GB/300 GiB)
[ 3.640662] sd 2:0:0:0: [sda] Write Protect is off
[ 3.640665] sd 2:0:0:0: [sda] Mode Sense: 0f 00 10 00
[ 3.641074] sd 5:0:1:0: [sdc] 597688320 512-byte logical blocks: (306 GB/285 GiB)
[ 3.641076] sd 5:0:1:0: [sdc] 4096-byte physical blocks
[ 3.641089] sd 4:0:0:0: [sdb] Write Protect is off
[ 3.641093] sd 4:0:0:0: [sdb] Mode Sense: 0f 00 10 00
[ 3.641149] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 3.641281] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 3.642407] sda: sda1
[ 3.643383] sd 2:0:0:0: [sda] Attached SCSI disk
[ 3.650755] sd 5:0:1:0: [sdc] Write Protect is off
[ 3.650758] sd 5:0:1:0: [sdc] Mode Sense: 0f 00 10 00
[ 3.652179] sd 5:0:1:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 3.652611] input: Microsoft Vmbus HID-compliant Mouse as /devices/virtual/input/input2
[ 3.652678] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[ 3.661150] hv_netvsc: hv_netvsc channel opened successfully
[ 3.715654] sdc: sdc1
[ 3.719763] sd 5:0:1:0: [sdc] Attached SCSI disk
[ 3.734274] sdb: sdb1
[ 3.738000] sd 4:0:0:0: [sdb] Attached SCSI disk
[ 3.754079] hv_netvsc vmbus_14: Send section size: 6144, Section count:2560
[ 3.762522] hv_netvsc vmbus_14: Device MAC 00:0d:3a:70:98:31 link state up
[ 4.912375] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 6.836076] psmouse serio1: trackpoint: IBM TrackPoint firmware: 0x01, buttons: 0/0
[ 6.850397] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/input/input3
[ 6.862904] input: AT Translated Set 2 keyboard as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:08/VMBUS:01/vmbus_5/serio2/input/input4
[ 7.385053] systemd-journald[102]: Received SIGTERM from PID 1 (systemd).
[ 12.547739] type=1404 audit(1495626527.504:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
[ 13.573789] SELinux: 2048 avtab hash slots, 104872 rules.
[ 13.649895] SELinux: 2048 avtab hash slots, 104872 rules.
[ 13.704475] SELinux: 8 users, 14 roles, 4980 types, 301 bools, 1 sens, 1024 cats
[ 13.704481] SELinux: 91 classes, 104872 rules
[ 13.711630] SELinux: Completing initialization.
[ 13.711636] SELinux: Setting up existing superblocks.
[ 13.731329] type=1403 audit(1495626528.688:3): policy loaded auid=4294967295 ses=4294967295
[ 13.744518] systemd[1]: Successfully loaded SELinux policy in 1.241254s.
[ 17.927889] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 17.933919] systemd[1]: Inserted module 'ip_tables'
[ 17.967895] systemd[1]: Relabelled /dev and /run in 27.118ms.
[ 26.290331] EXT4-fs (sda1): re-mounted. Opts: (null)
[ 27.048802] systemd-journald[373]: Received request to flush runtime journal from PID 1
[ 29.387549] hv_utils: Registering HyperV Utility Driver
[ 29.391450] hv_vmbus: registering driver hv_util
[ 29.395540] input: PC Speaker as /devices/platform/pcspkr/input/input5
[ 29.476113] piix4_smbus 0000:00:07.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[ 29.716259] sd 2:0:0:0: Attached scsi generic sg0 type 0
[ 29.721849] sd 4:0:0:0: Attached scsi generic sg1 type 0
[ 29.727280] sd 5:0:1:0: Attached scsi generic sg2 type 0
[ 29.795187] attempt to access beyond end of device
[ 29.800109] sda1: rw=0, want=266323528, limit=266323522
[ 30.914378] device-mapper: uevent: version 1.0.3
[ 30.922297] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[ 33.346246] type=1305 audit(1495626549.022:4): audit_pid=474 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 37.013867] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 49.034379] sdc: sdc1
[ 49.154566] sdc: sdc1
[ 49.163459] sdc: sdc1
[ 52.760099] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)
[ 364.992110] EXT4-fs error (device sdc1): ext4_map_blocks:560: inode #8: block 37257410: comm jbd2/sdc1-8: lblock 194 mapped to illegal pblock (length 1)
[ 365.003428] jbd2_journal_bmap: journal block not found at offset 194 on sdc1-8
[ 365.009244] Aborting journal on device sdc1-8.