OpenVZ Forum: Support » Lost connectivity of VPS after reboot.

Home » General » Support » Lost connectivity of VPS after reboot.

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Lost connectivity of VPS after reboot. [message #51332]

Tue, 15 April 2014 19:46

danl
Messages: 1
Registered: April 2014
Location: CANADA

Junior Member

Hello,

I have inherited of a legacy Linux park with most machines running OpenVZ. And it's always been working great. Must I also add that I really enjoy working with your solution? However a problem was brought up to me: some service stopped responding on one of the VPSes.

Before going any further I would like to mention that I have already read both of the following posts without any success.

1. http column slash slash forum.openvz.org/index.php?t=tree&goto=30837&&sr ch=Unable+to+get+tty+name%3A+Bad+file+descriptor#msg_30837
2. http column slash slash forum.openvz.org/index.php?t=tree&goto=29417&&sr ch=Unable+to+get+tty+name%3A+Bad+file+descriptor#msg_29417
(PS: You cannot use links until you have posted more than 10 messages. <- message I got when I clicked on "CREATE TOPIC")

Consider the following:

Physical server hosting the OpenVZ service:

Linux HOSTMAXI 2.6.24-29-openvz #1 SMP Tue Oct 11 18:05:43 UTC 2011 x86_64 GNU/Linux

The first conclusion I can draw is that the VPS is missing processes as it's only showing 5 processes running:

       118          5 running -               HOSTMINI

Entering the VPS works fine:

root@HOSTMAXI:/# vzctl enter 118
entered into VE 118
root@HOSTMINI:/#

However the mounted partitions are clearly shady as it's missing tons of data and possibly other partitions. Also, the system should hold 6Gbs of data:

root@HOSTMINI:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
devshm               1003M     0 1003M   0% /dev/shm
root@HOSTMINI:/#

Here's the proof of that off HOSTMAXI:

root@HOSTMAXI:/vz/private# du 118 -h | tail
68K     118/root/backup/jar_content/org
17M     118/root/backup/jar_content
22M     118/root/backup
4.0K    118/root/.aptitude
8.0K    118/root/.ssh
91M     118/root
4.0K    118/dev/pts
4.0K    118/dev/shm
12K     118/dev
6.0G    118
root@HOSTMAXI:/vz/private#

If I look at any other disk usage from any/all of the other running VPS on the same physical host (HOSTMAXI) they'll all look somewhat like this (With the exception of the total size of simfs):

root@HOSTMAXI:/vz/private# vzctl enter 114
entered into VE 114
root@HOSTMINI-2:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
simfs                 5.0G  914M  4.2G  18% /
tmpfs                1003M   44K 1003M   1% /var/run
tmpfs                1003M     0 1003M   0% /var/lock
devshm               1003M     0 1003M   0% /dev/shm
root@HOSTMINI-2:/#

Also, the content of /etc/fstab from that working VPS shows this:

root@HOSTMINI-2:/#cat /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
root@HOSTMINI-2:/#

However just the the content of HOSTMINI:/etc/fstab (AKA "The broken one") shows the following:

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>                       <dump>  <pass>
proc            /proc           proc    defaults                        0       0

Before going any further I would like to mention that BCFG2 was disabled around 6 months ago and those systems were all running perfectly fine ever since. Also, HOSTMINI was working ~1 month ago. At this point I'm suspecting file corruption somewhere, but where?

Let's continue analyzing the faulty VPS and see what other infos I can dig out:

root@HOSTMINI:/# runlevel
N 2
root@HOSTMINI:/# ls /etc/rc2.d/
S10sysklogd  S20archiva  S20bcfg2  S20makedev  S20postfix  S20rsync  S20ssh  S89cron  S98munin-node  S99rc.local  S99rmnologin
root@HOSTMINI:/# netstat -noa
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      0 10.32.57.76:4949        0.0.0.0:*               LISTEN     off (0.00/0/0)
tcp6       0      0 :::22                   :::*                    LISTEN     off (0.00/0/0)
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ]         DGRAM                    29877573 /dev/log
root@HOSTMINI:/# /etc/init.d/ssh stop
 * Stopping OpenBSD Secure Shell server...                                                                                                                                                                                            [ ok ]
root@HOSTMINI:/# /etc/init.d/ssh start
 * Starting OpenBSD Secure Shell server...                                                                                                                                                                                            [ ok ]
root@HOSTMINI:/# netstat -noa
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      0 10.32.57.76:4949        0.0.0.0:*               LISTEN     off (0.00/0/0)
tcp6       0      0 :::22                   :::*                    LISTEN     off (0.00/0/0)
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ]         DGRAM                    29877573 /dev/log
root@HOSTMINI:/# uname -a
Linux HOSTMINI 2.6.24-29-openvz #1 SMP Tue Oct 11 18:05:43 UTC 2011 i686 GNU/Linux
root@HOSTMINI:/#

From what we just saw, SSH port 22 only shows for IPv6 and not for IPv4. Yet, again, that(SSH on port 22 for IPv4) worked last month.

Here's something that also hopefully can help you out as this is sadly too far out of scope for me (for now):

root@HOSTMAXI:/etc/vz/conf# strace vzctl enter 118
execve("/usr/sbin/vzctl", ["vzctl", "enter", "118"], [/* 19 vars */]) = 0
brk(0)                                  = 0x609000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7f9696e000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7f9696c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/x86_64/libvzctl-0.0.2.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls/x86_64", 0x7fff32fd1d30) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libvzctl-0.0.2.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls", 0x7fff32fd1d30)    = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64/libvzctl-0.0.2.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64", 0x7fff32fd1d30) = -1 ENOENT (No such file or directory)
open("/usr/lib/libvzctl-0.0.2.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0Ph\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=152536, ...}) = 0
mmap(NULL, 2247848, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7f9652d000
mprotect(0x7f7f96550000, 2097152, PROT_NONE) = 0
mmap(0x7f7f96750000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f7f96750000
close(3)                                = 0
open("/usr/lib/libdl.so.2", O_RDONLY)   = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=10919, ...}) = 0
mmap(NULL, 10919, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7f96969000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libdl.so.2", O_RDONLY)       = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \16\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14624, ...}) = 0
mmap(NULL, 2109728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7f96329000
mprotect(0x7f7f9632b000, 2097152, PROT_NONE) = 0
mmap(0x7f7f9652b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7f9652b000
close(3)                                = 0
open("/usr/lib/libutil.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libutil.so.1", O_RDONLY)     = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\16\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=10584, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7f96968000
mmap(NULL, 2105624, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7f96126000
mprotect(0x7f7f96128000, 2093056, PROT_NONE) = 0
mmap(0x7f7f96327000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f7f96327000
close(3)                                = 0
open("/usr/lib/libc.so.6", O_RDONLY)    = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\342"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1436976, ...}) = 0
mmap(NULL, 3543672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7f95dc4000
mprotect(0x7f7f95f1c000, 2097152, PROT_NONE) = 0
mmap(0x7f7f9611c000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x158000) = 0x7f7f9611c000
mmap(0x7f7f96121000, 17016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7f96121000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7f96967000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MA

...

[ Show the rest of the message ]

Report message to a moderator