Dear Rick,
Here is an excerpt of my logs. I have remove many "fopen: Too many open files in system". Note that I have tried to connect using ssh before and after shutting down the VEs.
Dec 13 21:24:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:24:25 ns22021 Neighbour table overflow.
Dec 13 21:25:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:25:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:25:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:26:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:26:21 ns22021 IPv6 addrconf: prefix with wrong length 56
Dec 13 21:27:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:28:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:29:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:29:36 ns22021 IPv6 addrconf: prefix with wrong length 56
Dec 13 21:30:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:30:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:30:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:30:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:31:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:32:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:32:29 ns22021 IPv6 addrconf: prefix with wrong length 56
Dec 13 21:33:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:34:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:35:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:35:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:35:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:35:28 ns22021 VPS: 18639: stopped
Dec 13 21:35:38 ns22021 IPv6 addrconf: prefix with wrong length 56
Dec 13 21:35:52 ns22021 VPS: 18638: stopped
Dec 13 21:36:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:37:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:38:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:38:01 ns22021 Uncharging too much 2220 h 1368, res othersockbuf ub 18637
Dec 13 21:38:01 ns22021 Uncharging too much 2220 h 0, res othersockbuf ub 18637
Dec 13 21:38:01 ns22021 Uncharging too much 2220 h 0, res othersockbuf ub 18637
Dec 13 21:38:01 ns22021 Uncharging too much 2220 h 0, res othersockbuf ub 18637
Dec 13 21:38:01 ns22021 Uncharging too much 2220 h 0, res othersockbuf ub 18637
Dec 13 21:38:01 ns22021 Uncharging too much 2220 h 0, res othersockbuf ub 18637
Dec 13 21:38:07 ns22021 VPS: 18637: stopped
Dec 13 21:38:11 ns22021 IPv6 addrconf: prefix with wrong length 56Dec 13 21:38:26 ns22021 VPS: 18636: stopped
Dec 13 21:38:31 ns22021 Fatal resource shortage: kmemsize, UB 0.
Dec 13 21:38:31 ns22021 sshd[3014]: error: fork: Cannot allocate memory
Dec 13 21:38:35 ns22021 Fatal resource shortage: kmemsize, UB 0.
Dec 13 21:38:35 ns22021 sshd[3014]: error: fork: Cannot allocate memory
Dec 13 21:38:36 ns22021 Fatal resource shortage: kmemsize, UB 0.
Dec 13 21:38:36 ns22021 sshd[3014]: error: fork: Cannot allocate memory
Dec 13 21:38:37 ns22021 Fatal resource shortage: kmemsize, UB 0.
Dec 13 21:38:37 ns22021 sshd[3014]: error: fork: Cannot allocate memory
Dec 13 21:39:01 ns22021 cron[8714]: (CRON) error (can't fork)
Dec 13 21:39:16 ns22021 Ub 18636 helds 13320 in tcpsndbuf on put
Here is the beancounters as of now. Of course, I cannot produce those when the issue occurs since I cannot ssh anymore.
Version: 2.5
uid resource held maxheld barrier limit failcnt
0: kmemsize 109296870 109841586 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 3262 14897 2147483647 2147483647 0
shmpages 679 695 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
numproc 61 82 2147483647 2147483647 0
physpages 1560 6760 2147483647 2147483647 0
vmguarpages 0 0 2147483647 2147483647 0
oomguarpages 1560 6760 2147483647 2147483647 0
numtcpsock 3 5 2147483647 2147483647 0
numflock 1 2 2147483647 2147483647 0
numpty 1 1 2147483647 2147483647 0
numsiginfo 0 2 2147483647 2147483647 0
tcpsndbuf 31080 35520 2147483647 2147483647 0
tcprcvbuf 87564 87564 2147483647 2147483647 0
othersockbuf 11100 16224 2147483647 2147483647 0
dgramrcvbuf 0 8364 2147483647 2147483647 0
numothersock 23 27 2147483647 2147483647 0
dcachesize 0 0 2147483647 2147483647 0
numfile 525860 526009 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
numiptent 0 0 2147483647 2147483647 0
18636: kmemsize 2162620 2702410 18486804 20335484 0
lockedpages 0 0 902 902 0
privvmpages 63043 69808 152556 167811 0
shmpages 14 350 15255 15255 0
dummy 0 0 0 0 0
numproc 46 58 800 800 0
physpages 39341 42286 0 2147483647 0
vmguarpages 0 0 152556 2147483647 0
oomguarpages 39341 42286 152556 2147483647 0
numtcpsock 7 19 800 800 0
numflock 6 7 720 792 0
numpty 0 0 80 80 0
numsiginfo 0 2 1024 1024 0
tcpsndbuf 62160 417360 2885468 6162268 0
tcprcvbuf 114688 147456 2885468 6162268 0
othersockbuf 133012 376720 1442734 4719534 0
dgramrcvbuf 0 4268 1442734 1442734 0
numothersock 95 100 800 800 0
dcachesize 0 0 4026407 4147200 0
numfile 1339 1736 7200 7200 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 0 0 200 200 0
18637: kmemsize 890020 1425116 18486804 20335484 0
lockedpages 0 0 902 902 0
privvmpages 12696 19885 152556 167811 0
shmpages 14 350 15255 15255 0
dummy 0 0 0 0 0
numproc 18 31 800 800 0
physpages 3081 6911 0 2147483647 0
vmguarpages 0 0 152556 2147483647 0
oomguarpages 3081 6911 152556 2147483647 0
numtcpsock 11 25 800 800 0
numflock 2 3 720 792 0
numpty 0 0 80 80 0
numsiginfo 0 2 1024 1024 0
tcpsndbuf 97680 426240 2885468 6162268 0
tcprcvbuf 180224 368984 2885468 6162268 0
othersockbuf 15540 19128 1442734 4719534 0
dgramrcvbuf 0 606056 1442734 1442734 0
numothersock 7 11 800 800 0
dcachesize 0 0 4026407 4147200 0
numfile 619 820 7200 7200 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 0 0 200 200 0
18638: kmemsize 454989 1046206 18486804 20335484 0
lockedpages 0 0 902 902 0
privvmpages 1253 8677 152556 167811 0
shmpages 0 336 15255 15255 0
dummy 0 0 0 0 0
numproc 8 20 800 800 0
physpages 837 1734 0 214
...