🐧 CentOS / RHEL Kernel Panic – ошибка Out of memory включен параметр ядра panic_on_oom

Проблема

Перезагрузка системы CentOS / RHEL из-за ошибки нехватки памяти:

DUMPFILE: vmcore [PARTIAL DUMP]
  CPUS: 2
  DATE: Mon Feb 29 05:28:02 2016
  UPTIME: 33 days, 09:45:55
LOAD AVERAGE: 1.88, 1.52, 1.41
  TASKS: 218
  NODENAME: localhost
  RELEASE: 2.6.32-431.el6.x86_64
  VERSION: #1 SMP Sun Nov 10 22:19:54 EST 2013
  MACHINE: x86_64 (3000 Mhz)
  MEMORY: 8 GB
  PANIC: "Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled"
  PID: 31010
  COMMAND: "java"
  TASK: ffff88010055a040 [THREAD_INFO: ffff88001583e000]
  CPU: 0
  STATE: TASK_RUNNING (PANIC)

Решение

Перезагрузка происходит из-за:

1.  OOM killer и физическая память / память подкачки полностью используются.
2.  vm.panic_on_oom был включен.

>kmem -i

PAGES TOTAL PERCENTAGE
TOTAL MEM 1979578 7.6 GB ----
FREE 27818 108.7 MB 1% of TOTAL MEM
USED 1951760 7.4 GB 98% of TOTAL MEM   98% memory used
SHARED 270 1.1 MB 0% of TOTAL MEM
BUFFERS 53 212 KB 0% of TOTAL MEM
CACHED 1461 5.7 MB 0% of TOTAL MEM
SLAB 14026 54.8 MB 0% of TOTAL MEM

TOTAL SWAP 4196348 16 GB ----
SWAP USED 4196347 16 GB 99% of TOTAL SWAP
SWAP FREE 1 4 KB 0% of TOTAL SWAP      No free swap space

COMMIT LIMIT 5186137 19.8 GB ----
COMMITTED 1564979 6 GB 30% of TOTAL LIMIT

# sysctl_-a |grep vm.panic_on_oom
 vm.panic_on_oom = 1                 OOM enabled for panic.

>ps
..
30934 1 1 ffff88010043aae0 IN 78.3 29303436 7384984 java
30935 1 1 ffff88010043b540 IN 78.3 29303436 7384984 java
30936 1 1 ffff880238f62080 IN 78.3 29303436 7384984 java
31008 1 1 ffff8801060b0aa0 IN 78.3 29303436 7384984 java
31009 1 1 ffff8801060b0040 IN 78.3 29303436 7384984 java
> 31010 1 0 ffff88010055a040 RU 78.3 29303436 7384984 java
31690 1 0 ffff880239fc6aa0 IN 78.3 29303436 7384984 java
...