You could use lsof to see who's using any TCP/UDP sockets with IP 127.0.0.1
as one endpoint (or any other IP if any have been assigned to loopback
interface)
$ *sudo lsof -nPi | grep 127.0.0.1*
chronyd 5334 chrony 1u IPv4 35307 0t0 UDP 127.0.0.1:323
master 6535 root 13u IPv4 38297 0t0 TCP 127.0.0.1:25
(LISTEN)
Regarding the memory errors and reducing MTU size, there have been similar
issues with jumbo frames. "Large" frames and MTU sizes require kernel to
allocate large *contiguous* memory chunks from kernel memory. Since such
allocation sizes can vary (different kernel drivers & allocation reasons
using them), this kernel memory can get fragmented, thus finding a "large"
chunk on demand gets problematic. This can cause paging/reclamation issues
(even if there's some memory free) or kernel memory allocation failures.
One solution would be to tune the Linux kernel settings to keep more memory
free at any time (vm.min_free_kbytes) or use lower MTU/frame/kernel memory
allocation sizes.
You can monitor the fragmentation of (some) kernel memory via
*/proc/buddyinfo*. This is a good article explaining the buddy allocator
basics and how to read /proc/buddyinfo + a python script:
$ *cat /proc/buddyinfo *
Node 0, zone DMA 1 0 0 0 2 1 1
0 1 1 3
Node 0, zone DMA32 2656 4401 3224 4288 1480 277 29
0 0 0 0
Node 0, zone Normal 23 76 25 16 5 0 0
0 1 0 0
$ *./buddyinfo.py *
Node: 0
Zone: DMA
Free KiB in zone: 15876.00
Fragment size Free fragments Total available KiB
4096 1 4.0
8192 0 0.0
16384 0 0.0
32768 0 0.0
65536 2 128.0
131072 1 128.0
262144 1 256.0
524288 0 0.0
1048576 1 1024.0
2097152 1 2048.0
4194304 3 12288.0
Zone: DMA32
Free KiB in zone: 368760.00
Fragment size Free fragments Total available KiB
4096 1788 7152.0
8192 4401 35208.0
16384 3224 51584.0
32768 4288 137216.0
65536 1480 94720.0
131072 277 35456.0
262144 29 7424.0
524288 0 0.0
1048576 0 0.0
2097152 0 0.0
4194304 0 0.0
Zone: Normal
Free KiB in zone: 2956.00
Fragment size Free fragments Total available KiB
4096 23 92.0
8192 76 608.0
16384 25 400.0
32768 16 512.0
65536 5 320.0
131072 0 0.0
262144 0 0.0
524288 0 0.0
1048576 1 1024.0
2097152 0 0.0
4194304 0 0.0
Thanks
--
Tanel Poder
https://blog.tanelpoder.com/seminar
On Tue, Sep 10, 2019 at 10:20 PM Rakesh Ra <rakeshra.tr@xxxxxxxxx> wrote:
Hi All,
Can someone shed some light on for what operations Oracle can use loopback
interface? We had the below issue on one of our primary production database
running on 12.1.0.2 running on Red Hat Linux 7.5.
skgxpvfynet: mtype: 61 process 75262 failed because of a resource problem
in the OS. The OS has most likely run out of buffers (rval: 4) opiodr
aborting process unknown ospid (75262) as a result of ORA-603
Oracle asked us to fix the MTU by lowering it from 64k to 16k by following
the below note.There were just application user connections (jsut less than
400 user processes) and couple of scp process which was shipping the backup
files from the this primary database server to standby database server as
part of DR build.
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 16436 <<<<<<<<<<<< This value was
64K during the issue time. It looks starting 6.5 kernel , MTU is bumped to
64K. We lowered this to 16K
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 2689657 bytes 17458642228 (16.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2689657 bytes 17458642228 (16.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Oracle Linux: ORA-27301:OS Failure Message: No Buffer Space Available (Doc
ID 2041723.1)
I followed up with Oracle to see, in what way loopback interface is
contributing to the issue here, but they failed to give more details saying
it's OS specific and they don't have idea on the same. :(
Appreciate if someone has some previous experience working on similar
issue provide some information related to this.
BR,
Rakesh RA
