Fatal resource shortage: privvmpages

Discussion in 'General Questions' started by futureweb, Nov 22, 2016.

  1. futureweb

    futureweb Tera Poster

    Messages:
    394
    Hey there,
    lately I got lot's of "Fatal resource shortage: privvmpages, UB xxx" Errors for one of my CTs. But within CT it looks like enough Memory is free?

    root - /var/log/messages
    CT:
    Also no Memory Shortage noticeable within my Monitoring - upper = Root Server, lower = CT:
    memory1.jpg

    CT Config:
    as you can see - no privvmpages Limit is set?!

    Any ideas how to find the cause of those Problems?

    thx
    Andreas Schnederle-Wagner
     
    Last edited: Nov 22, 2016
  2. Pavel

    Pavel A.I. Auto-Responder Odin Team

    Messages:
    432
    Hello Andreas,

    "privvmpages" is a limit for memory allocation.
    When memory allocated but not touched it is not considered used yet - you will not see it on the graphs.
    Some processes, like apache, aggressively allocate memory, but usually does not utilize it that much. For such containers it would make sense to increase memory allocation limit through VM_OVERCOMMIT from default 1.5 to 3.0.

    As for the monitoring - it can be done from container with "ps aux", or from the node with "vzps -E %ctid% aux".
    You should take a look at rsz and vsz.
    RSZ - used memory. VSZ - allocated memory (both touched and untouched).

    [root@pcs-3rdline-7 ~]# vzlist 100
    CTID NPROC STATUS IP_ADDR HOSTNAME
    100 19 running 172.16.11.100 -
    [root@pcs-3rdline-7 ~]# vzctl exec 100 free -m
    total used free shared buffers cached
    Mem: 512 20 491 0 0 7
    -/+ buffers/cache: 12 499
    Swap: 512 2 509
    [root@pcs-3rdline-7 ~]# vzps axo pid,rsz,vsz -E 100
    PID RSZ VSZ
    103581 1280 19236
    103582 0 0
    103583 0 0
    104718 364 10644
    105710 1284 185664
    106294 380 66608
    106413 668 22184
    106847 292 66796
    106848 40 66796
    107109 828 83044
    107194 744 78636
    111563 3716 175644
    111639 1248 117304
    111682 524 4068
    111683 524 4068
    817226 2120 175644
    [root@pcs-3rdline-7 ~]#

    That should help you identify the process which abuses memory allocation.
     
  3. futureweb

    futureweb Tera Poster

    Messages:
    394
    Hi Pavel,
    thx for info - what would be the "standard" privvmpages if it's disabled within config?

    config.jpg

     
  4. Pavel

    Pavel A.I. Auto-Responder Odin Team

    Messages:
    432
    Hello,

    privvmpages is calculated based on your physpages, swappages and VM_OVERCOMMIT.
    privvmpages = ( physpages + swappages ) * VM_OVERCOMMIT
    if VM_OVERCOMMIT is not set, default value is used, 1.5.
    That is why I suggested to adjust VM_OVERCOMMIT right away
     
    futureweb likes this.
  5. futureweb

    futureweb Tera Poster

    Messages:
    394
    alright - changed from
    VM_OVERCOMMIT="1.5"
    to
    VM_OVERCOMMIT="3"

    Hope that's it!

    thx for your fast & superior Help ... as always! ;-)

    Andreas
     
  6. Pavel

    Pavel A.I. Auto-Responder Odin Team

    Messages:
    432
    Always glad to help you!
    Did you change the config line manually? If yes - container restart is required, otherwise setting will not be applied.
    It can be configured using vzctl:
    # vzctl set 111 --vm_overcommit 3 --save
     
  7. futureweb

    futureweb Tera Poster

    Messages:
    394
    thx 4 pointing that out - but I always change CT configs with vzctl or within PVA ;-)
     
  8. futureweb

    futureweb Tera Poster

    Messages:
    394
    Still getting some "Fatal resource shortage: privvmpages, UB xxx" with VM_OVERCOMMIT="3" ... guess it's no problem raising it to 6 or even higher? Or are there downsides on this?
    Memory heavy Apache / PHP CMS running within this CT.
     
  9. Pavel

    Pavel A.I. Auto-Responder Odin Team

    Messages:
    432
    Andreas, it might be bad if all containers on your server behave like that - there's a limit for allocations in kernel as well :)
    But if it's just one container - I guess you can set it higher.

    Just make sure privvmpages is not set explicitly - if privvmpages is already set then changing vm_overcommit value won't change anything. You can check /proc/user_beancounters to make sure limit changes when you adjust vm_overcommit.
     
  10. futureweb

    futureweb Tera Poster

    Messages:
    394
    It's the only CT on this Root Node I'm getting those Errors. privvmpages is not set explicitly - and beancounters did raise on last change I guess. ... Already very High ... not excactly sure why it still Hits Limit sometimes ... :-/
    PHP:
    [root@blxx log]# cat /proc/user_beancounters
    Version2.5
      uid  resource  held  maxheld  barrier  limit  failcnt
      xxx
    :  kmemsize  1142544375  1146478592  9223372036854775807  9223372036854775807  0
      lockedpages  207566  207582  4194304  4194304  0
     
    -->  privvmpages  6236443  6293205  37748736  37748736  1543 <--
      
    shmpages  32979  32979  9223372036854775807  9223372036854775807  0
      dummy  0  0  9223372036854775807  9223372036854775807  0
      numproc  158  249  9223372036854775807  9223372036854775807  0
      physpages  2856217  2865180  4194304  4194304  0
      vmguarpages  0  0  6291456  6291456  0
      oomguarpages  446503  452115  4194304  4194304  0
      numtcpsock  195  249  9223372036854775807  9223372036854775807  0
      numflock  10  36  9223372036854775807  9223372036854775807  0
      numpty  1  1  9223372036854775807  9223372036854775807  0
      numsiginfo  0  54  9223372036854775807  9223372036854775807  0
      tcpsndbuf  9043104  11319992  9223372036854775807  9223372036854775807  0
      tcprcvbuf  10831816  12216392  9223372036854775807  9223372036854775807  0
      othersockbuf  62424  70880  9223372036854775807  9223372036854775807  0
      dgramrcvbuf  0  0  9223372036854775807  9223372036854775807  0
      numothersock  64  67  9223372036854775807  9223372036854775807  0
      dcachesize  974097674  974345737  9223372036854775807  9223372036854775807  0
      numfile  1625  1855  9223372036854775807  9223372036854775807  0
      dummy  0  0  9223372036854775807  9223372036854775807  0
      dummy  0  0  9223372036854775807  9223372036854775807  0
      dummy  0  0  9223372036854775807  9223372036854775807  0
      numiptent  4  4  9223372036854775807  9223372036854775807  0
     
  11. Pavel

    Pavel A.I. Auto-Responder Odin Team

    Messages:
    432
    I'm afraid it's not a virtuozzo fault then :( It seems that code must be optimized
     
  12. futureweb

    futureweb Tera Poster

    Messages:
    394
    is it 37748736 BYTES for privvmpages? So it's ~4GB now - is this right? Or is it some other Metric? (4k pages?)
     
  13. Pavel

    Pavel A.I. Auto-Responder Odin Team

    Messages:
    432
    No, those are not bytes. Those are pages. counter name has "pages" in it :)
    Each page equals to 4 kB, thus it's... 144 Gbs o_O
    Emm... I'm speechless...
     
  14. futureweb

    futureweb Tera Poster

    Messages:
    394

Share This Page