Too many open files...

Discussion in 'Plesk 6.0 Troubleshooting and Problems' started by Dozer42, Oct 14, 2003.

  1. Dozer42

    Dozer42 Guest

    We ran into a nasty problem last night. Apache died a horrible death, and when we attempted to restart it we got the following errors:

    (24)Too many open files: apr_accept: (client socket)

    make_obcallback: could not import mod_python.apache.!

    (24)Too many open files: Init: Can't open server certificate file /usr/local/psa/var/certificates/certXV5Ld1w

    After a bit of investigation, the only way to get Apache back up and running was by deleting about 20 domains manually in /etc/httpd/conf/httpd.include. We finally found out Plesk never bothered to increase the max number of file descriptors from the default of 1024, which can get eaten up really fast with less than even 300 domains.

    It took a while, but we were finally able to recompile the httpd binary and raised the limit to 16384. I've made the file available for download if anyone wants it at: ftp://ftp.netwiz.net/httpd.gz

    It's the binary/executable only, compiled on a RH9 system/i386. I make absolutely no guarantees whatsoever with regards to it, but it might save other Plesk users some trouble. You will also need to add 'ulimit -n 16384' towards the beginning of /etc/rc.d/init.d/httpd so that it changes the linux limit each time the server boots or httpd restarts.

    If you're on a non-RH9 system or just don't want to use my binary, the fix would be to:

    download the src .rpm file from ftp.redhat.com (in this case ftp://ftp.redhat.com/pub/redhat/linux/9/en/os/i386/SRPMS/httpd-2.0.40-21.src.rpm)

    Install the gcc compiler and dependant files if you do not already have it loaded (we did the default RH9 server install, and it does *NOT* have the compiler by default).

    Run 'rpmbuild --rebuild httpd-2.0.40-21.src.rpm' (this will take a while, and you'll need to download another ton of dependant files).

    Grab your updated RPM from the /usr/src/redhat/RPMS directory and install it.

    In our case we found it easier to recompile the rpm on another RH9 machine we had that had gcc already installed, installed the new RPM there, and just grabbed the httpd binary alone and popped that onto the Plesk server in /usr/sbin and of course restart apache after editing /etc/rc.d/init.d/httpd.

    I find it completely and utterly inexcuseable that Plesk releases software that CANNOT do what they specifically advertise it will do. They have 10 user, 100 user, 300 user, and 'unlimited' user versions, and as shipped it can't even touch 300 users without puking and dying a horrid death.

    I don't expect to be able to host 50,000,000 domains on one machine, but at least it should be able to handle 300 without dying!!!

    And this is a known problem, the same thing happened with Plesk 5, 2.5, etc from what I can tell from looking through the forum. Plesk has full knowledge that their product does not, and CANNOT work for it's intended purpose, yet has no fix available, and wanted to charge us hundreds of dollars to recompile httpd!!!

    To me, this is blatant fraud. I also can't believe they release Plesk 6 w/o any sort of backup solution. Not to mention at least 10 issues we've had with proftpd, them not releasing a fix for a known root exploit, and downloading the fixed proftpd has broken other things for us! One of our users can't even LIST the directory in /httpdocs/images without the ftp client crashing!

    Man Plesk has been a real disappointment. We've got it working, but just barely, and with no real backup solution. I really think we made a mistake spending $1000 on broken software. :\
     
  2. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    You did read the FAQ right?

    http://www.plesk.com/Support/faq/psa/apache.htm#q18

    1. Run sysctl -w fs.file-max=16384
    2. Edit /etc/sysctl.conf. Add a new line with: fs.file-max=16384
    3. Edit /usr/local/psa/apache/bin/apachectl. Find the line that says ulimit -n 8192 and change it to ulimit -n 16384.
    4. Restart apache: /usr/local/psa/admin/sbin/my_apci_rst.


    While I can understand your frustration, blaming them for a problem in a package they dont even make is a bit excessive. I only count 1 issue with proftp, which they did fix, and the backup utilities were released a few weeks ago.
     
  3. Dozer42

    Dozer42 Guest

    Of course I read the FAQ, and as you know the solution in the FAQ does not resolve the problem. It's probably not even written/updated for Plesk 6. There's no way it can work until Apache is recompiled.

    It's not a 'problem' in a package they don't even make. It's a built in limitation that they're quite well aware of, and they know 100% that the Plesk server will not work for the purpose it's intended for until this limit is changed, and they're trying to charge the customer even more money to make Plesk work for it's original intended purpose.

    It cannot support even 300 domains with only 1024 file descriptors, much less any more than that. It certainly should seeing as we paid extra for the 'unlimited' domain package. If they need to change one whole line and recompile the httpd package for distribution with Plesk, they should DO IT. That's like a car company selling you a car that won't even get up to freeway speeds without crashing, it does not work for it's intended purpose.

    As for proftpd, it's horrible as they have it configured out of the box.

    A> The root exploit is unacceptable.

    B> Not issuing a fix for the root exploit in under 24 hours is unacceptable.

    C> Not issuing a working fix for weeks after the exploit is unacceptable. As it seems to be well known, the fix for the root exploit breaks other things, many of our users now cannot even go into a directory and list the files in there!

    D> The transfer mode defaulting to binary and 'corrupting' CGI's is just crazy. We've run webservers and FTP servers for over 10 years, and never encoutered anything like this.

    E> The nasty little glitch of showing clients timestamps based on GMT is stupid. Yes people should have that option, but to turn that on by default is nuts.

    F> Performance out of the box is far slower for proftpd than the other ftp daemons we've used due to ReverseDNS lookups and ident lookups being turned on by default.

    G> By default dotfiles are hidden via FTP, but not on the file manager or anywhere else. Great security feature if someone wants it, but it had our users pulling their hair out until we found out a way to turn this 'feature' off.

    I don't think I'm out of line here in thinking that a root exploit, broken functionality, and a ton of other glitches is just way too much for just one daemon out of an entire server. Why should I spend all of my day tracking down things that Plesk should have solved/configured properly from the start?

    Now I've got to go fix their problem with mailman and mailing lists, which is broken out of the box. Get postgresql running which didn't work out of the box. Find out why Tomcat is still showing application errors. And track down a ton of other issues that just continue to pop up with this buggy software. :(
     
  4. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    And how exactly is SW Soft supposed to fix a software package they dont make? I dont understand how you cant make this their problem when they cant do anything to fix it. Incidentally, had you looked around you'd see that both 4PSA and I have produced RPM's that fix this issue.

    http://www.atomicrocketturtle.com/m...ons&file=index&req=viewarticle&artid=6&page=1
    I understand your frustration here, I really do, and I went so far as to fix this one myself. But the folks you should be complaining to are redhat, they're the ones that made the package.
     
  5. Dozer42

    Dozer42 Guest

    I'm not trying to yell at you, I think you're a great guy for going to all the hard work you do and release things the Plesk users need that Plesk isn't releasing and should be.

    I did see the link that you've posted before, but of course that's for Apache 1.3.27 and Plesk 5.X and not Apache 2.0+ and Plesk 6, which is exactly why I left the compiled version available for download for anyone else whom it might help.

    Who cares if they created the package or not? They use it in Plesk, they say Plesk will do over 300 domains, it will not, and cannot as they ship it. All they have to do is change one number, recompile the RPM, and ship that one instead. They've recompiled and changed other packages, why not do the same for httpd which is the absolute core of their entire product offering?!?

    There simply is no excuse for shipping a product that cannot do what it was intended to, and then trying to rip the customer off by making them pay hundreds of dollars to make the product do what it should have right out of the box.
     
  6. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    Its the exact same method to fix it, edit this:

    /usr/include/linux/posix_types.h

    change this line:

    #define __FD_SETSIZE 1024

    to say, 4096, or 8192 (you can go higher, but there are big performance hits), and then --rebuild the apache src.rpm, and you're in business.

    Certainly SW Soft could do this, and release an RPM, but thats only half the problem. The other half is to tune the OS itself, by either upping the sysctl open files, or just echo straight to /proc/sys/fs/file-max. Then you're at the mercy of yet another vendor for updates.

    Architecturally, when you look at the way apache itself put together this package they took the bizzare step of setting max open files at compile time, typically this is the kind of thing you'd control with ulimit on a per process basis. Repackaging it is one solution, but long term, the real fix is to fix the application.
     
  7. lch_hosting

    lch_hosting Guest

    I have to say I agree with Dozer42 on this. I have had to spent serveral hundred dollars to have outside help come in a fix apache after upgrading to an RPM build from a standard build (this happened 2 times so far, I will be upgrading again shortly so soon to be 3 and I still don't know how to fix it myself). Plesk should read the key and based on the number of domains that the key supports setup apache using that (for example if it's a 10 doman key, setup an optimal apache version for a 10 domain setup, 100 domain key..... etc etc..). I am spending almost as much as I am on the Plesk Licesnce as a am on fixing this after upgrading. If I can't find a fix for this (or plesk doesn't) looks like CPanel here we come.
     
  8. lch_hosting

    lch_hosting Guest

    I mean hell.. Even just add a question to the auto installer such as how many sites do you plan on hosting say something like

    1.) 10
    2.) 100
    3.) 300
    5.) 500
    6.) 1000
    7.) 1000+

    And setup apache using that information. I guess it's just really frustrating to spend hours upgrading, then having to spend 3 more hours looking around trying to fix apache (while the server is down) and eventually having to give up and pay an arm and a leg to have someone come in and fix it for you.
     
  9. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    Thats all well and good to propose something like that, but the problem isnt that simple to solve from an engineering perspective. The whole file descriptor issue is just one very small (but very noticible) part of this issue. High file descriptor use environments will cause all sorts of subtle failures in other applications (example, fsockopen in php 4.3.x fails when youre using over 1024 FD's). Thats hardly the only application that breaks in extremely wierd ways. You're pushing the OS into a realm where it just isnt built to go. Another example is tomcat, just watch what happens when it cant open enough threads (hint, watch the temperature on your CPU. I had one run up to 85C for 3 days before it melted the motherboard)

    Do you really want to have someone make changes to your applications that can start a fire? :) I dont.
     
  10. lch_hosting

    lch_hosting Guest

    Hi,
    The setup they had in the standard version works great. I don't see why they didn't keep it like that. Your telling me that setup could cause fires?
     
  11. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    yes :) Well not so much a fire as the CPU was so hot for so long it actually damaged the motherboard physically. It could have started a fire if there was something flamable in there. Mainly it was one of those situations where it let out the magic blue smoke.
     
  12. hardweb

    hardweb Guru

    Messages:
    3,558
    This thread is funny. :) I am still wondering why Microsoft sells Windows without a standard MS computer attached. How can Windows work without any problems on a non-MS computer? They should fix this ASAP!
     
  13. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    it is an absurd thread indeed :)
     
  14. hanes

    hanes Mega Poster

    Messages:
    108
    recompiling apache

    Hi,

    Only our latest two machines are RPM versions , i need to recompile apache on both of them.

    This is just not working out though

    PSA6.02 and RH9

    Im using the instructions on

    http://www.plesk.com/Support/faq/psa/apache.htm#q18

    Since there is no FD_SETSIZE in /usr/include/bits/types.h

    i use /usr/include/linux/posix_types.h

    then recompile with

    rpmbuild --rebuild httpd-2.0.40-21.5.src.rpm

    rpm -Uvh --test /usr/src/redhat/RPMS/i386/httpd-2.0.40-21.5.i386.rpm

    cp /usr/sbin/suexec /usr/sbin/suexec.plesk

    rpm -Uvh --force /usr/src/redhat/RPMS/i386/httpd-2.0.40-21.5.i386.rpm

    cp /usr/sbin/suexec.plesk /usr/sbin/suexec

    /usr/local/psa/admin/sbin/my_apci_rst -a -v

    service httpd restart

    This is not working out still,

    Errors in the log fd_setsize to small.

    Can someone help me out here with recompling Apache
    for RH9

    Regards.

    Hanes
     
  15. Dozer42

    Dozer42 Guest

    Please feel free to download mine, I'm happy to share them. :)

    ftp://ftp.netwiz.net/plesk

    There's 4 files in there that are recompiled, you might need to use --force to get them to install over the old versions.

    Also be sure to save the old /usr/sbin/suexec of about 15k bytes and reinstall it afterwards, or all your cgi-bins will be pooched!

    Other than that, follow the rest of Plesk's instructions and you should be good to go.

    These files of course are not warrantied or supported in ANY way.

    Make sure to save /usr/sbin/httpd and the .rpms as httpd will get overwritten when Plesk is upgraded say to version 6.0.3.
     
  16. hanes

    hanes Mega Poster

    Messages:
    108
    link not working

    The link is not working,

    but i rather have some tips on recompiling apache.

    Regards,

    Cor
     
  17. Dozer42

    Dozer42 Guest

    The link has been fixed. :) I accidentally put in www.netwiz.net instead of ftp.netwiz.net! Please share your tips on recompiling, I'm sure they'd help people too!
     
  18. hanes

    hanes Mega Poster

    Messages:
    108
    answer by plesk

    Yes, that FAQ is for RedHat 7.2 and 7.3. I think we should modify it for
    RedHat 9. Thank you very much for reporting that to us.

    But about the problem. At first, please make sure that the following RPM
    already installed on your server:
    glibc-devel-2.3.2-27.9.7 (version number could be a little different)

    After that, please edit the following file:
    /usr/include/bits/typesizes.h

    #define __FD_SETSIZE 32768

    Then download httpd-2.0.40-21.5.src.rpm and recompile it using
    "rpmbuild" command. Please, submit new support ticket via our Online
    Support Form (https://www.plesk.com/Support/form.php) if you have
    further questions.

    --
    Thanks,
     
  19. Ales

    Ales Guru

    Messages:
    867
    Just a side note:

    IMHO, that's incorrect. Plesk rpm will not alter your httpd in any way... but it doesn't hurt to have backups :)

    It might alter it only if you use some kind of an autoinstaller and there is a newer version of Apache available than the one you have installed... don't know about that, since I don't use autoinstallers. But you'd probably install and recompile Apache it on your own before that.
     
  20. helius

    helius Guest

    Hi!

    Can anyone here help with some advise?

    Before reading this enlightening thread I upgraded httpd ... then immediately had a problem with the 1024 file limitation. So I disabled all SSL service on the server and am saved for the day -- but that's about it. In addition, I missed to save the old Plesk suexec file. So I did about everything wrong that there was to do wrong (cgi isn't working now).

    Questions:

    (1) When I recompile httpd with a higher file size limitation ... would you guys recomment to go with the (not anymore latest) version posted by Dozer42 (ftp://ftp.netwiz.net/httpd.gz ... OR ftp://ftp.netwiz.net/plesk) or should I just download the lastest httpd version somewhere and follow the advise of hanes and edit the file /usr/include/bits/typesizes.h
    What's best and/or easiest?

    (2) Since I messed with the suexec file -- can I just copy that over from another RH 9 Plesk 6.0.2 server ?? Any other way to repair that?

    I'd appreciate your input.

    John
     

Share This Page