make_sock: could not bind to port 443

Discussion in 'PSA 5.0 Troubleshooting and Problems' started by Redah, Mar 14, 2003.

  1. Redah

    Redah Mega Poster

    Messages:
    130
    This is basically all about an earlier topic by some other people, located at:

    http://forum.plesk.com/showthread.php?threadid=5440

    Basically, when we change somethign in PSA (5.0.5) Apache might catch a sigterm, quit, and won't restart since it thinks something else is bound to port 443.

    We never had this problem in PSA 2.x (standard) or when we were running the standard PSA 5 on RedHat 7.2

    Now we have 7.3 which means we could only get the RPM install, and now problems like this arise.

    Will there be a definate fix for this, or is a patch in the works? Because frankly, I do not have time to check for a working Apache every single time I've been in the Control Panel. This is a major downside I've found in PSA now, whereas I used to be pretty satisfied.

    The provided solution to restart Apache with a cron doesn't work, nor does manually restarting Apache, or PSA alltogether. Usually only a reboot seems to fix it.

    Plesk, any updates on this?
     
  2. hraiser

    hraiser Guest

    Did you ever get a solution to this, as we have noticed sometimes that apache shuts down with this in the error_log

    [Tue Apr 22 18:41:03 2003] [warn] child process 16937 still did not exit, sending a SIGTERM
    [Tue Apr 22 18:41:07 2003] [warn] module mod_frontpage.c is already added, skipping
    [Tue Apr 22 18:41:07 2003] [crit] (98)Address already in use: make_sock: could not bind to port 443
     
  3. Redah

    Redah Mega Poster

    Messages:
    130
    None whatsoever, seems to be a bug more people had, let's just hope it's fixed in 6.0

    On my new server (with 5.0.5 RPM) I haven't had the problem yet btw.
     
  4. hraiser

    hraiser Guest

    Thanks for the reply.
    I just made a little bash script up to check httpd is running every 15 minutes, and if its not restart it. Works but a bit of a bodge :)
     
  5. jimroe

    jimroe Guru

    Messages:
    2,067
    You should do a bit more than just restart Apache - concerning is the error:

    Address already in use: make_sock: could not bind to port 443

    With Apache shut down there shouldn't be anything listening on port 443 that would generate this error when you go to start it. If something is, you need to find out what it is - because it could be a bindshell trying to hide behind Apache.

    The error message:

    [Tue Apr 22 18:41:07 2003] [warn] module mod_frontpage.c is already added, skipping

    is the result of the FrontPage RPM not loading the module correctly - the httpd.conf file needs to be modified to load the frontpage DSO the same way Apache loads the others DSO modules - it can be the last one in the load module and add module lists.
     
  6. hraiser

    hraiser Guest

    Thanks for the reply.

    regarding the port 443, do you have any suggestions on the best method to look for what is trying to use port 443 as well.
    This is a new 5.x rpm install on 7.3, with firewall and ssh access is restricted to only 2 ip addresses which can connect to it, this being our own, so the possiblility of this being compromised is very small. Just got us a bit baffled.

    Regarding FrontPage, the only mention of FrontPage in the httpd.conf is at the bottom which reads,

    <IfDefine HAVE_FRONTPAGE>
    LoadModule frontpage_module modules/mod_frontpage.so
    AddModule mod_frontpage.c
    </IfDefine>

    Is this then incorrect ?
     
  7. jimroe

    jimroe Guru

    Messages:
    2,067
    Two things you might do:

    1. Check to see if all Apache children exit when you shut down httpd (ps -e | grep httpd) - if one hangs on shutdown, that might be causing the error when you go to start - all Apache processes should exit when you stop httpd.

    2. With httpd shut down, use netstat -nl to see if there's anything still listening on port 443 - there shouldn't be. If there is, then you need to find out what it is and where it came from.
     
  8. hraiser

    hraiser Guest

    Thanks for the info on this.
    This was clear on both counts, which is good news.

    Could it sometimes just be a case of httpd tripping over itself ? Maybe the issuing of a sleep command in the istart files etc, might prevent this from happening.
     
  9. jimroe

    jimroe Guru

    Messages:
    2,067
    The probability is that an Apache thread using SSL has hung, and when you attempt to restart Apache it finds that something (like this thread that didn't end) is already listening and so it fails. When you ran the test, Apache exited cleanly and left nothing running, so it started OK. This error message from your initial posts seems to show this:

    [Tue Apr 22 18:41:03 2003] [warn] child process 16937 still did not exit, sending a SIGTERM

    Aparrently the SIGTERM didn't stop it, and so the error on the restart.

    You might look for some site that is doing something not quite right with SSL and hanging Apache when than pages loads - or attempts to load.
     
  10. hraiser

    hraiser Guest

    I will look into this. Again, many thanks for your time on this matter, much apprec. :)
     
  11. Traged1

    Traged1 Guru

    Messages:
    1,085
    Ok, ya we got the smae thing going on over here, and it seems to be some sort of an attack against SSL, but our ssl is up2date?

    Look:
    -------------------------------------------------------------------------

    [Mon Jun 2 16:44:21 2003] [notice] child pid 9133 exit signal Segmentation fault (11)
    [Mon Jun 2 17:16:22 2003] [error] [client 211.139.36.180] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Mon Jun 2 17:44:15 2003] [error] [client 199.35.212.97] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    sh: /uptime: No such file or directory
    sh: /uptime: No such file or directory
    [Mon Jun 2 18:35:18 2003] [error] [client 206.217.9.231] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Mon Jun 2 19:27:56 2003] [error] [client 81.72.45.43] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Mon Jun 2 19:57:13 2003] [error] [client 68.62.57.82] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Mon Jun 2 20:36:52 2003] [error] mod_ssl: SSL handshake failed (server default:443, client 206.71.120.1) (OpenSSL library error follows)
    [Mon Jun 2 20:36:52 2003] [error] OpenSSL: error:140943E8:lib(20):func(148):reason(1000)
    [Mon Jun 2 21:19:31 2003] [error] [client 199.243.136.215] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Mon Jun 2 21:43:14 2003] [error] [client 81.2.119.210] File does not exist: /home/httpd/vhosts/default/htdocs/robots.txt
    [Mon Jun 2 21:43:14 2003] [error] [client 81.2.119.210] File does not exist: /home/httpd/vhosts/default/httpsdocs/robots.txt
    [Mon Jun 2 22:37:01 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:37:03 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:37:03 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin

    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 22:43:55 2003] [error] [client 211.240.124.254] File does not exist: /home/httpd/vhosts/default/htdocs/sumthin
    [Mon Jun 2 23:16:06 2003] [error] [client 218.104.4.121] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Tue Jun 3 00:31:21 2003] [error] [client 220.170.66.37] File does not exist: /home/httpd/vhosts/default/htdocs/default.ida
    [Tue Jun 3 00:50:18 2003] [error] [client 210.69.138.109] script not found or unable to stat: /home/httpd/vhosts/default/cgi-bin/formmail.cgi
    [Tue Jun 3 00:53:45 2003] [warn] child process 9124 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 9125 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 9135 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 9136 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 9141 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 9149 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 9157 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 24076 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 29678 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 27082 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 29951 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 30649 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 29730 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 30662 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 29972 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 31362 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:45 2003] [warn] child process 31477 still did not exit, sending a SIGTERM
    [Tue Jun 3 00:53:51 2003] [warn] Loaded DSO /opt/casp/module/linux2_optimized/apache_1.3.27/eapi/mod_casp2.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
    [Tue Jun 3 00:53:51 2003] [crit] (98)Address already in use: make_sock: could not bind to port 443

    ---------------------------------------------------------------------------


    Seems to be the culprit!

    Can anyone make any further sence out of this?

    I've researched this ip and it appears to be one of our clients ip's. So I checked into his ssl_access.log for his site, and there is nothing for that date, I then checked the the error_log and still no entry for this ssl call.

    So I go into this vhost access.log and find this at that time:




    I just do not know what to figuire of this, but it is a real drag.

    About the bindshell, I have a rootkit checker that checks for signs of common rootkits and then emails me the results, it is run through cron twice every day, and it always tells me:

    But this has been reported as a false positive on redhat systems, not sure why. But now Im begining to wonder if it is false?

    Anyhow I shutdown httpd and do ps -e | grep httpd

    it returns nothing,

    I do the netstat -nl , nothing running on port 443 or 465.

    However every once and awhile our server will just crash with this warning, and it is always in the middle of the night!

    It is causing a big problem, and I need to figuire out this one, really I do. Any help would be greatly welcomed and appreciated.

    :confused:
     
  12. AWSolutions

    AWSolutions Guest

    I have had similar problems :-(


    Don't know what to do!

    -Brian
     
  13. Traged1

    Traged1 Guru

    Messages:
    1,085
    It was not what I though it may have been as stated above, as that client has left and his account was terminated, however the problem still presists :(


    Here is what is in our httpd.conf file:

    My question is; should I also comment out the <IfDefine HAVE_SSL> directive? It does say Listen 443 ?? Could this be the source of the trouble? Or do I need that directive?
     
  14. Traged1

    Traged1 Guru

    Messages:
    1,085
    Ok, I tested out removing that directive, and it stops my server from listening for ssl calls, so I think i am safe to leave it in there.

    However, Im now not sure about this:

    # SSL Session Cache:
    # The cache speeds up processing of multiple parallel requests from
    # the same client.
    SSLSessionCache shm:/var/cache/ssl_gcache_data(524288)

    Because if there was a mal formed request sent to the cache and the server had trouble looking it up at the time of a apache restart, then maybe that could cause a stalling of the child pid to the SSLSessionCache ? Not sure, just an idea, I am commenting out the SSLSessionCache directive anyhow, to see if the problem will occurr again.

    Also, has anyone actually done the netstat -nl command when thier apache crashes from this, and not just when they shutdown the apache service manually themselves?
     
  15. Traged1

    Traged1 Guru

    Messages:
    1,085
    Well this seems to help, but as I researched this problem, I stumbled accross another piece of info that may play a big part in this crash.

    Please read this page it details an openssl-0.9.6e + GET_CLIENT_HELLO exploit

    http://cert.uni-stuttgart.de/archive/incidents/2003/02/msg00082.html

    This exploit or a variation of this exploit may very well be responsible for the server crashes we have all been expiencing!

    I would assume that redhat would patch this and Im not sure if it also applies to openssl-0.9.6-16 which is the curreny up2date version of our openssl rpm.

    For now I am going to begin blocking all ofending ips through firewall.
    :(
     
  16. Traged1

    Traged1 Guru

    Messages:
    1,085
    Yep, this could very well be the heart of the problem, lets see;

    The attacker hits the openssl with the child jammer request of "GET /sumthin HTTP/1.0"

    Multiple apache child processes get wedged together and stall, during this time the intruder proceeds to establish some sort of scanning mechanism that binds itself to port 443 taking up the place of the stalled child PID's in an attempt to conceal itself. But just at that moment a change is made via PSA control panel and apache restarts.

    Thus it exits with the [crit] cannot bind to port 443 error, because the apache scanner is still bound port 443. It takes a reboot to kill all processes and thus clearing the port again for normal use.

    Now that I think I have found the source of the bug, does anyone have any idea's on how to prevent it from happening?
     
  17. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    Have you tried upgrading openssl? I believe redhat released a fixed rpm for this already.
     
  18. Traged1

    Traged1 Guru

    Messages:
    1,085
    I run up2date to update our RPM's every 24 hours. It is telling me that our server is completely up2date but our ssl version is still 0.9.6-16 I know that there is an openssl 0.9.7 which is apperently immune to this exploit, but I run RPMs and have to wait for redhat to put the updated version for our distro on up2date.
     
  19. atomicturtle

    atomicturtle Product Expert

    Messages:
    3,812
    redhat will often backport in features and fixes from later versions. So its possible that the fix is already in that version of openssl. However, if there is enough interest, I can make an openssl 0.9.7 rpm and put it up on my site.
     
  20. Traged1

    Traged1 Guru

    Messages:
    1,085
    I just was responded to by PLESK Support, this work around is suppost to kill the scanner, and the child pid'd so that at least we can restart without having to reboot. For those of you interested please try it.

    We are trying it now, and I know that I've seen this solution somewhere else for a similar issue.

    Hope this will work for now, if it does not, I will try to work with you on the rebuilt openssl.

    Thanks for the help/offer atomicturtle. I am not in any way trying to put down Redhat as we are pro REDHAT all the way.
     

Share This Page