Urban planning community

+ Reply to thread
Results 1 to 11 of 11

Thread: Server issues: an update

  1. #1
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,573
    Blog entries
    3

    Server issues: an update

    I'm trying to keep the creaking VPS alive until I'm in a financial state where I can upgrade. (Still have to sell my house in suburban Cleveland, unfortunately.)

    Cyburbia was getting hammered by a South Korean search engine bot. I banned the IP addresses, and the server seems to be operating much smoother; still on the slow side, but not as bad. You'll also probably notice that the "Users Online" count will be much lower than usual.

    Thanks for your patience these past few months. You've all been great!
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  2. #2
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,573
    Blog entries
    3
    For some reason, Cyburbia seems to be the only site getting absolutely hammered with spider bots from Daum. (There's no other reports of aggressive Daum activity online, or at least none that I could find.) Daum is supposedly a legitimate Korean search engine. They're been hitting Cyburbia every half second to five seconds, and they ignore robots.txt, a file that contains instructions for search engines that visit the site. In the past week, Daum bots account for about 1/4 to 1/3 of all traffic on Cyburbia.

    I tried to drop Daum IP blocks in iptables, but unfortunately for some reason iptables on my server is flushed when the server is restarted. Daumoa/2.0 bots are hitting the Cyburbia server so hard I think it's been a major factor in the recent, frequent server crashes.

    IP ranges for the Daum bots include:

    61.247.222.0/24
    116.125.140.0/24
    116.125.141.0/24
    211.233.79.0/24
    222.231.40.0/24
    222.231.42.0/24

    I think that's all of them. When I tried banning a range early on, the bots would move to a different IP block.

    I resorted to blocking all of these ranges in .htaccess. This means results for Cyburbia will drop in South Korea's leading search engine. However, Cyburbia does not rank very highly in searches for "urban planning" in Daum, and there have been very few active users from South Korea on Cyburbia through the years.

    Also, there was a line in .htaccess that somehow caused the web server to perform domain name lookups on every visitor to the site, despite instructions in the server configuration not to do so. This causes the server to slow down. The errant line in .htaccess was removed, so there will no longer be domain name lookups.

    I added a script that will check every minute to see if Apache (the Web server program) is running. If it is not, it will restart Apache automatically.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  3. #3
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,573
    Blog entries
    3
    Apache is still crashing. A lot. It automatically restarts, but it shouldn't be crashing to begin with. I'll keep y'all posted about what I'm trying to do to fix it.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  4. #4
    Cyburbian The One's avatar
    Registered
    Mar 2004
    Location
    Where Valley Fever Lives
    Posts
    7,303

    ahh.....

    THEY ARE WATCHING US
    Skilled Adoxographer

  5. #5
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,573
    Blog entries
    3
    Quote Originally posted by The One View post
    THEY ARE WATCHING US
    Maybe it's the North Koreans that are DOSing Cyburbia?



    Anyhow, I tried another configuration for httpd.conf. Let's see if that stops the crashing.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  6. #6
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,573
    Blog entries
    3
    Nope, that didn't help.

    EDIT: It looks like the server can either be fast and efficient with memory and load, but crash-prone, or slow and inefficient, but more reliable. I'm trying settings for the latter, and seeing how that works.

    In the coming weeks, Cyburbia should be moving to a new VPS at the same host. We desperately have to upgrade to MySQL 5, but that means the Drupal integration -- the front page, feeds, and so on -- will break. For a while after the move, Cyburbia will consist of just the forums and gallery, until I can reintegrate the site with Drupal and re-enter, by hand, all the articles and blog/RSS feed information.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  7. #7
    Cyburbian Plus Salmissra's avatar
    Registered
    Aug 2004
    Location
    Texas
    Posts
    4,088
    I freely admit, I'm not that versed in behind-the-scenes stuff on computers. I'm the typical end-user. I want to turn the dang thing on, and it should work. When it doesn't, I call either Tech Support (at work) or Hubby (at home). So the details of what you're going through with these server problems just goes in one eye and out the other.

    These problems sound like they are starting to get on your last nerve, and that's not good for you. Hopefully the upgrade/whatever else needs to happen, happens soon, for your sanity!

    You take very good care of us here on Cyburbia, and I appreciate it.

    Kudos, and Good Luck!
    "We do not need any other Tutankhamun's tomb with all its treasures. We need context. We need understanding. We need knowledge of historical events to tie them together. We don't know much. Of course we know a lot, but it is context that's missing, not treasures." - Werner Herzog, in Archaeology, March/April 2011

  8. #8
    Cyburbian natski's avatar
    Registered
    Jun 2005
    Location
    In my own little bubble
    Posts
    2,562
    Quote Originally posted by Salmissra View post
    You take very good care of us here on Cyburbia, and I appreciate it.

    Kudos, and Good Luck!
    I second this, thanks for everything Dan
    "Have you ever wondered if there was more to life, other than being really, really, ridiculously good looking?" Zoolander

  9. #9
    Cyburbian Emeritus Chet's avatar
    Registered
    Aug 2001
    Location
    South Milwaukee
    Posts
    8,935
    Agreed, Dan thanks for all the hard work. I think you are often underappreciated for everything you do.

  10. #10
    Cyburbian Plus PlannerGirl's avatar
    Registered
    Mar 2002
    Location
    Va
    Posts
    4,604
    Thanks Dan for a while I thought our IT folks were screwing with me as about every third attempt to access the site I get denied. Hang in there Dan we all super appreciate what you do and are very willing to open the pocketbook to help out again.
    "They who can give up essential liberty to obtain a little temporary safety, deserve neither liberty nor safety." Ben Franklin

    Remember this motto to live by: "Life should NOT be a journey to the grave with the intention of arriving safely in an attractive well preserved body, but rather to skid in sideways, chocolate in one hand, martini in the other, body thoroughly used up, totally worn out and screaming 'WOO- HOO what a ride!'"

  11. #11
    CNET is reporting that at least part of the Conficker worm is hosted at an IP in Korea. Coincidence?

    Let me add my thanks and appreciation to Dan as well.

+ Reply to thread

More at Cyburbia

  1. Server issues: September 2012
    Cyburbia Issues and Help
    Replies: 7
    Last post: 19 Oct 2012, 10:52 AM
  2. Replies: 12
    Last post: 16 Apr 2012, 10:29 AM
  3. Server issues: November 2008
    Cyburbia Issues and Help
    Replies: 0
    Last post: 14 Nov 2008, 11:36 AM
  4. Server attacked
    Cyburbia Issues and Help
    Replies: 5
    Last post: 07 Oct 2006, 7:28 PM
  5. Server issues and intermittent outages
    Cyburbia Issues and Help
    Replies: 0
    Last post: 01 Jan 2006, 1:55 PM