Urban planning community

+ Reply to thread
Results 1 to 8 of 8

Thread: Server issues: September 2012

  1. #1
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3

    Server issues: September 2012

    This thread is for announcement and discussion related to what's going on with the site, technically speaking.

    First, the Cyburbia Facebook group: http://www.facebook.com/groups/159681217455629/ . I made a few announcements there already, when I turned off the site for a couple of hours last night. If Facebook is blocked at your workplace, here's what I wrote ...

    I shut Cyburbia down for now, while I try to figure out what's wrong with the server. Basically, we're having massive random spikes in server load that show very little correlation to traffic; for example the server load might be 0.50 with 80 users, but spike to 15.0 with 85.

    The spikes started happening about two weeks ago. There were no changes to vBulletin, no increases in traffic, no changes to any server settings. The host, Liquid Web, said we may need to upgrade to a new server. However, we upgraded during our last episode with spikes and slowdowns this April and it seemed to work. Traffic on Cyburbia has been about the same for the past few years; maybe down a little bit. To be honest, we lost a lot of users during the problems in April, and we'll probably lose a lot of use
    rs this time, too.

    Cyburbia has a managed virtual private server (1GB, 2GB burst, 2GHZ processor), which should be more than adequate to handle the amount of traffic Cyburbia gets. Liquid Web tried some tweaks to the Apache (web server) and MySQL settings, with no luck. I tried some tweaks I've seen in various places, with no luck. I installed a firewall to block all traffic from some trouble-prone countries, and beefed up the robots.txt file, which controls what search engine spiders are allowed to crawl Cyburbia, and at what rate, with no luck. For a while, we were getting swamped with search engine spiders, content scrapers, and spambots from China, Russia, and the Ukraine, some hammering the site several times a second, but blocking them didn't help.

    So, what now? I don't know. A problem is that the site crashes while I'm at work, which takes me away from what I'm paid to do full time. If I make tweaks now, I won't be able to see their effect until a random spike happens, or around 10-11 AM the following morning, when the first server-crushing spikes of the day hit us.

    I'll try to have the site up again tonight. And, as usual, it will crash tomorrow at 10:00 in the morning, when I'll be in a meeting with the developer of a 68 unit affordable housing project.
    A week or so ago, I put in a server optimization request at a vBulletin forum where a volunteer will post optimized my.cnf (MySQL) and httpd.conf (Apache) configurations based on your server and message board stats. Unfortunately, the person that does this now requires a special script to obtain those server stats, rather than from the easier cut-and-paste-the-results-from-Linux-shell-commands method used in the recent past,. Unfortunately, I can't get the script working on the Cyburbia server. Thus, no optimization.

    I'm trying settings from an optimization request someone else made about a year ago, for a server and forum with similar stats to Cyburbia. How's it working? Meh. Like I said, I really won't know until the 10-2 rush tomorrow, and for me, it's almost solid meetings durin
    g that time. Loads seem swingy; 0.1 to 1.5, where normally they'd be in the 0.2 to 0.5 range this time of the night. (In planner speak, think of server load as the equivalent of the volume to capacity ratio on a road.) A couple of pages might load fast, and then another takes 30 seconds. It's up and running, though.

    I also put in a request at the vBulletin forums to see if 1GB is really enough for a site with Cyburbia's traffic. Like I said, traffic has been steady over the past few years, but it's about half that of the 2003-2005 "glory days", when the site was on a crappy shared hosting account.

    And there goes the load up to 3 ... 3.5 ... 4.65 ... for no reason. No rush of visitors. This sucks.
    And ...

    Well, the first beta of vBulletin 5 was released today. It sucks. It could suck a golf ball through a garden hose. It could suck start a Harley. It's really that bad for a beta.

    I'm probably going to upgrade Cyburbia to the most recent version of vBulletin 4.2 this weekend, in the hopes that it might fix some of what ails the site. After that, though, it's probably going to be Invision Power Board, maybe around New Year's Day, when activity on Cyburbia is usually at its slowest.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  2. #2
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3
    So, what now?

    * I'm not upgrading the site to vBulletin 5. The reviews (here's one) are universally bad, and it appears to be extremely bloated.

    * This weekend, I'll be upgrading the site to vBulletin 4.2, to see if some of the bugs related to the site are resolved. I think the front page is responsible for some of the problems we're having.

    * Before the end of the year, I'll be converting Cyburbia over to Invision Power Board, This includes the gallery, which is now powered by Photopost.

    * For the short term, I may have to bite the bullet and upgrade the server to 2 GB. We can always cut back down to 1 GB when we move to IPB.

    Here's our server stats, updated every six minutes: http://www.cyburbia.org/loadavg/index.php. In plannerspeak, rhink of server load average as the equivalent of volume to capacity ratio on a road; we want to be under 1.0, although there may be some times when "congestion" is unavoidable. However, when the server load is above 1.0 a lot, or spikes into the double digits, it's bad,
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  3. #3
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3
    The site was likely under attack today, between 6:00 and 7:00 PM. This isn't a normal spike or irritating scrapers hitting the site several times a second; the spike correlates to a HUGE jump in traffic.

    5-minute load average
    High (6:15pm): 183.85
    Low (7:12pm): 0.02
    Mean: 4.32
    Latest: 0.17

    transmit rate
    High (6:24pm): 4,703.2 KB/s
    Low (6:15pm): 2.0 KB/s
    Mean: 49.5 KB/s
    Latest: 22.3 KB/s
    Today's estimate: 4.3 GB

    receive rate
    High (6:24pm): 8,702.2 KB/s
    Low (2:36am): 1.4 KB/s
    Mean: 49.1 KB/s
    Latest: 5.2 KB/s
    Today's estimate: 4.2 GB
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  4. #4
    NIMBY asshatterer Plus Richmond Jake's avatar
    Registered
    Aug 2001
    Location
    Jukin' City
    Posts
    16,479
    I'm too old to understand geek-speak, Thus, I get a pass.

    NTTAWW geeks.

  5. #5
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3
    Quote Originally posted by Richmond Jake View post
    I'm too old to understand geek-speak, Thus, I get a pass.

    NTTAWW geeks.
    It means that between 6:00 and 7:00 PM, about 100 times the amount of data was sent from the Cyburbia server than normal for that time, and thousands of times the amount of data was received by the server.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  6. #6
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3
    The upgrade from vBulletin 4.1.7 to 4.2.0 didn't work. The issues are related to problems we had last year when we upgraded from vBulletin 3.8 to 4.1.7

    Since 2001, with the directory structure for Cyburbia, the Forums have been in /forums; e.g. http://www.cyburbia.org/forums. This is a problem for vBulletin 4 and its content management system (CMS), because it expects the forums to be in the root directory; e.g. http://www.cyburbia.org/, and the functional front page of the site in a subdirectory. Rearranging the site to put everything in root would be extremely difficult, and would hurt Cyburbia's SEO, or search engine optimization. The URLs of all the posts, threads, and so on would be completely different. Like other sites, I wanted the articles/portal page to be the front page and the forum in a subdirectory,
    not vice versa.

    Basically, Cyburbia has a nonstandard installation of vBulletin. If I depended on vBulletin technical support to help with the upgrade, I'd be married to the computer for a few days, just like with the vBulletin 3 to 4 upgrade ... all while the site is down.

    The next upgrade for Cyburbia will be from vBulletin to Invision Power Board. I'll be paying the folks at IPB to do the conversion, so I don't have to worry about it. URLs will be redirected, so there's no loss of SEO. This means another somewhat jarring change to the look and feel of Cyburbia, just like that from vBulletin 3 to 4. However, considering that the vBulletin 4.* series is nearing end-of-life, and the vB 5 beta is quite awful, there's really no choice.

    When will Cyburbia changes to IPB? I don't know, but I hope it's sometime in the next few months. After nursing the server over the past couple of weeks, and the task of creating a new emergency/temporary board this week, I need a break.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  7. #7
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3
    Temporarily disabling the CMS.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

  8. #8
    Cyburbia Administrator Dan's avatar
    Registered
    Mar 1996
    Location
    Upstate New York
    Posts
    14,552
    Blog entries
    3
    Server load is extremely low with the CMS disabled. That's both good and bad - good because we should have no problems with Invision Power Board, bad because vBulletin's CMS is an extreme hog, and we can't upgrade from 4.1.7 to 4.2 in place until the switch.
    Growth for growth's sake is the ideology of the cancer cell. -- Edward Abbey

+ Reply to thread

More at Cyburbia

  1. Replies: 4
    Last post: 01 Oct 2012, 1:07 PM
  2. Replies: 12
    Last post: 16 Apr 2012, 9:29 AM
  3. Server issues: an update
    Cyburbia Issues and Help
    Replies: 10
    Last post: 09 Apr 2009, 11:54 AM
  4. Server issues and intermittent outages
    Cyburbia Issues and Help
    Replies: 0
    Last post: 01 Jan 2006, 12:55 PM