• Ongoing coronavirus / COVID-19 discussion: how is the pandemic affecting your community, workplace, and wellness? 🦠

    Working from home? So are we. Come join us! Cyburbia is a friendly big tent, where we share our experiences and thoughts about urban planning practice, planning adjacent topics, and whatever else comes to mind. No ads, no spam, no social distancing.

The NEVERENDING Cyburbia security thread

Dan

Dear Leader
Staff member
Moderator
Messages
18,609
Points
69
This is an ongoing thread to report on security issues with Cyburbia, like bot and denial-of-service attacks.

Traffic from forum registration and scraping bots jumped over the past weekend. I had to add a bunch of IP ranges to the block list, the bulk from Web hosting companies in Europe (and the Netherlands in particular, for some reason) that tend to turn a blind eye to malicious traffic. A lot of VPN/proxy and Tor traffic, in our case consisting mostly of wannabe spammers and content scrapers, is also funneled through those Web hosts. I get that some people use anonymizing services for legitimate reasons, but most traffic we get via VPNs and Tor is malicious.

If you can't access Cyburbia because we blocked an IP address you often use, please post a message here from another ISP -- work, home, wifi from a coffee shop, wireless/mobile, or whatever -- and let us know.

Aggressive bots from China (blocking with .htaccess, etc.)

EDIT 2020-04-09: For those who might stumble on this thread, here's how I try to block the bots in Cyburbia's .htaccess file.
Apache config:
RewriteCond %{HTTP_USER_AGENT} 11A465|2345Explorer|360Spider|6899browser|adscanner|AhrefsBot|AliApp|AskTbCS-ST|AskTbAVR|AskTbNRO|AspiegelBot|BaiduClient|Baiduspider|BIDUBrowser|bsala|Bytespider|Campaign|Datanyze|DingTalk|DOWNLOADED|GreenBrowser|JuziBrowser|Kinza|LBBROWSER|LieBaoFast|LMY47V|MANAGED|MARKANYEPS|MauiBot|Mb2345Browser|MetaSr|MicroMessenger|MQQBrowser|OnalyticaBo|OPPO|OWASMIME|QQBrowser|serpstatbot|Shuame|SlimBrowser|Sogou|spaziodati|TencentTraveler|TheWorld|ToBeFilledByOEM|UBrowser|UCBrowser|wget|YaBrowser|Yowser|zh-CN|zh_CN|ZHCN [NC]
RewriteRule ^.* - [F,L]
Alternative .htaccess rule. Add to the end of your .htaccess file.
Apache config:
BrowserMatchNoCase "11A465" bad_bot
BrowserMatchNoCase "2345Explorer" bad_bot
BrowserMatchNoCase "360Spider" bad_bot
BrowserMatchNoCase "6899browser" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "AhrefsBot" bad_bot
BrowserMatchNoCase "AliApp" bad_bot
BrowserMatchNoCase "AskTbCS-ST" bad_bot
BrowserMatchNoCase "AskTbAVR" bad_bot
BrowserMatchNoCase "AskTbNRO" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "BaiduClient" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "BIDUBrowser" bad_bot
BrowserMatchNoCase "bsala" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Campaign" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "DingTalk" bad_bot
BrowserMatchNoCase "DOWNLOADED" bad_bot
BrowserMatchNoCase "GreenBrowser" bad_bot
BrowserMatchNoCase "JuziBrowser" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "LBBROWSER" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "LMY47V" bad_bot
BrowserMatchNoCase "MANAGED" bad_bot
BrowserMatchNoCade "MARKANYEPS" bad_bot
BrowserMatchNoCase "MauiBot" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "MetaSr" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "MQQBrowser" bad_bot
BrowserMatchNoCase "OnalyticaBot" bad_bot
BrowserMatchNoCase "OPPO" bad_bot
BrowserMatchNoCase "OWASMIME" bad_bot
BrowserMatchNoCase "QQBrowser" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "Shuame" bad_bot
BrowserMatchNoCase "SlimBrowser" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "spaziodati" bad_bot
BrowserMatchNoCase "TencentTraveler" bad_bot
BrowserMatchNoCase "TheWorld" bad_bot
BrowserMatchNoCase "ToBeFilledByOEM" bad_bot
BrowserMatchNoCase "UBrowser" bad_bot
BrowserMatchNoCase "UCBrowser" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "YaBrowser" bad_bot
BrowserMatchNoCase "Yowser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "zh_CN" bad_bot
BrowserMatchNoCase "ZHCN" bad_bot
Order Deny,Allow
Deny from env=bad_bot
If you visit Cyburbia at the right time, you might notice that the visitor count is unusually high -- 200 or more visitors and users, instead of the normal 50 or so.

What's happening? About two or three times a day, Cyburbia gets hammered with visitors from various IP addresses throughout China, mostly from two ISPs -- Chinanet and China Unicom. They all arrive in a swarm, visit various threads, and disappear about 15 to 45 minutes later. This article on John Large's tech blog describes the phenomenon. The server seems to be handing the swarms well, but the page hits bring no value to Cyburbia. Their intent is likely malicious, like a denial-of-service attack, or content scraping.

At first, I tried blocking as many Chinanet and China Unicom IP ranges as I could find. (We already block new user registration from the PRC, due to spam.) However, entering them into the list of banned IPs for XenForo is tedious. When bots from blocked IPs visit, they still use some CPU cycles, even though it's just for Xenforo to display a "you're banned" message. After a swarm last night, I'm trying a different approach -- blocking certain user agents (browser types) that are common among the swarms, along with Chinese language browsers (zh_CN in the user agent). This way, the server will respond with a 403 Forbidden error, which consumes far fewer CPU cycles than IP blocking in Xenforo. It won't stop the Chinabots, but it'll be more effective at keeping them from viewing the site.

The most common user agents:
Code:
Mozilla/5.0(Linux;U;Android 5.1.1;zh-CN;OPPO A33 Build/LMY47V) AppleWebKit/537.36(KHTML,like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.7.0.953 Mobile Safari/537.36

Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/42.0.2311.138 Mobile Safari/537.36 Mb2345Browser/9.0

Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36 LieBaoFast/4.51.3

Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN
 
Last edited:

Dan

Dear Leader
Staff member
Moderator
Messages
18,609
Points
69
And now, for the past several hours, we've been hammered with hits from IPs from Qualitynet (yeah, right) in Russia and Ukraine. Unlike Chinanet or China Unicom, there's no large, clean IP ranges to block, so I'm having to block a bunch of little /22,, /23, and /24 CIDR ranges.
 

Dan

Dear Leader
Staff member
Moderator
Messages
18,609
Points
69
The attack from certain IP ranges in Russia and Ukraine (Fineproxy: depo40.ru, Region40, QualityNetwork, Trusov Ilya Igorevych, Atomohost, Rackray, Petersburg Internet Network, Transit Telecom) has been going on continuously for about a day now. I'm blocking the IP ranges by hand in Xenforo, but they still try to get through. Thus, the higher than normal visitor count.
 

Dan

Dear Leader
Staff member
Moderator
Messages
18,609
Points
69
The Russian bots stopped hammering Cyburbia at about 9:30 PM EST.

There's been a renewed "attack" from the Chinese bots the past couple of days. We're getting hits every 2-3 seconds from Chinanet and China Unicom IP addresses.. They're not getting far. The site's .htaccess file (one of the server configuration files) blocks the user agents (browser type ID), the server gives them 403 Forbidden errors, and they're not seeing the message board. Folks at some other sites are reporting tens to hundreds of hits per second; enough to be a low scale denial-of-service attack.
 
Top