• Cyburbia is a friendly big tent, where we share our experiences and thoughts about urban planning practice, planning adjacent topics, and whatever else comes to mind. No ads, no spam, no echo chambers. Create your FREE Cyburbia ID, and join us today! You can also register through your Reddit, Facebook, Google, Twitter, or Microsoft account.

Aggressive bots from Chinanet and China Unicom IPs

Dan

Dear Leader
Staff member
Moderator
Messages
17,847
Points
59
If you visit Cyburbia at the right time, you might notice that the visitor count is unusually high -- 200 or more visitors and users, instead of the normal 50 or so.

What's happening? About two or three times a day, Cyburbia gets hammered with visitors from various IP addresses throughout China, mostly from two ISPs -- Chinanet and China Unicom. They all arrive in a swarm, visit various threads, and disappear about 15 to 45 minutes later. This article on John Large's tech blog describes the phenomenon. The server seems to be handing the swarms well, but the page hits bring no value to Cyburbia. Their intent is likely malicious, like a denial-of-service attack, or content scraping.

At first, I tried blocking as many Chinanet and China Unicom IP ranges as I could find. (We already block new user registration from the PRC, due to spam.) However, entering them into the list of banned IPs for XenForo is tedious. When bots from blocked IPs visit, they still use some CPU cycles, even though it's just for Xenforo to display a "you're banned" message. After a swarm last night, I'm trying a different approach -- blocking certain user agents (browser types) that are common among the swarms, along with Chinese language browsers (zh_CN in the user agent). This way, the server will respond with a 403 Forbidden error, which consumes far fewer CPU cycles than IP blocking in Xenforo. It won't stop the Chinabots, but it'll be more effective at keeping them from viewing the site.

The most common user agents:

"Mozilla/5.0(Linux;U;Android 5.1.1;zh-CN;OPPO A33 Build/LMY47V) AppleWebKit/537.36(KHTML,like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.7.0.953 Mobile Safari/537.36"

"Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN"
 

Hink

OH....IO
Staff member
Moderator
Messages
14,945
Points
41
I have nothing of value to add, but that is super interesting.
 

Dan

Dear Leader
Staff member
Moderator
Messages
17,847
Points
59
And now, for the past several hours, we've been hammered with hits from IPs from Qualitynet (yeah, right) in Russia and Ukraine. Unlike Chinanet or China Unicom, there's no large, clean IP ranges to block, so I'm having to block a bunch of little /22,, /23, and /24 CIDR ranges.
 

Dan

Dear Leader
Staff member
Moderator
Messages
17,847
Points
59
The attack from certain IP ranges in Russia and Ukraine (Fineproxy: depo40.ru, Region40, QualityNetwork, Trusov Ilya Igorevych, Atomohost, Rackray, Petersburg Internet Network, Transit Telecom) has been going on continuously for about a day now. I'm blocking the IP ranges by hand in Xenforo, but they still try to get through. Thus, the higher than normal visitor count.
 

Dan

Dear Leader
Staff member
Moderator
Messages
17,847
Points
59
The Russian bots stopped hammering Cyburbia at about 9:30 PM EST.

There's been a renewed "attack" from the Chinese bots the past couple of days. We're getting hits every 2-3 seconds from Chinanet and China Unicom IP addresses.. They're not getting far. The site's .htaccess file (one of the server configuration files) blocks the user agents (browser type ID), the server gives them 403 Forbidden errors, and they're not seeing the message board. Folks at some other sites are reporting tens to hundreds of hits per second; enough to be a low scale denial-of-service attack.
 
Top