Trying to update a wiki article I noticed that the wiki is extremely slow and responds to saving or previewin an edited article with the error message shown below. I've been told that the wiki is maintained by the fpc team. Could somebody please take a look?

---------------------------

Sorry! This site is experiencing technical difficulties.

Try waiting a few minutes and reloading.

(Cannot access the database)

Backtrace:

#0
/srv/www/lazaruswiki/includes/libs/rdbms/loadbalancer/LoadBalancer.php(972):

Wikimedia\Rdbms\LoadBalancer->reportConnectionError() #1
/srv/www/lazaruswiki/includes/libs/rdbms/loadbalancer/LoadBalancer.php(944):

Wikimedia\Rdbms\LoadBalancer->getServerConnection() #2 /srv/www/lazaruswiki/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal() #3 /srv/www/lazaruswiki/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/www/lazaruswiki/includes/libs/rdbms/database/DBConnRef.php(707): Wikimedia\Rdbms\DBConnRef->__call()
#5 /srv/www/lazaruswiki/includes/libs/rdbms/database/Database.php(3154): Wikimedia\Rdbms\DBConnRef->getSessionLagStatus()
#6 /srv/www/lazaruswiki/includes/user/User.php(527): Wikimedia\Rdbms\Database::getCacheSetOptions()
#7
/srv/www/lazaruswiki/includes/libs/objectcache/wancache/WANObjectCache.php(1689):

User->{closure}()
#8
/srv/www/lazaruswiki/includes/libs/objectcache/wancache/WANObjectCache.php(1522):

WANObjectCache->fetchOrRegenerate()
#9 /srv/www/lazaruswiki/includes/user/User.php(560): WANObjectCache->getWithSetCallback()
#10 /srv/www/lazaruswiki/includes/user/User.php(471): User->loadFromCache() #11 /srv/www/lazaruswiki/includes/user/User.php(404): User->loadFromId() #12 /srv/www/lazaruswiki/includes/session/UserInfo.php(92): User->load() #13
/srv/www/lazaruswiki/includes/session/CookieSessionProvider.php(131): MediaWiki\Session\UserInfo::newFromId()
#14 /srv/www/lazaruswiki/includes/session/SessionManager.php(537): MediaWiki\Session\CookieSessionProvider->provideSessionInfo() #15 /srv/www/lazaruswiki/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getSessionInfoForRequest() #16 /srv/www/lazaruswiki/includes/WebRequest.php(841): MediaWiki\Session\SessionManager->getSessionForRequest() #17 /srv/www/lazaruswiki/includes/session/SessionManager.php(164): WebRequest->getSession()
#18 /srv/www/lazaruswiki/includes/Setup.php(448):
MediaWiki\Session\SessionManager::getGlobalSession() #19 /srv/www/lazaruswiki/includes/WebStart.php(86): require_once('/srv/www/lazaru...')
#20 /srv/www/lazaruswiki/index.php(44): require('/srv/www/lazaru...') #21 {main}

Werner Pamler wrote

Trying to update a wiki article I noticed that the wiki is extremely slow and responds to saving or previewin an edited article with the error message shown below. I've been told that the wiki is maintained by the fpc team. Could somebody please take a look?

The wiki is either being ddos'ed or scraped (probably for "AI") by thousands of IP-addresses from the US and Singapore hosted by Alibaba Cloud LLC.

Many of them have been blacklisted automatically already by fail2ban. I'll manually blacklist all of their IP ranges now.

Jonas

Jonas Maebe wrote

Many of them have been blacklisted automatically already by fail2ban. I'll manually blacklist all of their IP ranges now.

Thank you. It's working again.

Werner Pamler wrote

Am 22.02.2025 um 09:25 schrieb Jonas Maebe via fpc-devel: > Many of them have been blacklisted automatically already by fail2ban. > I'll manually blacklist all of their IP ranges now. Thank you. It's working again.

They're not giving up...

3 135K 8094K REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 source IP range 47.74.0.0-47.87.255.255 reject-with icmp-port-unreachable

(the 135K is the number of packets, the 8098K is the number of bytes)

Jonas

FWIW: instead of rejecting, i would drop... that way their networking stack has to wait on timeouts and adds a delay load to their stuff... dropping also makes you "invisible" to them... with your rejects, your (icmp) responses tell them the system still exists, so they keep hammering...

Hi!

Am Sonntag, dem 23.02.2025 um 12:19 +0100 schrieb Jonas Maebe via fpc- devel:

Try to just (silently) DROP the incoming packets instead of replying with ICMP (or any other reply). This causes them to have to wait for a timeout of the connection.

Bye
Hansi

I know, but this rejecting is not causing issues for now, and I prefer to behave as a good net citizen as long as possible.

Jonas

Same for the forum, but manually adding subnets was almost a day job.... to many of them.

I've installed https://github.com/WKnak/fail2ban-block-ip-range as a cron job, running it every 5 min. (i've edited the script a but so it blocks /20 subnets and smaller

Marc

Maybe I should reconsider for Alibaba Cloud in particular: they actually posted a howto on building a web scraper using their cloud infrastructure in December 2024:
https://www.alibabacloud.com/blog/building-a-custom-chatbot-with-web-scraping-and-alibaba-cloud-model-studio_601863

You just pass it the address of a website, and they scrape it for you. Since the scraping happens using faked browser agents and without any regard for robots.txt, it's clearly abusive.

I would recommend anyone administrating a web server to permanently blacklist the following ranges:

47.235.0.0 - 47.246.255.255
47.250.0.0 - 47.254.255.255
47.74.0.0 - 47.87.255.255 (*)
43.135.128.0 - 43.135.191.255
43.153.0.0 - 43.153.127.255
47.88.0.0 - 47.91.255.255

Jonas

(*) this is the worst offender for us,  million rejected
packets totaling 91MB

The Lazarus top 10 of last week:

count net
45013 185.191.171.0/27
41427 85.208.96.192/27
23870 47.82.10.0/23
19461 51.8.102.0/24
7309 114.250.48.0/20
6215 84.54.64.0/20
5487 47.79.0.0/21
5369 172.203.190.128/28
5067 20.42.10.176/28
4904 39.156.168.0/21

The Lazarus top 10 of last week:

count net
45013 185.191.171.0/27
41427 85.208.96.192/27
23870 47.82.10.0/23
19461 51.8.102.0/24
7309 114.250.48.0/20
6215 84.54.64.0/20
5487 47.79.0.0/21
5369 172.203.190.128/28
5067 20.42.10.176/28
4904 39.156.168.0/21