PlanetZ Forums Update & Bot trafffic

[spacef]

you are right, it's not a protection, it's a human process, because I grant specific user role manually. It let's me filter out and delete spam accounts more easily. It also discourages more attempts.

[valis]

Agree spacef, discerning the 'why' would potentially fall into a million splinters and rabbit holes, for the reasons May be as many as the bad actors. But a few more thoughts: The server used to have multiple unique IP addresses, these have become more costly and less necessary for modern configurations, and so there are a dozen (ish) other domains on this same box. Most of them are very small, friend's side hustles or old projects of mine. The old gaming forums (which would SEO like this due to age though in a different market segment). And my personal domain, which has been around since the 90's and is mostly used for email but has a web frontend.

None of the traffic is IP based, because it strictly comes to this forums, not the gaming forums, and not via email traffic or etc where server info might be exposed beyond the current cloudflare configuration. It's explicitly targeting the forums.scopeusers.com subdomain too, not the parent domain, and also explicitly https traffic. So it's targeted, whatever the reason(s) may be.

I wouldn't paywall the servers, even if it was a small amount that you can all afford. But I am also being clear that there's no banners here, and when I have offered merch it was purely for memorabilia due to having other purposes for the storefront beyond this site.

And so again spacef we are in agreement, although I take a slightly different tack. I think the landscape online has evolved to the benefit of the big platforms and at the expense of the older model of self hosting and small community projects etc. It is what it is.

Welcome back all.

[at0m]

Hi Valis,

Noticed your struggle with bots traffic. Haven't had any need for these myself, but you might:
https://arstechnica.com/tech-policy/202 ... obots-txt/

Another way I frequently see is some work that needs to be done by the visiting client, like some math or encoding exercise. Very easy for machines operated by humans, but it eats the CPUs of bots hosts who visit tons of websites. An example can be found at https://anubis.techaro.lol/

Not sure if any of these could be of use to discourage the forums. Should be pretty easy to set up, I think.

Best of luck, and thanks again for your efforts to keep on keeping up the z!!

[valis]

Cloudflare is pretty good at limiting the bots, AI bots and scrapers alike. Spam dropped massively, at least as far as I can tell. I don't think Gary is doing much cleaning without me. Those things weren't a big hassle before though, and when I block all bots completely it doesn't impact these traffic patterns enough to care, although it reduces traffic about 3-5%.

Thanks for the info! I'll scan the article and see if there's more that I could apply, I briefly read it when it was published but didn't read it in full. I read ars a few times a week to ensure I don't miss anything, and still read slashdot, toms etc etc. I miss some of the old tech sites that we've lost, like xbit labs and so on.

[LorenOA76]

Valis thanks so much!

[Spielraum]

The real question is, WHY?

>Okay, the devil seems to be in the details of the system already in use; that's tragic.
Fundamentally, I don't really understand why an absolutely underground site is being targeted so aggressively. What data are these hunters hoping to get? My Scopeuser email addresses seem to be well secured or hidden, thankfully.

Thanks so much for your hard work! I also missed something during the shutdown. It's great that it's up and running again, but I'm asking for a backup plan, like Sunmachine's offline compilation. It would be truly tragic if all the content were lost...

...it looked like kids trying tricks they saw on the internet about breaking...

> Yes, that certainly seems to be the case. But let's be honest, what cool bandit wants to brag about crashing a Scope site?


Fazit:
We're pretty much alone here, and extremely niche-oriented when it comes to Scope.
Only the off-topic posts might be discoverable via search engines. Perhaps the "Off Topic" section could be made visible only to logged-in users. This way, forum.scopeuser would have fewer opportunities to target users if kids, for example, search for "Native Instruments"... to plan new attacks...

Ps.:
My website was hacked years ago and filled with junk. It was completely unusable. I then decided to abandon the search algorithm obsession. People who are genuinely interested can find me without a search engine.

good luck to us
\o/

[spacef]

>>>> > Yes, that certainly seems to be the case. But let's be honest, what cool bandit wants to brag about crashing a Scope site?
some old timer who was expecting a mac version soon

[valis]

This site is backed up, always. Worst case scenario (I get hit by a bus), an email will be triggered and sent to Gary and Holger etc, and instructions given. It's also possible to pass stewardship on, but I don't see where that would help as I've done fairly well at knocking these things down one by one. Takeaway: I grip loosely and am being transparent about the issues because it was obvious something would be noticeable at some point (slowdowns, downtime, etc).

Anyone that follows the machine learning/AI stuff should understand how easy it is now to spin up remote cloud resources and automate things like never before. That's what I think the traffic is, and it's not the scraping/training bots as Cloudflare will block those. Whether it's a zillion devs doing their own crawling, or someone specifically attacking the audio industry I can't say (this forum SEOs well just by longevity for that market, even if the frontend of the site doesn't because I don't monetize anything). It is specific to this forum, John's forum for the Solaris (which I also manage along with other things for him), and a few other people I know like Assaf's current Polyverse, we're all seeing the same things.

[nebelfuerst]

Although I had an idea, that the unreachable forum was related to the bot attacs, I somehow felt a bit lost.
Maybe we could add an email address to the profile, where you could send some status in case of severe forum problems ?
(sort of mailinglist)

I prefer sites not monetized, but in the current situation an odd idea comes to my mind:
If those bots would load the first page completely, why not put a banner up there, to make them pay the bill they cause ?

[valis]

Sure, I can setup a mail list. Let me think on that for a bit. Since it was DNS related, it wouldn't have worked via this domain, but a third party free service like mailchimp or etc shouldn't have any issues as our userbase and mail volume would be low.

No paywalling. I was more referring to setting up banner ads or etc, the sort of things many forums tried when the FB era was killing their traffic and costs were rising. Maybe text ads from Google or etc, but I prefer to see the world move on from those business models. Not sure...but no paywall up front, and they come right to the forums.scopeusers.com domain and hit random pages in the forums, not the main forum page. Any frontend would add extra clicks for real users, and the bots would just find ways around theme eventually.

You will note btw that the actual forum spam signups are very low, for the last several years the ones we've gotten are actual people doing it manually. So, all hope is not lost.

Appreciate the thoughts and feedback, always.

[flux]

many thanks Valis, and of course to Gary

[sunmachine]

Just found this: https://jan.wildeboer.net/2025/04/Web-i ... net-Part-2

I guess you have all heard about the growing problem of AI companies trying to aggressively collect whatever data they can get their hands on to train their models. This has caused an explosive surge in web crawlers relentlessly hitting servers big and small. But who runs these crawlers? Turns out — it could be you!
...
So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth.
...
What these companies then sell to their customers is network access through the devices/PCs that have an app with this SDK installed. They are proud to tell you how you can funnel your (AI) web scraping etc through millions of rotating, residential and mobile IP addresses. Exactly the pattern we see hitting our servers.

Didn't know about this "business model".

[valis]

Most of the endpoints hitting us hard are overseas datacenters, so undeclared scraping bots for AI is certainly one of the considerations. There are not as many consumer endpoints hitting us as there were 5-10 years ago, when most of them were compromised machines in Russia, Eastern EU, Africa etc.

Note I can restrict access for web crawlers (google etc bots) and AI training bots through conventional means, both through .htaccess rewrite rules and robots.txt/llms.txt files. However I actually want legitimate models to crawl this site, and hope that the better Scope sites show up in queries to help Holger's business and bring us new users who are interested in learning and/or sharing their Scope efforts. And, restricting bots & web crawlers dropped HTTPS connects but those were always listed separately as they are properly declared, so it didn't impact the traffic patterns that concern.

Whatever the case is, I spent the last week analyzing traffic from the previous month and clamping down on the new traffic using those rules. I will report back when ready, but as of now I did reduce traffic.

[valis]

Extrapolating out from 7 days, it looks like traffic will be at less than 50% (total DNS hits) than before the downtime, and bandwidth will be 20% of before. Most of China has been eliminated now, I had the hosting company restore some firewall changes from before the server move (they had previously integrated my rules into their upstream firewall but I couldn't view or confirm anything done nor make changes), and tuned cloudflare a bit more, and made one or two changes to permissions here at the forums themselves.

That's good progress, and some of it could be attributed to the downtime but the hits the first 2-3 days after being back don't confirm that.

[valis]

Down to 10% of before outage.

Other than some communication issues with the current server provider, I think we're golden now.

[sunmachine]

Sounds great!
I used to use the Active Topics link without logging in, but that doesn't work anymore, because the search can only be used when being logged in. But that's fine if it helps to reduce the traffic.

[garyb]

thumbs up

[valis]

Yes I changed several permissions for guests and new users too.

Sounds great!
I used to use the Active Topics link without logging in, but that doesn't work anymore, because the search can only be used when being logged in. But that's fine if it helps to reduce the traffic.

[nebelfuerst]

I also bookmarked "Active Topics" as an entry point.
But it only requires a short press on login, to get "Active Topics".
As this extra click helps fighting the bots, it's a sacrifice I'm willing to give

[valis]

After a month since restoring the DNS, the analytics on cloudflare shows that we're slowly rising in bandwidth usage and visitor count but still lower than before we went offline. Without adjustment my projection is each quarter it rises back to the previous norms. I wonder how long I can play wack-a-mole with Class B/C blocks until it noticeable affects users.