Is it possible to make the forum and wiki data available on a mirror for the use of AI crawlers?
(are search engine crawlers also blocked?)

You asked ChatGPT5:
"If websites with large Pascal forums and wikis block AI data harvesting bots, what will be the result for LLMs missing this training data and what will be the effect on these LLMs for Pascal related queries? Provide a simple concise answer with bullet points."

ChatGPT said:

- Thinner Pascal knowledge: fewer examples and patterns from FPC/Lazarus communities.
- Outdated answers: misses recent compiler/RTL/LCL changes and community best practices.
- Lower code quality: more non-idiomatic style, wrong units in uses, bad type handling, bogus directives.
- More hallucinations: invented packages/components, incorrect API names, fake compiler switches.
- Poor ecosystem awareness: weaker guidance on common libs (e.g., Synapse, Zeos, BGRABitmap) and tools (fpcupdeluxe, cross-compile flows).
- Weaker troubleshooting: less helpful fixes for linker/Widgetset errors, package management, and IDE quirks.
- Language bias: pushes solutions in C/Python/JS rather than solid Pascal.
- Narrower coverage: fewer edge cases (Windows/Gtk/Qt specifics, generics/modes nuances).
- Less reliable citations: can’t cross-check forum/wiki “canonical” answers.

Wayne Sherman wrote

Is it possible to make the forum and wiki data available on a mirror for the use of AI crawlers? (are search engine crawlers also blocked?)

Would you pay money for the hosting and for the maintenance of such a mirror?

You should've asked:

"If websites with large Pascal forums and wikis get DDoS'd by AI data harvesting bots..."

The answer should be that, if they get DDoS'd, then nobody would be able to access them - neither AI bots, nor regular users. So, the outcome would be worse.

ChatGPT said:

Please don't post ChatGPT generated responses as if they have any kind of authority. ChatGPT comes with a fine print: "ChatGPT can make mistakes." In this particular case, the company that trains ChatGPT has a clear conflict of interest. It is only normal that it exaggerates the effect of blocking AI harvesting bots.

Nikolay

Hello,
I had a moderate DDOS attack on my forum too, but I am sure it was not related to AI crawling. Too many different IP addresses.

It was very easy to solve it with Cloudflare. My forum now detects if the visitor is a real user, but lets known search engines and AI bots thru.

The known bots registered with Cloudflare behave well and don't flood forums.

I chose a payed Cloudflare plan (around $25 per month), but it might be possible with the free plan too.

If not, I would be more than happy to pay for this for the FPC forum.

I think that it is very important that legitimate AI crawlers and search engines have access.

On Sat, Sep 6, 2025 at 9:38 AM Nikolay Nikolov via fpc-pascal <

Nikolay wrote

Would you pay money for the hosting and for the maintenance of such a mirror?

The forums and wiki content could be pushed as static HTML or txt files to an FPC gitlab project, or use a free gitlab account which has 10 GiB of storage.

"If websites with large Pascal forums and wikis get DDoS'd by AI data harvesting bots..."

The main forums and wiki websites can continue to be blocked, but the mirrored data is still available to train LLMs.

Please don't post ChatGPT generated responses as if they have any kind of authority.

No, we don't need a ChatGPT response to understand what will happen. Something similar can happen when search engine crawlers are blocked. How useful would Google Search be for Free Pascal related queries if none of the Free Pascal websites were indexed?

What do you think? How would FPC and Lazarus related queries perform on an LLM with no training data for FPC and Lazarus in the form of articles, answers, and discussions?

Op 5-9-2025 om 23:05 schreef Wayne Sherman via fpc-pascal:

Is it possible to make the forum and wiki data available on a mirror for the use of AI crawlers? (are search engine crawlers also blocked?)

To my best knowledge, only misbehaving, high frequency robots are banned.

These mostly operate out of Asia with very high numbers of distinct IPs and very high fetch frequencies. It appears there are Chinese ISPs that offer packages to "accelerate" gathering content for AIs.

It is unlikely to affect good citizens like ChatGPT.

Tobias Klaus Giesen wrote

It was very easy to solve it with Cloudflare. My forum now detects if the visitor is a real user, but lets known search engines and AI bots thru. The known bots registered with Cloudflare behave well and don't flood forums.

This is probably a more practical solution than mirroring the content as static HTML or TXT files in a repo. People are using LLMs like a search engine and modern LLMs produce reference links back to the source of the information. A link back to the actual forum post or wiki page would be more useful. The LLMs also do realtime web searches during query processing to improve the response. If these are blocked it will also degrade the response.

As far as I understand neither search engines, nor AI crawlers are blocked by default.

The pages have a robot.txt, and any bot that obeys that afaik can crawl. Bots that don't, get blocked. (AFAIK)

On the wiki there is Anubis, but some bots are excluded from being checked, and in principal, bots can solve this, the same as normal browsers do for their users. Anubis does not block access. It just adds a cost, and that cost is tiny, unless you keep getting your IP banned, and you keep coming from multiple IP, and have to solve it many thousand (or 100k) times. Then it starts accumulating cost.

There are Companies that provide crawl services, using that many different IP, from all over the world. Those companies know that they break their target content providers (they go to great length to keep breaking others services). They don't care.
But as long as there are other companies that respect the robot.txt, then those abusers will loose out eventually, as they will not have the data to compete. Their problem.

A solution: do not use AI for things they weren't designed for.

If you really need an AI to help with your programming (you don't) then use it properly: install it in your computer and train it by yourself, or get one trained properly if it yet exists. Or (better) learn how to program properly and build your own knowledge library.

In any case DO NOT use any AI badly trained as the ones they offer "for free".[1]

Just my two cents.

Regards,
Guillermo Martínez.
____________________________________________

[1].- They actually cost us A LOT: our own culture and whole planet.

Wayne Sherman wrote

Is it possible to make the forum and wiki data available on a mirror for the use of AI crawlers?

The completely wrong approach. Why should we help companies that violate copyright laws to feed their bloody LLM? Just so that some script kiddies think they can actually write functioning software with it.

All this rubbish is useless and it's time for it to disappear. Since this hype began, our forum has suddenly seen a fivefold increase in traffic. Knowledge is being sucked up without permission, put through the mincer and usually reproduced incorrectly. There's nothing good about it.