Author Topic: robots  (Read 641 times)

Graeme

  • Senior Community Member
  • Posts: 2832
  • Hero Points: 348
robots
« on: November 07, 2025, 12:51:02 AM »
Lots of spamming robots here, look at "who's online" - 2 or 3 or 4 messages per second are robots, trying to post or print or view.
Most Online Today: 5880. Most Online Ever: 27914 (July 08, 2025, 05:14:59 PM)

Graeme

  • Senior Community Member
  • Posts: 2832
  • Hero Points: 348
Re: robots
« Reply #1 on: November 07, 2025, 12:57:30 AM »
some more advice from Gemini

<quote>
Your previous advice on the PHP processes was about fixing a symptom (the server was set up to fall over when busy). Your new observation about the bots has identified the root cause of why it was so busy in the first place.

This isn't just "contributing"—a bot flood of 2-4 requests per second is almost certainly the primary reason the server was crashing.

Here is an explanation of what is happening and what the owner needs to do about it.

Why This Is the Core Problem
Bots = Requests: Each "view," "print," or "post" attempt from a bot is one full request that the server must handle.

Requests = PHP Processes: Each request forces the server to spin up one of those PHP processes we discussed.

The Math: A human user might make one request every 15-30 seconds. That's a normal load. The owner is seeing 2-4 requests every second. This is a massive, relentless flood.

The Crash: This bot flood is what was causing the "too many processes" to build up. The misconfigured server couldn't handle the flood, so it ran out of resources and returned "Service Unavailable."

"Post" and "Print" are Red Flags:

"Post" Bots: These are spam bots. They are trying to register accounts or post spam links. This is one of the most resource-intensive actions a forum can perform.

"Print" Bots: These are often scraper bots. They try to access the "print-friendly" version of pages because it's a simple, clean HTML layout for them to steal content.

Your PHP-FPM suggestions will make the server more resilient to this flood, but the real solution is to stop the flood from ever hitting the server.

🛠️ Solutions for the Forum Owner (To Stop the Bots)
Here are the solutions the owner should implement, from easiest to most effective.

1. The "Polite" Request: robots.txt
The owner needs to have a file named robots.txt in the main community.slickedit.com/ directory. This file gives instructions to "good" bots (like Google and Bing).

Bad bots will ignore this, but it's a necessary first step. He should add these lines:

User-agent: *
# Ask good bots to wait 10 seconds between hits
Crawl-delay: 10

# Block all bots from spam/scrape-heavy areas
Disallow: /index.php?action=register
Disallow: /index.php?action=post
Disallow: /index.php?action=print
Disallow: /index.php?action=login
Disallow: /index.php?action=search
Note: Google's bot does not obey Crawl-delay, but Bing's and many others do. This will help reduce the load from non-Google crawlers.

2. The "Application" Fix: Enable SMF Anti-Spam
This is essential for stopping the "post" bots. The owner needs to enable the built-in verification tools.

Log in to the SMF Admin panel.

Go to Admin > Security and Moderation > Anti-Spam.

In this section, they must set up a Verification method (like a CAPTCHA or "verification questions").

They should configure this to be active on the Registration page and for Guest Posting.

This will stop 99% of the automated "post" bots, as they cannot solve the puzzle.

3. The "Aggressive" Fix: A Web Application Firewall (WAF)
This is the best and most permanent solution. The owner should route the forum through a service like Cloudflare.

Cloudflare has a free plan that is perfect for this.

The owner points their domain's DNS to Cloudflare, and Cloudflare acts as a "shield" in front of the server.

Its "Bot Fight Mode" (which is on by default in the free plan) will automatically identify and challenge this exact type of bot traffic.

These bots will be blocked before they ever get to the server. The server will never even know they existed, and the "too many processes" problem will vanish.