How to Block Bad Bots From Your Website: A Complete Guide for 2026
Bots are automated programs that visit websites to perform tasks such as indexing pages, monitoring uptime, collecting information, and submitting forms. Some bots are useful, but others can scrape content, submit spam, attempt password attacks, or consume server resources.
If your website receives excessive automated traffic, you may notice slower page loading, repeated login attempts, spam submissions, or unusually high bandwidth usage.
The solution is not to block every bot. Search engines and legitimate monitoring services may need to access your website. Instead, you should identify unwanted activity and apply appropriate protection without disrupting real visitors or useful crawlers.
This guide explains how to identify suspicious bot traffic, configure robots.txt, use Cloudflare, protect WordPress, and troubleshoot common bot-related problems.
1. Understand the Difference Between Good Bots and Bad Bots
Before blocking automated traffic, determine what the bots are doing.
Useful Bots
Legitimate automated services can help your website operate and reach an audience. Examples include:
- Googlebot, which crawls pages for Google Search.
- Bingbot, which crawls pages for Bing.
- Uptime monitoring services that check whether your website is available.
- Approved tools that monitor website performance.
- Authorized integrations that access your website’s APIs.
Blocking these services unnecessarily may affect search visibility, monitoring, or business functionality.
Unwanted or Malicious Bots
Some automated traffic may attempt to:
- Submit spam through comments and contact forms.
- Guess passwords through repeated login attempts.
- Scrape website content at excessive rates.
- Search for vulnerable software or exposed files.
- Abuse registration, search, or checkout forms.
- Consume server resources with excessive requests.
- Attempt credential stuffing using previously exposed passwords.
Not all automated traffic is harmful, and a suspicious request does not automatically prove malicious intent. Investigate traffic patterns before applying restrictive rules.
2. Identify Suspicious Bot Traffic
Start by examining the requests reaching your website.
Useful information may be available in your hosting access logs, security plugin, CDN dashboard, or web application firewall.
Look for patterns such as:
- Repeated requests to the same endpoint in a short period.
- Large numbers of failed login attempts.
- Excessive requests to search or form submission pages.
- Repeated requests for URLs that do not exist.
- Sudden increases in traffic from a narrow group of IP addresses.
- Requests that consume significant resources without a legitimate purpose.
For example, a large number of failed requests to /wp-login.php may indicate automated login attempts. However, legitimate users can also make repeated requests, and a bot can distribute traffic across many IP addresses.
Use multiple signals rather than relying on one characteristic.
Check Your Server Logs
If you manage your own server, inspect the access logs for your web server. Common log locations include:
/var/log/nginx/access.logfor some Nginx installations./var/log/apache2/access.logfor some Debian-based Apache installations.- Hosting-provider dashboards that provide request analytics.
Actual log paths depend on your server configuration.
Review the request path, response status, request frequency, and user-agent string. Keep in mind that user-agent strings can be forged, so they are not reliable proof of a bot’s identity on their own.
3. Use robots.txt to Express Your Crawling Preferences
A robots.txt file tells compliant crawlers which parts of a website they should avoid crawling.
It is commonly available at:
https://yourdomain.com/robots.txt
You can create or edit this file through your website’s file manager, a suitable SEO tool, or another method supported by your hosting environment.
For example, the following rules ask a crawler named ExampleBot not to crawl your website:
User-agent: ExampleBot
Disallow: /
Replace ExampleBot with the actual user-agent token of the crawler you want to address.
Important Limitations of robots.txt
A robots.txt file is not a security barrier. Malicious crawlers may ignore its instructions, and the file does not prevent direct requests to restricted URLs.
Do not use it to protect passwords, private files, database backups, or confidential information. Secure those resources using authentication and server-side access controls.
Also, avoid disallowing important public pages from legitimate search engines unless you understand the SEO consequences.
For technical guidance, see Google’s robots.txt documentation.
4. Block Unwanted Bots With Cloudflare
Cloudflare can help filter unwanted traffic before it reaches your origin server. Depending on the features available to your plan, you may be able to use bot controls, security rules, managed challenges, and other protections.
Step 1: Add Your Website to Cloudflare
If your website is not already using Cloudflare:
- Create a Cloudflare account.
- Add your domain.
- Review the DNS records imported for your website.
- Confirm that important records, including email-related records, are correct.
- Follow Cloudflare’s instructions to activate the service.
Before changing nameservers, verify your existing DNS configuration. Incorrect DNS settings can interrupt website access or email delivery.
Official resource: Cloudflare setup documentation.
Step 2: Review Bot Traffic
Open your domain in the Cloudflare dashboard and review the available security and bot traffic information.
Look for repeated requests, suspicious patterns, and endpoints that receive unusual amounts of automated traffic.
Do not block traffic based on country, IP address, or user-agent string alone unless you have a clear reason and have considered the possibility of legitimate requests matching the same rule.
Step 3: Configure Available Bot Protection
Cloudflare offers bot-related controls, with availability and customization depending on your plan.
On eligible plans, Bot Fight Mode can challenge traffic that matches known bot patterns. Other plans may offer more granular controls, bot scoring, and custom security rules.
Start with the controls available in your dashboard and review their effects before adding complex rules.
Official resource: Cloudflare bot protection documentation.
Step 4: Use Challenges Carefully
A security challenge can require a request to pass additional checks before access is granted.
Challenges may help reduce unwanted automated traffic, but overly aggressive settings can also inconvenience real visitors, APIs, accessibility tools, and legitimate integrations.
Test changes against important website functions, including login, contact forms, checkout, and API endpoints.
Step 5: Monitor the Results
After enabling a security rule, review the traffic and security events again.
Check whether unwanted requests have decreased and whether legitimate users are still able to use your website.
Adjust the rule if it blocks legitimate activity or fails to address the intended problem.
For a practical introduction, see Cloudflare’s guide to stopping malicious bots.
5. Protect a WordPress Website From Bad Bots
WordPress websites can attract automated traffic to login pages, comments, registration forms, and search endpoints.
Use multiple layers of protection rather than relying on one plugin.
Keep WordPress Updated
Install supported security updates for WordPress core, plugins, and themes. Remove components that you no longer need, and obtain software from trusted sources.
Outdated software can expose vulnerabilities that automated scanners may attempt to exploit.
Use Login Protection
If your site has a WordPress login page, consider appropriate protections such as:
- Strong, unique passwords.
- Multifactor authentication.
- Login rate limiting.
- Monitoring for repeated failed login attempts.
- Additional access restrictions for administrative accounts where appropriate.
Avoid changing login URLs as your only defense. Attackers may discover alternative paths, and obscuring a URL does not replace proper authentication.

Reduce Comment and Form Spam
For sites that accept comments, use WordPress’s moderation settings and an appropriate spam protection solution.
For contact and registration forms, consider reputable anti-spam tools, rate limits, and CAPTCHA alternatives where appropriate.
Test the form after enabling protection. Excessive challenges may prevent genuine visitors from submitting important information.
Review Security Plugin Features
A reputable WordPress security plugin may provide login monitoring, firewall rules, suspicious request detection, and other protections.
Choose a solution that meets your needs and review its compatibility, maintenance history, and configuration options.
Avoid installing multiple security plugins that apply overlapping rules without understanding how they interact.
6. Rate-Limit Excessive Requests
Rate limiting restricts how frequently a client can make requests to a particular resource.
It can be useful for login endpoints, search forms, APIs, and other areas that are being abused.
For example, a website may apply stricter request limits to login attempts than to normal public pages.
However, a single rate limit should not necessarily apply to every URL. Pages, APIs, and static resources have different traffic patterns.
If you operate your own server, rate limiting can be implemented through supported web server modules, a reverse proxy, a CDN, or a security application.
Before deploying a rule, consider:
- The normal request rate for real users.
- Whether visitors share public IP addresses.
- Whether legitimate API clients need higher limits.
- How the system handles requests that exceed the limit.
- How to identify and correct false positives.
Use server or CDN controls to enforce restrictions. A script running in the visitor’s browser cannot reliably prevent a malicious client from making direct HTTP requests.
7. Block Confirmed Malicious IP Addresses Carefully
If logs show repeated abusive requests from a particular source, you may consider blocking or challenging that source.
However, IP blocking has limitations.
A single IP address may represent many users behind a shared network, and malicious traffic can move between IP addresses. Some attackers also use compromised devices or proxy services.
Before applying an IP-based rule:
- Review the relevant requests.
- Check whether the source is associated with legitimate activity.
- Decide whether a temporary challenge or rate limit is more appropriate.
- Apply the rule at a suitable layer, such as a CDN, firewall, or web server.
- Monitor the results and remove unnecessary restrictions.
Avoid copying large, unverified IP blocklists into production without understanding their source, update process, and potential impact.
8. Protect APIs and Other Sensitive Endpoints
If your website provides an API, ensure that automated access follows the intended authentication and authorization rules.
Depending on the application, consider:
- API keys or other appropriate authentication.
- Permission checks for sensitive operations.
- Rate limits for resource-intensive endpoints.
- Request validation.
- Logging and monitoring.
- Restrictions on unnecessary public access.
Do not assume that a request is legitimate because it comes from a browser or includes a familiar user-agent string.
Likewise, do not block all automated requests to API paths without checking whether mobile apps, payment integrations, or other services rely on them.
9. Avoid Blocking Legitimate Search Engines
Blocking bad bots should not mean preventing search engines from crawling your public content.
Before creating a rule, determine whether it affects verified search crawlers or other services your website needs.
For example, blocking all requests with a generic user-agent pattern may affect both unwanted scrapers and legitimate tools.
Use verified bot information where available, and do not rely solely on a claimed user-agent name because it can be spoofed.
After changing bot protection, review your search visibility, important pages, and server logs to identify unintended consequences.
Cloudflare explains the importance of distinguishing useful crawlers from unwanted automation in its bot management guide.
10. How to Troubleshoot Problems After Blocking Bots
Security changes can occasionally block legitimate visitors or interfere with application features.
If users report problems after a rule change, follow this process:
Review Security Events
Check which rule triggered the action and which request was affected. Look for patterns involving a specific URL, client, or request type.
Test Important Website Features
Verify that visitors can still:
- Load public pages.
- Log in.
- Submit forms.
- Complete checkout.
- Use legitimate APIs.
- Access required website resources.
Adjust the Rule
If a rule is too broad, narrow its scope or use a challenge rather than an outright block where appropriate.
Retest and Monitor
Make one change at a time, verify the result, and monitor for recurring issues.
Keep a record of the rule and why it was created. This helps prevent future administrators from reintroducing the same problem.
Common Mistakes When Blocking Bots
Blocking Every Bot
Some bots help search engines discover your content or perform important monitoring tasks. Identify the traffic you want to stop rather than blocking everything indiscriminately.
Treating robots.txt as a Firewall
robots.txt communicates crawling preferences but does not enforce access restrictions. Use server-side security controls for actual blocking.
Relying Only on User-Agent Strings
User-agent strings can be forged. Use multiple signals, such as request behavior and verified bot information, when evaluating suspicious traffic.
Blocking Large IP Ranges Without Testing
Broad IP restrictions can block real users, business partners, and shared networks. Review the evidence and monitor the effects of each rule.
Installing Too Many Security Plugins
Overlapping security tools can create configuration complexity and interfere with legitimate requests. Use a manageable set of tools with clearly defined responsibilities.
Ignoring Security Logs
Without reviewing the results, you may not know whether a rule is effective or is blocking real visitors. Monitor security events and application behavior after each important change.
Frequently Asked Questions
Does robots.txt block bad bots?
No. It asks compliant crawlers to follow your stated preferences. Malicious bots may ignore it, so enforced restrictions require server-side access controls, firewall rules, or an appropriate bot-management solution.
Can Cloudflare stop all bad bots?
No single service can guarantee that every malicious request will be blocked. Cloudflare can reduce unwanted traffic through available bot controls and security rules, but protection depends on the traffic, configuration, and features available to your account.
Should I block AI crawlers?
That depends on your content strategy. Some website owners allow search-related crawlers while restricting certain training or scraping bots. Review the available controls and their effects before blocking a category of crawlers.
Will blocking bots improve website speed?
It can help when unwanted traffic consumes substantial server resources. However, if your website is slow for other reasons, such as inefficient database queries or large assets, bot blocking alone may not resolve the problem.
How can I tell whether traffic is malicious?
Look for patterns such as excessive requests, repeated failed logins, suspicious endpoint probing, and abusive form submissions. No single signal proves that traffic is malicious, so use logs and context before applying restrictions.
Can a WordPress security plugin block bots?
Many security plugins provide features that can limit or block unwanted requests. Choose a reputable solution, review its settings, and test it carefully to avoid disrupting legitimate users.
Conclusion
Blocking bad bots is an important part of website security, but effective protection requires identifying unwanted activity without disrupting useful crawlers or real visitors.
Start by reviewing your traffic and server logs. Use robots.txt to communicate crawling preferences, apply enforced restrictions through an appropriate firewall or bot-management service, and protect WordPress login pages and forms.
Finally, monitor security events and test your website after making changes. A layered approach, combined with regular maintenance and careful rule configuration, can reduce unwanted automated traffic while keeping your website accessible to legitimate users.
No Comment! Be the first one.