Bot Filtering
Bot filtering is the process of identifying automated, non-human requests and excluding them from analytics, ad reports, or site access. It keeps visit and click counts closer to real people. Filtering usually removes bot traffic from reports after the request arrives, while bot protection stops it at the server or network edge.
Why Bot Filtering Matters
Bot traffic is a large share of the web, so unfiltered numbers are wrong by default. Imperva’s 2025 Bad Bot Report found automated traffic reached 51% of all web traffic in 2024, the first time it passed human traffic in a decade. Bad bots alone made up 37%.
Unfiltered bot traffic inflates sessions and clicks, drags down conversion rates, and distorts which channels look efficient. A campaign can appear to win on traffic while its real audience is much smaller.
Not every bot is a problem. Search crawlers and uptime monitors are bots too. The goal of filtering is accurate data, not zero bots.
Good Bots vs Bad Bots
Good bots identify themselves honestly and follow the rules. Cloudflare defines a verified bot as one that is transparent about who it is and what it does. To qualify, it must declare its identity reliably (through a signature, a published IP list or reverse DNS) and obey robots.txt and crawl limits.
Cloudflare’s verified categories include search engine crawlers, archivers, monitoring services, feed fetchers and page preview tools for platforms such as Facebook, Slack, Twitter and Discord.
Bad bots hide. They spoof browser user agents, ignore robots.txt, rotate IP addresses and mimic human behavior. These are the bots a list cannot catch, which is why detection has to go beyond lists. Link preview bots sit in between: they are legitimate, but their requests still look like clicks.
Bot Filtering vs Bot Detection vs Bot Protection
The terms overlap in search, but each does a different job.
| Term | What it does | Where it acts |
|---|---|---|
| Bot detection | Decides whether a request is automated | Server, edge or analytics tool |
| Bot filtering | Excludes detected bots from data or reports | Analytics, ad platform, link tracker |
| Traffic filtering | Excludes any unwanted traffic, bots, internal staff and test hits | Analytics view or report |
| Bot protection | Acts on the request: block, challenge or rate-limit | Server, firewall or CDN |
Detection comes first. Filtering and protection both depend on it. The difference is the action taken. A filtered bot still reached your server and only disappears from the report. A blocked bot never gets the page. Bot filtering is also one tool for handling invalid traffic, the wider category that includes crawlers, accidental clicks and fraud.
How Bot Detection Works
Detection methods run from simple lists to scoring models. Most systems combine several.
- Known-bot lists. Requests are matched against lists of declared bots, such as the IAB/ABC International Spiders and Bots List. Fast and cheap, but it only catches bots that announce themselves.
- IP and network checks. Traffic from data center and hosting ranges is far more likely to be automated than traffic from a home ISP.
- User agent and header checks. Non-browser user agents, missing headers and mismatched values are common signs.
- Behavioral and machine learning scoring. Models rate how human a request looks. Cloudflare scores requests from 1 to 99: 1 means certainly automated, 2 to 29 likely automated, and 30 to 99 likely human. Cloudflare states that machine learning produces the majority of its detections.
- Challenges. CAPTCHAs and JavaScript checks make the client prove it can behave like a browser.
Bot Filtering in GA4
Google Analytics 4 filters known bots automatically. Google’s help page says traffic from known bots and spiders is excluded using Google research and the IAB International Spiders and Bots List. It also states that this exclusion cannot be turned off, and that the amount excluded cannot be viewed.
The practical limit is the word “known”. A bot that sends a normal browser user agent is not on the list, so it can still appear as a session. Treat GA4’s filter as a baseline, not a guarantee.
How to Check What Gets Through
Platform filters report only what they caught. To find the rest, compare sources.
- Compare clicks against sessions. Link or ad clicks far above analytics sessions point to requests that never loaded the page. The click-to-session method walks through it.
- Segment by network. Look for volume from hosting providers or cloud ranges.
- Check engagement depth. Sessions with zero seconds engaged, no scroll and one pageview are rarely human.
- Look at timing. Humans click irregularly. Many requests at near-identical intervals, or a flat overnight volume, suggest a script.
- Read the request headers. Browser prefetch requests announce themselves, which makes them easy to separate from visits.
GET /pricing HTTP/1.1
Sec-Purpose: prefetch
Limits and False Positives
No filter is complete, and every filter makes mistakes in both directions.
Evasive bots look human, so they pass. Strict rules go the other way and flag real people, such as visitors on a VPN or corporate network that shares a data center IP. A filter that is too aggressive removes real users from reports, or blocks them from the site.
Filtering also hides a problem rather than solving it. If bots drive load or cost, filtering the reports changes nothing on the server.
Frequently Asked Questions
What is bot filtering?
Bot filtering is the practice of identifying automated traffic and removing it from analytics, ad reports or site access. It aims to keep metrics based on real people. It relies on bot detection methods such as known-bot lists, IP checks and behavioral scoring.Is bot filtering the same as bot protection?
No. Bot filtering usually removes bots from reports after they arrive. Bot protection acts on the request itself by blocking it, challenging it or limiting its rate. A filtered bot has already reached your server, a blocked one has not.Does Google Analytics filter bot traffic automatically?
Yes, for known bots. GA4 excludes traffic from known bots and spiders using Google research and the IAB International Spiders and Bots List. You cannot disable this or see how much was excluded. Bots that do not identify themselves are not covered.How do you tell bot traffic from human traffic?
Compare several signals. Bot traffic often shows no engagement, one pageview, requests from data center networks, non-browser user agents and near-identical timing. No single signal is conclusive, so check clicks against sessions and look for patterns across more than one.Can bot filtering block real users?
Yes. These are called false positives. Visitors on a VPN, shared corporate network or cloud-hosted connection can resemble automated traffic. Filtering tuned too tightly drops real visitors from reports, and blocking tuned too tightly can lock them out of the site.To see how many of your link clicks never became visits, run the click-to-session check on your last campaign.