Crawler REGEX (Prebuilt)
As I promised, here are the latest pre-built crawler expressions that you just need to copy/paste.
"Crawler Spam Filter" configuration:
trello|asana|redmine
If one of the tools you use internally sends you traffic from real visitors, don't filter it out. Instead, use "Exclude internal URL queries" below.
For example, I use Trello, but since I share belgium number data guides on my site, some people link them to their Trello accounts.
Filters for language spam and other types of spam
The previous two filters will block most spam. However, some spammers use different methods to bypass the previous solutions.
For example, they try to confuse you by combining one of your valid hostnames with a well-known source like Apple, Google, or Moz. Even my site has been a target (not to say that everyone knows my site; it seems that spammers disagree with my guides).
However, even if the source and host look fine, the spammer inserts their message as another part of your reports, such as a keyword, page title, and even language.
In these cases, you need to take the dimension/report where you found the spam and select that name in the filter. It is important to note that the report name does not always match the name in the filter field:
"Crawler Spam Filter" configuration:
-
- Posts: 306
- Joined: Tue Jan 07, 2025 4:41 am