Filter URL Crawling Issue (Mitigation Applied)
Known Issue:
Automated bots have been crawling filter URLs, especially where multi-select filters are in use. This has resulted in an excessive number of requests and unnecessary server load.
What we’ve done:
We’ve replaced the href attributes on filter <a> tags with a custom data-filter attribute.
A jQuery event now listens for user interaction and manually redirects to the selected filter.
This approach reduces the exposure of filter URLs to basic web scrapers and mitigates request overload.
Note:
This does not prevent advanced bots that simulate a browser and interact with the page (e.g. headless browsers), but it significantly reduces the impact of standard link crawlers.
Last updated