Observation:
- People started deploying anti-scraping measures to fight LLM scraping
- Web indexers can't index stuff anymore
- Search results are even worse than before
- The only way to retrieve the information is to use models that were trained in pre-anti-scraping times (or beat anti-scraping)
If I'm right, anti-scraping can actually push people towards LLM's (who currently absolutely have the capacity to circumvent most anti-scraping).
If you think you share knowledge worth finding, please consider this before deploying coutnermeasures!
#scraping #search #llm