Sicklick
kiwifarms.net
- Joined
- Jun 25, 2020
What I mean by the title is are there any search engines that don't rely on Google/Microsoft's web crawler but uses a custom one? DuckDuckGo and Brave both use Bing's web crawler, at least to a certain degree and Searx just uses any other regular web crawler, but just anonymizes the results (like giving you the option of not directly visiting the web page which can still be tracked by Google). The only one I know for a fact that doesn't is Yahoo, but that's only because they're not so much of a "search engine" as they are a list of user-submitted links that were indexed in their search engine manually and doesn't rely on any web crawlers.
Naturally, 99% of websites can't be indexed unless it's through Google or Bing due to robots.txt. So some smaller search engine out there that relies entirely on a custom crawler, independent of Google and Bing, would be rather obsolete. Plus a search site without Google or Bing-level traffic would also lack large scale metrics to see what sites are most popular. DuckDuckGo also has terrible GPS.
Naturally, 99% of websites can't be indexed unless it's through Google or Bing due to robots.txt. So some smaller search engine out there that relies entirely on a custom crawler, independent of Google and Bing, would be rather obsolete. Plus a search site without Google or Bing-level traffic would also lack large scale metrics to see what sites are most popular. DuckDuckGo also has terrible GPS.
