Bookmarked: Some of the challenges of building an internet search

Of recent indexing failures, 62% are sites going offline, 19% are sites where robots.txt blocks indexing, and 5.5% are where Cloudflare blocks indexing.