Is today’s world (AI models training, security, etc.), I wonder if site owners are well equipped with robots.txt as the only way to control who’s in and who’s out when it comes to deciding free and massive content consumption on their site.
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.