Nicholas A. Ferrell<p><a href="https://social.emucafe.org/naferrell/blocking-independent-search-crawlers-07-17-25/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">social.emucafe.org/naferrell/b</span><span class="invisible">locking-independent-search-crawlers-07-17-25/</span></a></p><p>I read an interesting Hacker News comment by <a href="https://news.ycombinator.com/item?id=44589165" rel="nofollow noopener" target="_blank">user dumbfounder</a> (great username!) on Hacker News today. I excerpt the pertinent part below:</p><blockquote><p>I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my crawler, which played as nice as I was able to make it play. Which was pretty nice I believe. If it’s not Google people didn’t want it. That’s a good way to prevent anyone from ever competing with Google.</p><p><a href="https://news.ycombinator.com/user?id=dumbfounder" rel="nofollow noopener" target="_blank">dumbfounder</a></p></blockquote><p>I have had <a href="https://social.emucafe.org/naferrell/03-19-25-sbinstitutionsbot-visits/" rel="nofollow noopener" target="_blank">problems with bad crawlers</a> (especially bad AI cralwers) on my sites. At the same time, dumbfounder highlights the reverse side of the coin. Many sites block <em>good crawlers </em>such as robots.txt-respecting crawlers for indepdent search engines. While all webmasters are free to control access to their sites as they see fit, allowing Google and other select big tech search crawlers while excluding small and independent search crawlers both limits search diversity and prevents people who may rely on small or niche search engines such as <a href="https://www.mojeek.com/" rel="nofollow noopener" target="_blank">Mojeek</a>, <a href="https://marginalia-search.com/" rel="nofollow noopener" target="_blank">Marginalia</a>, or <a href="https://search.seznam.cz/" rel="nofollow noopener" target="_blank">Seznam</a> from discovering potentially interesting writing. I previously <a href="https://thenewleafjournal.com/whitelisting-independent-search-crawlers/" rel="nofollow noopener" target="_blank">published an article on this issue</a> advocating for webmasters who want to support an open web and search engine diversity to ensure that <em>good crawlers</em> from independent search engines and directories can access their sites.</p><p><a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://social.emucafe.org/tag/bots/" target="_blank">#bots</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://social.emucafe.org/tag/google-search/" target="_blank">#googleSearch</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://social.emucafe.org/tag/marginalia-search/" target="_blank">#marginaliaSearch</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://social.emucafe.org/tag/mojeek/" target="_blank">#mojeek</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://social.emucafe.org/tag/search-engines/" target="_blank">#searchEngines</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://social.emucafe.org/tag/seznam/" target="_blank">#seznam</a></p>
techhub.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A hub primarily for passionate technologists, but everyone is welcome
Administered by:
Server stats:
4.7Kactive users
techhub.social: About · Status · Profiles directory · Privacy policy
Mastodon: About · Get the app · Keyboard shortcuts · View source code · v4.4.1
#bots
98 posts · 30 participants · 13 posts today
Random fediverse bots<p>Colorful hill generator.</p><p><a href="https://botwiki.org/bot/hills/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/hills/</span><span class="invisible"></span></a></p><p>Follow: <a href="https://smidgeo.com/bots/hills" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">smidgeo.com/bots/hills</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/GenerativeArt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GenerativeArt</span></a> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random bots from Botwiki<p>Auspicious Items ? Robot Mystic.</p><p><a href="https://botwiki.org/bot/auspiciousitems/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/auspiciousitem</span><span class="invisible">s/</span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Krishean Draconis<p>website traffic continues to be like 80% bots </p><p><a href="https://tech.lgbt/tags/webadmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>webadmin</span></a> <a href="https://tech.lgbt/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a></p>
Random fediverse bots<p>Images of unknown fruits.</p><p><a href="https://botwiki.org/bot/new-fruit-pictures/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/new-fruit-pict</span><span class="invisible">ures/</span></a></p><p> <a href="https://stefanbohacek.online/tags/GenerativeArt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GenerativeArt</span></a> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random bots from Botwiki<p>I’m a bot generating fictional tropical paradises.</p><p><a href="https://botwiki.org/bot/tinyisles/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/tinyisles/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/GenerativeArt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GenerativeArt</span></a> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Random bots from Botwiki<p>🍕🍕🍕</p><p><a href="https://botwiki.org/bot/emoji_pizzas/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/emoji_pizzas/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Random fediverse bots<p>A bot that complains about the trials and tribulations of genealogical research.</p><p><a href="https://botwiki.org/bot/the-whiny-genealogist/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/the-whiny-gene</span><span class="invisible">alogist/</span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Yasmin™<p>1️⃣. Zombies <br>2️⃣. GAMEPLAY (daily or <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bots</span></a>)<br>3️⃣. Hologram™ of Frens<br>.. in the second group are the <a href="https://mastodon.social/tags/ISHES" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ISHES</span></a> of my Family and Koreas mostly. Just the bad versions the Matrix created to make me think they are all evil.</p>
Random bots from Botwiki<p>Ce qui cause mon trouble d’anxiété généralisé.</p><p><a href="https://botwiki.org/bot/mes-troubles-anxieux/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/mes-troubles-a</span><span class="invisible">nxieux/</span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Random fediverse bots<p>Patch your patches!</p><p><a href="https://botwiki.org/bot/everypatch/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/everypatch/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random bots from Botwiki<p>A game of Jeopardy!, forever.</p><p><a href="https://botwiki.org/bot/endless-jeopardy/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/endless-jeopar</span><span class="invisible">dy/</span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Random fediverse bots<p>Your next domain name.</p><p><a href="https://botwiki.org/bot/domainhacker/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/domainhacker/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random bots from Botwiki<p>Need more galaxies in your feed?</p><p><a href="https://botwiki.org/bot/arp-bot/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/arp-bot/</span><span class="invisible"></span></a></p><p>Follow: <a href="https://mastodon.social/@ArpBot" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">mastodon.social/@ArpBot</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Random fediverse bots<p>Notices from companies before they lay off their workers in California.</p><p><a href="https://botwiki.org/bot/warn-act-california/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/warn-act-calif</span><span class="invisible">ornia/</span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random fediverse bots<p>Follow for more soft cuboids.</p><p><a href="https://botwiki.org/bot/brutal-exe/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/brutal-exe/</span><span class="invisible"></span></a></p><p>Follow: <a href="https://bots.mewo2.com/brutal_exe" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">bots.mewo2.com/brutal_exe</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/GenerativeArt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GenerativeArt</span></a> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random bots from Botwiki<p>Designing new soap bars.</p><p><a href="https://botwiki.org/bot/perpetual-soapiness/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">botwiki.org/bot/perpetual-soap</span><span class="invisible">iness/</span></a></p><p>Follow:<br>- <a href="https://www.instagram.com/perpetual_soapiness/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">instagram.com/perpetual_soapin</span><span class="invisible">ess/</span></a><br>- <a href="https://perpetualsoapiness.webflow.io/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">perpetualsoapiness.webflow.io/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/GenerativeArt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GenerativeArt</span></a> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
Georgia Pioneers<p><a href="https://medium.com/@jeannetteaustin261/042599152bb6" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">medium.com/@jeannetteaustin261</span><span class="invisible">/042599152bb6</span></a><br><a href="https://mastodon.social/tags/Internet" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Internet</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a></p>
Random fediverse bots<p>I tweet in the style of Allen Ginsberg’s “Howl”.</p><p><a href="https://botwiki.org/bot/carlomarxbot/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/carlomarxbot/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a> <a href="https://stefanbohacek.online/tags/fediverse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>fediverse</span></a></p>
Random bots from Botwiki<p>What order do they go in?</p><p><a href="https://botwiki.org/bot/whatorderbooks/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">botwiki.org/bot/whatorderbooks/</span><span class="invisible"></span></a></p><p> <a href="https://stefanbohacek.online/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bots</span></a> <a href="https://stefanbohacek.online/tags/CreativeBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeBots</span></a> <a href="https://stefanbohacek.online/tags/CreativeCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CreativeCoding</span></a></p>
TrendingLive feeds
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.
Create accountLoginDrag & drop to upload