TechHub

1 post1 participant0 posts today

AndyHachyderm is an amazing server. The mods are the best, and the infrastructure team manages the infrastructure better than most companies! If your account is on <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> I highly encourage to donate to help the team keep doing what they’re doing. <a href="https://mastodon.social/@flexghost/114292526374604847" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://mastodon.social/@flexghost/114292526374604847</a>

HachydermHello, hachyderm! we've been working hard on building up our ansible runbooks and improving hachyderm's overall resilience. Recently, we've been focusing on is database resilience.We're getting close to retiring our original database server (finally!) and preparing to move to a fully ansible-managed set of databases servers, primary and replica on new hardware. We'll send another announcement when we do the cut over. The team has done excellent work to make this highly automated, quick, and painless! :blobfoxscience: Done:✅ author ansible roles for managing postgresql, pgbackrest (backups), pgbouncer, and primary/replica failover ✅ decide to continue with pgbouncer and *not* use pgcat ✅ rotate database passwords ✅ order new replica database hardware ✅ order new future primary database hardwareTo do soon:🟨 rebuild replica database with ansible scripts 🟨 prepare primary database with ansible scripts 🟨 start replicating to new database replica 🟨 cut over to new database server 🎉 We're also planning on open-sourcing our ansible roles in the coming weeks - just a little housekeeping & tidying up before we do!<a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/ansible" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ansible</a> <a href="https://hachyderm.io/tags/postgresql" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#postgresql</a>

HachydermHello again Hachyderm! :hachyderm: As promised, here is the announcement blog post regarding the Pixelfed vulnerability, outlining the impacts to our community.Our blog post outlines the data behind the decision and steps for admins to reach out to us to refederate after they patched the vulnerability.Although the number of following relationships was low, the number of instances was high relative to that number. Basically: we're still in the process of reaching out to impacted instances as of this publication. Thank you for your patience!<a href="https://community.hachyderm.io/blog/2025/04/03/pixelfed-vulnerability-and-impacts-to-federation/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://community.hachyderm.io/blog/2025/04/03/pixelfed-vulnerability-and-impacts-to-federation/</a>As a reminder: we're happy to answer questions and engage with the Fediverse community. We're here to help :blobfox: <a href="https://hachyderm.io/tags/Hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hachyderm</a> <a href="https://hachyderm.io/tags/Pixelfed" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Pixelfed</a>

AndyI think the <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> admins take their infra more seriously than some companies do. Well done! <a href="https://hachyderm.io/@esk/114242919601414036" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://hachyderm.io/@esk/114242919601414036</a>

Esk 🐌⚡💜howdy, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>!over the last week or so, we've been preparing to move hachy's <a href="https://hachyderm.io/tags/DNS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DNS</a> zones from <a href="https://hachyderm.io/tags/AWS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AWS</a> route 53 to bunny DNS.since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like <a href="https://hachyderm.io/tags/duckdb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#duckdb</a> to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.then, after we've flipped the switch and while DNS is propagating -- :blobfoxscared: -- we can watch in real-time as different servers begin flipping over to the new provider.we named the tool hachyboop and it's available publicly --> <a href="https://github.com/hachyderm/hachyboop" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/hachyderm/hachyboop</a>please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code. :blobfoxlaughsweat: attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.<a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> <a href="https://hachyderm.io/tags/infrastructure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infrastructure</a>

Maho Pacheco 🦝🍻What a great work of moderation <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> admins do.

StoneBear :potion_genderqueer:Well, I'm back. <a href="https://hachyderm.io/tags/introduction" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#introduction</a> <a href="https://hachyderm.io/tags/genderqueer" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#genderqueer</a> :potion_genderqueer: dà-anam 🧙 :rainbowinfinity: storyteller/beat poet making their way in the greasy underbelly of the 'net. <a href="https://hachyderm.io/tags/filk" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#filk</a>, <a href="https://hachyderm.io/tags/scifi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scifi</a>, and thinking about <a href="https://hachyderm.io/tags/aviation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aviation</a> are pastimes. <a href="https://hachyderm.io/tags/activism" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#activism</a> , <a href="https://hachyderm.io/tags/GroenLinks" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GroenLinks</a> , <a href="https://hachyderm.io/tags/ClimateChange" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ClimateChange</a> , <a href="https://hachyderm.io/tags/TransrightsAreHumanrights" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TransrightsAreHumanrights</a> , stomping <a href="https://hachyderm.io/tags/antisemitism" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#antisemitism</a>; <a href="https://hachyderm.io/tags/TikkunOlam" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TikkunOlam</a> means *everyone*. I am Universalist with strong Old Celtic leanings, Jewish-adjacent, Baha'i-adjacent, my bestie is Christian, and my circle is generally eclectic, including at least one athiest. None of them try to *impose* their views. I am also a former professional and current amateur journalist. Sources get cited, mistakes corrected; I invite you to do likewise. Love wins, hate gets used for a DANCE FLOOR. Can't stop the :signal: AND MY INTRO FITS IN ONE! Thanks, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> !! It's good to be back.

Esk 🐌⚡💜howdy, folks - it's been a bit since our last <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> infra check in.stuff in motion:- ditching <a href="https://hachyderm.io/tags/terraform" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#terraform</a> cloud & tf for <a href="https://hachyderm.io/tags/opentofu" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opentofu</a> and <a href="https://hachyderm.io/tags/atlantis" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#atlantis</a>. we are just about to import our dev environment and put it through its paces. - bringing <a href="https://hachyderm.io/tags/postgresql" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#postgresql</a> under ansible management. the team has been doing awesome work, and we've started to spin up dev nodes using the new playbooks. soon: production! - moving <a href="https://hachyderm.io/tags/DNS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DNS</a> zones away from AWS route 53. we chose bunny DNS as our provider and have been doing basic tests in dev. we'll likely prep our records for production this week with a plan for a cutover in one of the coming weekends.and if you filled out our volunteer form and haven't heard from me in a bit - you're still on the list. we'll onboard a new batch of folks in the next couple of weeks.:hachyderm: :blobfoxheartcute: <a href="https://hachyderm.io/@hachyderm" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyderm</a> <a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/mastoadmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastoadmin</a>

RetrogradeJFC, what a <a href="https://oldbytes.space/tags/Hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hachyderm</a> shitshow...

Mr. Lance E Sloan (IRL) 👤<a href="https://hachyderm.io/@hachyderm" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyderm</a>, have you reviewed this case involving <a href="https://hachyderm.io/@anderseknert" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@anderseknert</a>? If not, why‽ Do you wish for <a href="https://mastodon.social/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> to become known for unreasonable judgements?

Anders EknertI didn't expect my post about Swedish Television mistakenly labeling Vladimir Putin as "president USA" yesterday to blow up the way it did. What I *really* didn't expect was <a href="https://hachyderm.io/tags/Hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hachyderm</a>'s admins deleting my post just as it was approaching a thousand boosts. This because someone allegedly reported it as "misinformation", and that what I saw on TV "does not appear to have actually happened"... which ironically is about as Putinesque a statement can be.So how would you fact check this? A few people commenting on the post had already done that, the way people normally do — by asking for the source. And they got it. I even boosted the link to SVT Play, where the broadcast can be streamed by anyone. It's still there <a href="https://www.svtplay.se/video/jAM5B9o/aktuellt/tor-13-mar-21-00?position=135" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.svtplay.se/video/jAM5B9o/aktuellt/tor-13-mar-21-00?position=135</a>The Hachyderm "fact checking" process doesn't ask for a source though. In fact, they didn't ask me for anything. Since there were no news headlines mentioning this blunder, it must not have happened. I appreciate the lecture for me to "fact check my posts before posting", but the assumption that I'd want to post anything on this instance again could certainly need some fact checking.To the rest of you, enjoy this last screenshot of Putin definitely not getting labeled as "president USA" on SVT2 News on Swedish Television. And for those of you who can't get enough of things that never happened, I'll post a video in a comment below too.

Julien RiouI was looking for an Ansible role to manage tailscale certificates for the Hachyderm infrastructure when I found this "collection" <a href="https://robertdebock.nl/ansible.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://robertdebock.nl/ansible.html</a>.Unfortunately, Robert doesn't seem to be on Mastodon so I can't mention him. Anyway, I would like to thank him for his INSANE contributions to Ansible. Look at all the playbooks and roles. The question is what software is not covered here?<a href="https://hachyderm.io/tags/ansible" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ansible</a> <a href="https://hachyderm.io/tags/tailscale" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tailscale</a> <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/tags/community" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#community</a> <a href="https://hachyderm.io/tags/foss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#foss</a>

Emelia 👸🏻The other root cause of our outage last night was a misconfiguration in how we handled requests through to mastodon-web (rails), which resulted in rack_attack believe all requests were coming from 127.0.0.1, instead of their actual IP address.Essentially our infrastructure looks like this| CDN | web server -> nginx —> nginx —> mastodon-web (rails/puma)We were correctly setting X-Real-IP, X-Forwarded-For and X-Forwarded-Proto headers on nginx at CDN edge, however when the request went through nginx on the web servers, these headers weren't passed through correctly all the time.We've since deployed changes to both our nginx configuration and our mastodon configuration (TRUSTED_PROXY_IP’s) to ensure that requests are correctly identified with the requestor's IP address, and rate limited accordingly.This should reduce the rate of “Rate Limit Exceeded" errors some people may have been seeing.Most folks running mastodon are only using a single nginx server in the request pipeline, so this is a very "us" problem, as we run our own CDN powered by nginx.<a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/@hachyinfra" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyinfra</a>

Emelia 👸🏻Heads up Mastodon operators: libvips 8.16.0 doesn't work with ruby-vips 2.2.2, which is currently used by Mastodon.*** watch out if your system automatically updates libvips to 8.16.0 ***This was one of the root causes of our media processing outage last night. Essentially when we were trying to process media, the conversion between different file sizes and metadata stripping was erroring out do to an incompatibility between ruby-vips 2.2.2 and libvips 8.16.0, resulting in the error:``` Error while optimizing <name>: VipsForeignLoad: "/tmp/<name>.<ext>“ is not a known file format ```This occurred for jpg, webp, and several other file formats, despite `vips -l foreign` showing everything setup correctly.ruby-vips 2.2.3 is known to fix the issue, and will be used in mastodon 4.3.4 when it is released. Until then either don't update libvips or switch MASTODON_USE_LIBVIPS to false until mastodon 4.3.4 (unless you're comfortable modifying the Gemfile to use the newer ruby-vips version)What we still have absolutely no explanation for though is that libvips 8.16.0 was installed on hachyderm systems on January 20th 2025, but we only started seeing issues yesterday after running regular system updates wherein libvips was not touched. Ruby and gems were reinstalled yesterday due to an update of libicu, but libicu was also updated on Jan 20th 2025 so we did the same reinstall steps on that date too.We didn't receive any error reports from people using hachyderm of not being able to upload media or media failing to load for them.<a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/@hachyinfra" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyinfra</a>

Julien RiouVolunteering for Hachyderm is a breath of fresh air to me.On the technical side, creating and adding Ansible roles for PostgreSQL (including PgBouncer and pgBackRest) is fun. I'm learning a lot, even with more than 10 years of experience.On the human side, I'm honoured to be part of such an amazing worldwide team sharing the same values.💜💜💜<a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/tags/postgresql" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#postgresql</a> <a href="https://hachyderm.io/tags/pgbouncer" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#pgbouncer</a> <a href="https://hachyderm.io/tags/pgbackrest" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#pgbackrest</a>

Emelia 👸🏻Apologies hachydermians for the outages tonight and the issue with rate limiting and images not loading. We discovered several issues that needed resolutions, and everything appears to be working fine now. There'll be a write up sometime in the future.Only negative? We've had to disable image processing with libvips, as it was failing to process media, so we're back to the imagemagik or whatever pre-libvips.<a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/@hachyderm" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyderm</a> <a href="https://hachyderm.io/@hachyinfra" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyinfra</a>

Esk 🐌⚡💜so, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>: funny story, turns out we *did* break image processing for a bit.we aren't entirely sure what happened, but it looks like the latest version of arch linux broke $something with libvips. this was causing mastodon to blow up on any image processing - attaching images, pulling remote images, creating thumbnails.as a temporary fix, we failed back to use image magick.additional hypothesis, but less sure about this one: we think because the image processing was failing, it was then triggering clients to re-request more frequently, which then triggered rate limiting in our nginx/mastodon-web.full write up coming soon (tm)!<a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/hugops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hugops</a> <a href="https://hachyderm.io/tags/nginx" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nginx</a> <a href="https://hachyderm.io/tags/archlinux" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#archlinux</a> <a href="https://hachyderm.io/tags/mastoadmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastoadmin</a> <a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a>

Esk 🐌⚡💜good morning <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>, don't mind me, just doing system updates. checking that i didn't break image processing or something.

Esk 🐌⚡💜this morning in <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> infrastructure, enjoying the extra `\n` that echo adds for free(pgbouncer, md5 passwords, and newlines)

slampNew emojis (logos) from <a href="https://hachyderm.io/tags/OpenInfra" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenInfra</a> foundation have been added to <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> :airship: <a href="https://airshipit.org" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://airshipit.org</a> :katacontainers: <a href="https://katacontainers.io/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://katacontainers.io/</a>:openstack: <a href="https://www.openstack.org/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.openstack.org/</a> <a href="https://hachyderm.io/tags/openstack" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#openstack</a> :starlingx: <a href="https://www.starlingx.io/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.starlingx.io/</a>:zuul: <a href="https://zuul-ci.org/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://zuul-ci.org/</a>/cc <a href="https://hachyderm.io/@kencrandall" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@kencrandall</a>

Recent searches

Search options

Administered by:

Server stats:

#hachyderm