It's a minor issue that we had with Dnsmasq, but we've run into this issue and documented it, so you don't have to 
"Dnsmasq does not start because port 53 is busy"
https://blog.zero-iee.com/en/posts/dnsmasq-startup-error-with-systemd-resolved/
It's always DNS, right? If someone experiencing some strange issues with ubuntu, maybe this toot is for you.
tl;dr: switch from systemd-resolved to resolvconf.
I thought, the saying from the beginning was just something from the "old days". No DNS Problems in 2024 anymore, right? But Ubuntu taught me different.
Ubuntu is using systemd-resolved since 20.04 (if I'm correct). But I was shocked, when I was looking at my uptime kuma Container on a Ubuntu 22.04 LTS Host. It was constantly failing. Sometimes 3 services at the same time, sometimes just 1 service a day. One Check suddenly failed. 60 seconds later, the next check, switched back to green again. But all fails had the same error message: "getaddrinfo ENOTFOUND domain.com". Doesn't matter if they were internal domains or external. Sometimes some of them just failed.
I thought it could be an old Firewall Applience that were running at like 120% system utilization and were serving DHCP and (with this) internal DNS. But no. Not even high latencies from that Firewall. Then I thought it might be AdGuard (in a Docker Container). So I switched to PiHole. But the problems were still the same.
Then I turned on debug logs of systemd-resolved and found out that sometimes it was switching to the secondary DNS Server for whatever reason and just attaching the search domain to the following requests:
1. AAAA of demodomain.com
--> no answer (because only A were available)
2. A of demodomain.com
--> somehow failed, systemd-resolved switched to second DNS (debug log of systemd-resolved is hard to read, not sure why it somehow failed)
3. AAAA of demodomain.com.local
--> it just attached the searchdomain of the system to the domain which now resulting in errors from all following DNS Server
After another round of wrong requests it suddenly get back his head. But in the meantime, uptime kuma already failed.
The solution in my case: switch "back" to resolvconf package on Ubuntu. Which comes to at least one downside: it seems to not have an interface to netplan and/or networkmanager (which leads to manual creating and managing of resolv.conf, not via DHCP, bummer). But after I switched: Everything is working fine and without any problems since days.
"We" also have an open bug report which since 3 years: https://github.com/systemd/systemd/issues/21123
It's not exactly the same issue, but I think the root cause is connected somehow: it seems to be a problem of IPv6.
But a) I need (or better: want) IPv6 in my case/that network and b) WTF? How can this be a good solution to turn off IPv6 (https://github.com/systemd/systemd/issues/21123#issuecomment-2028976737)? Not to mention that we still need a solution for Post-IPv4.
By the way: If you still experiencing DNS issues inside Docker Container, maybe Alpine could be another issue: https://martinheinz.dev/blog/92
Just blogged: "Specific DNS servers for specific domain names with systemd-resolved"
https://www.christianfosli.com/posts/2024-systemd-resolved-private-ep/
