techhub.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A hub primarily for passionate technologists, but everyone is welcome

Administered by:

Server stats:

4.6K
active users

#localllm

0 posts0 participants0 posts today
Replied in thread

@system76
I love #LLM, or as they're often called, #AI, especially when used locally. Local models are incredibly effective for enhancing daily tasks like proofreading, checking emails for spelling and grammatical errors, quickly creating image descriptions, transcribing audio to text, or even finding that one quote buried in tons of files that answers a recurring question.

However, if I wanted to be fully transparent to #bigtech, I would use Windows and Android with all the "big brotherly goodness" baked into them. That's why I hope these tools don't connect to third-party servers.

So, my question to you is: Do you propose a privacy-oriented and locally/self-hosted first LLM?

I'm not opposed to the general notion of using AI, and if done locally and open-source, I really think it could enhance the desktop experience. Even the terminal could use some AI integration, especially for spell-checking and syntax-checking those convoluted and long commands. I would love a self-hosted integration of some AI features. 🌟💻
#OpenSource #Privacy #AI #LocalModels #SelfHosted #LinuxAI #LocalLLM #LocalAI

So, I had an idea while messing with #LocalLLM geolocation with #Gemma3.

I fed it this photo from Wikipedia of Governors Island in NY.

Unsurprisingly, it figured it out immediately.

> Considering all the evidence, I am confident that the image depicts Governors Island in New York City. The shape of the island, the surrounding skyscrapers, the port facilities, and the presence of the Verrazano-Narrows Bridge all point to this location.

Fair enough. That's baby stuff at this point. ;)

But then I had an idea: "Assuming this refined guess is correct, where might the photo have been taken FROM?"

The ol' geolocate the camera trick. (Usually finding a CCTV or something, in some cases.)

> Based on the clarity of the view of Governors Island, the partial obstruction of other buildings, and the overall angle, I believe the photo was most likely taken from One World Trade Center. The height and location provide the perfect vantage point for this shot.
>
> Specifically, I'd estimate the photo was taken from the *observation deck or a high-floor office within One World Trade Center, likely facing southeast towards Governors Island and the Verrazano-Narrows Bridge*.

And uh... well, yeah probably. :D

Though I don't expect nearly as much success on other photos. I'll keep trying and see what I get.

[cc: @simon]

Continued thread

Then I installed Ollama and pulled Qwen from there — agent mode worked right away.

But when I tried Deepseek again through Ollama, it still said agent mode was unsupported.

I couldn’t figure out the difference — kinda confused.
What are you using locally, and how’s it working for you?

#LLM#LocalLLM#Qwen

🚀 Local LLM inference: where #ambition meets #reality in a glorious code spaghetti that no sane developer wants to untangle. 🎉 It's like building a rocket 🚀 only to realize you forgot the launchpad—works great in theory but crashes spectacularly in the real world. 🌎🔧
medium.com/@aazo11/local-llm-i #LocalLLM #Inference #CodeSpaghetti #TechHumor #SoftwareDevelopment #HackerNews #ngated

Medium · Local LLM inference - Amir Zohrenejad - MediumBy Amir Zohrenejad

I've done some #vibehosting yesterday... I couldn't be bothered investigating why #fail2ban keeps banning my IP after fetching emails from my email server, so I've decided to delegate my issues to #ollama.

I've set a knowledge base with all the necessary config and log files, etc, and asked #QwQ to investigate... Since it's a #localLLM, I had no issues submitting even the most sensitive information to it.

QwQ did come up with tailored suggestions on how to fix the problem. #openwebui

youtu.be/J4qwuCXyAcU

In this video, Ollama vs. LM Studio (GGUF), showing that their performance is quite similar, with LM Studio’s tok/sec output used for consistent benchmarking.

What’s even more impressive? The Mac Studio M3 Ultra pulls under 200W during inference with the Q4 671B R1 model. That’s quite amazing for such performance!

youtu.be- YouTubeEnjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

I have worked on a CPU-only inference portable solution so far, but Vulkan graphics api actually supports some older intel iGPUs.

The performance gain with llama.cpp is anecdotal, but it does actually helps with alleviating some load from the CPU.

This could interest owners of old intel laptops looking to play with 1B to 8B models (up to 14B works with 16GB of ram; a 14B is slow @ ~ 1.6 t/s).

Check here for GPUs tested: vulkan.gpuinfo.org/