Recent searches

No recent searches

Search options

Only available when logged in.
techhub.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A hub primarily for passionate technologists, but everyone is welcome

Administered by:

Server stats:

4.6K
active users

techhub.social: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.2

#llamacpp

0 posts0 participants0 posts today
Grigory Shepelev

I am an #AI-enhanced coding believer now since I've started working at the new place (3-4 months ago). Using #openrouter is a corporate practice there and it's kinda obligatory.

Now I want to enhance my #guix setup with all mcp's possible, upgrade a video card in my desktop and start local #llamacpp server and share it with some friends.

Saemon Zixel

Запустил llama.cpp на другой материнке с процессором AMD E2-3000. Это хоть и аналог Intel Atom, но посовременнее.

Разбор промпта и генерация ответа стали чуть-чуть быстрее. На 10 процентов примерно. Хотя память DDR3 работает на шине 1600МГц и быстрее в 1,5 раза, чем предыдущая DDR2 на 1066МГц шине. Зато процессор был на 2,6ГГц. А у этого всего лишь 1,6ГГц.

Перекомпилировал llama.cpp на этом процессоре, и скорость прям удвоилась.
Vikhr-Llama-3.2-1B-Q8_0 выдаёт 2 токена в секунду.
А QwQ-500M.Q8_0 выдаёт 6 токенов в секунду и прям так бодренько пишет ответ. Правда, моделька глупенькая, склонна рассуждать и редко правильно отвечает.

Как я понял, это всё из-за поддержки процессором AVX1 и FP16C. А скорость оперативной памяти, к сожалению, тут почти не играет роли.

#llamacpp#vikhr#qwq
Saemon Zixel

А llama.cpp достаточно легко и просто скомпилировалась в моей 32битной altlinux. Зависимостей мизер. Ничего не потребовалось доустанавливать, компилить. При этом работает стабильно, не ругается, не сегфолтиться.

Тестил с Vikhr-Llama-3.2-1B-Q8_0.gguf, которая на 1,2ГБ и знает русский язык. Скорость "чтения" промпта 2 токена/сек. А скорость генерации ответа 1 токен/сек. Для вопросов "не к спеху" можно использовать, но качество ответа так себе.

Замечу, что компьютер у меня старенький: Pentium D E6300 на 2,8Ггц, поддерживает максимум SSSE3 и работает с памятью DDR2 на 4ГБ. По этому, то, что есть уже радует меня)

#llama#llamacpp#linux
Phil

Big hopes for Qwen3. IF the 30A3B model works well, gptel-org-tools will be very close to what I envision as a good foundation for the package.

It's surprisingly accurate, especially with reasoning enabled.

At the same time, I'm finding that #gptel struggles a lot with handling LLM output that contains reasoning, content and tool calls at once.

I'm stumped. These new models are about as good as it's ever been for local inference, and they work great in both the llama-server and LM Studio UI's.

Changing the way I prompt doesn't work. I tried taking an axe to gptel-openai.el, but I frankly don't understand the code nearly well enough to get a working version going.

So... yeah. Kinda stuck.

Not sure what next. Having seen Qwen3, I'm not particularly happy to go back to older models.

#emacs #gptelorgtools #llamacpp

Nexus6

I've just published the first part of my guide on setting up an AI/LLM stack in Haiku. If you've been curious about running AI models on alternative operating systems, this one's for you!
🔗 blog.nexus6.me/new%20adventure
#HaikuOS #langchain #openai #llamacpp

Nexus6's Blog · Setting up an AI/LLM Stack in Haiku: A Practical Guide part II wanted to push the boundaries of what Haiku can do, so I decided to experiment with setting up a complete AI stack on it. My goal was to see if Haiku could actually run modern Large Language Models without GPU acceleration by using the most used frameworks like LangChain. While mainstream operating systems often require powerful hardware for AI workloads, I was curious if Haiku might offer a practical alternative for enthusiasts who want to explore AI without investing in specialized equipment. In this article, I’ll walk you through how I built a functional Python environment for AI in Haiku and demonstrate how to leverage essential components for working with LLMs, all running on modest hardware like my ThinkPad T480s.
Todd A. Jacobs | Rubyist

It seems like metal-enabled #llamacpp using #gguf is faster than llama.cpp with #mlx on my #AppleSilicon. #Ollama is mlx-only and slower, so not just a tool optimization.

MLX was designed for Metal so should be faster. Maybe it helps more with Apple Intelligence or something? I now choose GGUF over MLX unless I specifically need Ollama.

Anyone else had similar experiences? Do newer M-series chips do a better job with it, or did I not account for something?

github.com/ggerganov/llama.cpp

LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.
GitHubGitHub - ggerganov/llama.cpp: LLM inference in C/C++LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.
Olivier Chafik

llama.cpp now supports tool calling (OpenAI-compatible)

github.com/ggerganov/llama.cpp

On top of generic support for *all* models, it supports 8+ models’ native formats:
- Llama 3.x
- Functionary 3
- Hermes 2/3
- Qwen 2.5
- Mistral Nemo
- Firefunction 3
- DeepSeek R1

Runs anywhere (incl. Raspberry Pi 5).
On a Mac:

brew install llama.cpp
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M

Still fresh / lots of bugs to discover: feedback welcome!

This supersedes #6389 (now using a fully C++ approach), #5695 (first attempt at supporting Functionary) and #9592 (more recent Python wrapper). Which models are supported (in their native style)? W...
GitHubTool call support (generic + native for Llama, Functionary, Hermes, Mistral, Firefunction, DeepSeek) w/ lazy grammars by ochafik · Pull Request #9639 · ggerganov/llama.cppBy ochafik
#llamacpp
TrendingLive feeds
About