I’ve been looking into self-hosting LLMs or stable diffusion models using something like LocalAI and / or Ollama and LibreChat.
Some questions to get a nice discussion going:
- Any of you have experience with this?
- What are your motivations?
- What are you using in terms of hardware?
- Considerations regarding energy efficiency and associated costs?
- What about renting a GPU? Privacy implications?
You must log in or register to comment.
Ive been playing with the nixified.ai project, which packages two web interfaces for LLMs and image generation. Im also looking into Tabby.ml for code assistant as well. I haven’t gotten deep, but these all look like promising options for utilitizing a server’s hardware but offering the functionality across the network.
You want https://tabby.tabbyml.com/ instead of tabby.ml
