There’s a lot of explaining to do for Meta, OpenAI, Claude and Google gemini to justify overpaying for their models now that there’s l a literal open source model that can do the basics.
The same could be said for when Meta “open sourced” their models. Someone has to do the training, or else these models wouldn’t exist in the first place.
The cost is a function of running an LLM at scale. You can run small models on consumer hardware, but the real contenders are using massive amounts of memory and compute on GPU arrays (plus electricity and water for cooling).
Correct. But what’s more expensive a single computing instance that’s local or cloud based credit eating SAS AI that does not produce significantly better results?
I’m testing 14B Qwen DeepSeek R1 through ollama and it’s impressive. I would think I could switch most of my current usage of chatgpt to this one (not alot I should admit though). Hardware is amd 7950x3d with nvidia 3070 ti. Not the cheapest hardware but not the most expensive either. It’s of course not as good as the full model on deepseek.com but I can run it truly locally, right now.
There’s a lot of explaining to do for Meta, OpenAI, Claude and Google gemini to justify overpaying for their models now that there’s l a literal open source model that can do the basics.
The same could be said for when Meta “open sourced” their models. Someone has to do the training, or else these models wouldn’t exist in the first place.
The cost is a function of running an LLM at scale. You can run small models on consumer hardware, but the real contenders are using massive amounts of memory and compute on GPU arrays (plus electricity and water for cooling).
ChatGPT is reportedly losing money on their $200/mo pro subscription plan.
Yes GPT4All of you want to try for yourself without coding know how.
I’m testing right now vscode+continue+ollama+gwen2.5-coder. With a simple GPU it’s already OK.
You still need an expensive hardware to run it. Unless myceliumwebserver project will start
Correct. But what’s more expensive a single computing instance that’s local or cloud based credit eating SAS AI that does not produce significantly better results?
I’m testing 14B Qwen DeepSeek R1 through ollama and it’s impressive. I would think I could switch most of my current usage of chatgpt to this one (not alot I should admit though). Hardware is amd 7950x3d with nvidia 3070 ti. Not the cheapest hardware but not the most expensive either. It’s of course not as good as the full model on deepseek.com but I can run it truly locally, right now.
How much vram does your TI pack? Is that the standard 8gb ddr6?
I will because I’m surprised and impressed that a 14b model runs smoothly.
Thanks for the insights!