Run your own LLM locally, like ChatGPT or Claude

[Edgar_]

Why it says 'likely too large' at 15 g, i have no idea. i have 134 gigs available on my c drive

I don't know, but I doubt it's because of the available size on your c drive, how much RAM and VRAM do you have? If it's struggling with a 15gb model, try to download a quantized model instead, a Q4 would be smaller and still produce nice results.

RAM is the biggest candidate for that message because for a model to work, it full size gets dumped in the RAM, if there is not enough available memory to withstand the model, it will complain, so try closing some apps, check your task manager and you'll fund the reason, probably. Again, quantized models exist precisely because of this.

 

[Jon]

It's a gaming laptop I bought 2 years ago, it came with Windows 11.

It's a TUF DASH F15, it has an i7 12th gen processor, nvidia Geforce RTX 3070 card with 8GB VRAM, 16GB RAM. I don't play games, sadly, but I do test AI tools all the time. I can generate video, images, and recently, I've been testing LLMs.

Sounds like you have a powerful setup. My 13 year old Mac is far too weak to run anything like that. I look forward to an upgrade. Currently, I feel I am missing out on the AI boom.

The only thing I can think of is that perhaps you are light on RAM, with recommendations of 32gb of ram required for running LLM efficiently. But do your own research because I don't have 100% confidence in that claim. It does sound like fun what you are doing though. I am curious how fast AI's run with 32gb of RAM vs 16 gb of RAM.

Edit: I just asked AI and it said 2 seconds to start a reply is quite good and extra RAM is unlikely to speed it up. What extra RAM allows is a more intelligent model.

 

[Jon]

I've been following the topic, cybersecurity analysts say it's a nightmare because it exposes all your APIs and it opens the door to your computer to the internet and it seems to be by design, I can't confirm, but that's one of the next things to test in the near future.

Some people have been buying Mac Minis just to run ClawBot. This way you can separate your own computer from the wild ClawBot. I've been watching some videos on ClawBot and it seems like a huge moment in AI. It is tantamount to an autonomous agent that runs 24/7 making its own decisions. Quite remarkable. You give it an identity, a soul.md file for who it is, a tools.md file for what tools it can use etc. You can even get ClawBot to create a whole team of different agents for you, where it can act as a meta agent who controls the sub agents. It will create all the soul.md tool.md etc files for you. Take everything I say with a pinch of salt because I am new to learning about this.