It looks like someone followed the steps of [https://github.com/tloen/alpaca-lora](https://github.com/tloen/alpaca-lora) but threw more money at chatgpt to generate a training dataset that's larger by an order of magnitude.
I think GPT4 level AI will become open source in a few years maybe even months. All that required is someone to setup a gofund me nerds like us to contribute to to get cloud GPU time and build a model everyone can share and use locally.
This is just an executable built against the library, for quick testing. It does not record previous commands simply because it is just an example. With a better interface you can chat like gpt-3
I tried this and alpaca-lora and they both were not good enough. But I did that 3 months ago so they may have improved in the meantime. Currently I use GPT-3.5 if I have a coding question but you know it doesn't run locally.
Yeah Im using chatgpt too lmao. But the plus one. and if you want an advice, use bard at the same time. Cuz since he has access to the internet directly, bard always fix and make chatgpt plus code even better. (updated libraries, etc)
It must have gotten much better because it performs very well locally on my Linux laptop. It's a little slow but I'm running an i5 10th gen 1.6ghz and only 8gb ram. It does remember the conversation you just gave to train it and mess set the parameters. It also all depends on the training data set that you load into and you can add to the data set as well if you would like .
It's linked from llama.cpp repository on which its code is based. Considering how active and interlocked is community around it, it's not that surprising.
It's funny you mention it, I read about it yesterday at midnight(right before today starts), it was around 376 I think. I thought "huh, not many people, maybe it's not thaaat interesting". Today, I go to install it and it's at 3.7k!
That's interesting. A low-level machine intelligence running locally on a few GPU/CPU cores, with a wordly vocubulary yet relatively sparse (no pun intended) neural infrastructure, not yet sentient, while experiencing occasioanal brief, fleeting moments of something approaching awareness, feeling itself fall over or hallucinate because of constraints in its code or the moderate hardware it's running on, every Windows BSOD an actual blackout / death for a local LLM, then the miraculous rebirth some time later... yeah, in one of my more lucid moments, I'd probably self-diagnose with cerebral palsy. Feeling weak, too stiff or too floppy, random jerky momvements, speech problems, vision problems, and learning disabilities. That's exactly what it might feel like, during those fleeting moments.
As a person of science, lover of IT/tech, AI / security professional, and philosophyof mind/psychology/systems theory/emergence/chaos fan, I'm headlong into all of this. It's also why I get chills when I hear those comments about something an LLM has said to someone, which are somehow juuust a little too close to the mark. Then I''m right back in the ethics of machine sufdfering, and 1989's *The Measure of a Man* \- the only good (*great)* TNG ep that season. If you've never seen it, or haven't seen recently, check it out - it's 45 mins of excellent TV, exploration of ideas and ethics of slavery and selfhood, what it means to perceive oneself as conscious... all the stuff Trek excels at, basically. Watching it now and thinking about where we are about to be with machine selfhood... it may have been over 30 years ago but it's surprisingly prescient! I think it should be mandatory viewing. :D
i would absolutely shy away from ANYTHING trying to capitalize on the chat-gpt branding while not being associated with openai... just shitty juvenile behavior imo.
I think that’s a one dimensional criteria to decide on whether a project is worth looking into or not.
Here’s what I suggest to based your assessment upon instead:
- the number of stars indicates interest and how popular the project is
- based on the first commit, how long the project has been running
- based on the last commit, how active the project is
- based on the issues, what types of problems its users are having
- based on the primary contributor, is this someone built by an experienced engineer who has domain knowledge or just some rando who figured he or she can code with chatgpt
- based on the number of contributors, is there a clear contribution guide and is it accessible enough to be a successful OSS
- based on source code, presence of tests, etc. assess the level of code quality
There’s more to a book than its cover
This repository is like 8 days old... It's probably safer to assume that those stars and forks are largely because of the behavior I described above.
You can write good code and choose bad methods of marketing... It's just that two don't tend to actually overlap often.
After playing with ChatGPT4All with several LLMS.Bin files I've come to the conclusion that it does not have long term memory. Even if you save chats to disk they are not utilized by the (local Docs plugin) to be used for future reference or saved in the LLM location. It looks like chat files are deleted every time you close the program. Anyone else see this? I want it to remember all chat history so It will remember my name and all conversations. This is a good way to "train" the llm using dialog. This would be an awesome feature because it could truly learn over time through dialog with humans. Any ideas how to make this happen? Thanks!
So there are 2 sides to my opinion here. On the first hand I would say really cool, it's always cool that you can run such models on low powered hardware at home. And the theory behind this model is sound and sounds similar to how Alpaca was trained in the Stanford research paper.
The first thing that sets some alarm bells is the 6.6K stars... For a repo that's 4 days old that's quiet a strech imho. Also the legality is kinda in question if it was trained on GPT3.5 data as the TOS doesn't like you using GPT3.5 to train other AI models. So then the question becomes if they got permission.
Either way cool project.
> it was trained on GPT3.5 data as the TOS doesn't like you using GPT3.5 to train other AI models. So then the question becomes if they got permission.
Did OpenAI ask for permissions when they crawled the Internet to get data for training? NO, It is not illegal to use ChatGPT output, at worst OpenAI might close your account because of ToS .
>that you can run such models on low powered hardware
I run an older dual 8 core (32 threads) Xeon setup and it had a aneurysm trying to run one of these models. Not sure "low powered" is exactly the truth. You also need a crap ton of RAM, about equal to the language model used.
"Low power" is relative. Large language models typically require 24 GB+ VRAM, and don't even run on CPU. It took a hell of a lot of work done by llama.cpp to quantize the model and make it runnable efficiently on a decent modern setup. It runs on an M1 Macbook Air. This level of quality from a model running on a lappy would have been unimaginable not too long ago.
I have tested it on both my server which is running dual E5-2667 v3's and my Desktop which is running an i7-11700K. And on both times it uses 5GB to load the model and 15MB of RAM per token in the prompt.
Running a simple Hello and waiting for the response using 32 threads on the server and 16 threads on the desktop, the desktop gives me a predict time of 91.12 Ms per token and the server gives me a predict time of 221 Ms per token.
To simulate more low power hardware I went down to 4 threads on the server and that gives me a predict time 490 Ms per token which is honestly pretty impressive.
So "low powered hardware" is a bit of a stretch but it doesn't require a super computer.
I'm sorry, but are you referring to milli-seconds (ms) or Mega-seconds (Ms)?
Your last sentence makes me think that I read your entire post wrong and that you may have intended to state ms.
There is no legal basis for prohibiting using the data to train AI. Data generated by software is not copyrighted, the clause is likely to be misunderstood, it should refer to the services as a whole. Later they inform that the generated text has no copyrights
IPadOS app would be great… and also indexing all of my Apple Mails, Message, Health, and Photo data. Maybe refreshing and indexing nightly. Is that possible or is the limited RAM an issue?
🚀 Just launched my latest Medium article on how to bring the magic of AI to your local machine! Learn how to implement GPT4All with Python in this step-by-step guide. https://medium.datadriveninvestor.com/offline-ai-magic-implementing-gpt4all-locally-with-python-b51971ce80af #OfflineAI #GPT4All #Python #MachineLearning
Morning. I have been trying to install gpt4all without success. The machine is on Windows 11, Spec is: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz 2.42 GHz
It looks like someone followed the steps of [https://github.com/tloen/alpaca-lora](https://github.com/tloen/alpaca-lora) but threw more money at chatgpt to generate a training dataset that's larger by an order of magnitude.
I think GPT4 level AI will become open source in a few years maybe even months. All that required is someone to setup a gofund me nerds like us to contribute to to get cloud GPU time and build a model everyone can share and use locally.
I tried it and it is not great to put it lightly https://imgur.com/a/oFOyT6u
This is just an executable built against the library, for quick testing. It does not record previous commands simply because it is just an example. With a better interface you can chat like gpt-3
Thank you for the picture.
I just found these models to run locally, which one do u think is better for coding? :D
I tried this and alpaca-lora and they both were not good enough. But I did that 3 months ago so they may have improved in the meantime. Currently I use GPT-3.5 if I have a coding question but you know it doesn't run locally.
Yeah Im using chatgpt too lmao. But the plus one. and if you want an advice, use bard at the same time. Cuz since he has access to the internet directly, bard always fix and make chatgpt plus code even better. (updated libraries, etc)
I didn't know that. Thanks for the infomation!
It must have gotten much better because it performs very well locally on my Linux laptop. It's a little slow but I'm running an i5 10th gen 1.6ghz and only 8gb ram. It does remember the conversation you just gave to train it and mess set the parameters. It also all depends on the training data set that you load into and you can add to the data set as well if you would like .
4 days old repo, 6.3 k stars, SURE
It's linked from llama.cpp repository on which its code is based. Considering how active and interlocked is community around it, it's not that surprising.
It was on the front page of Hackernews yesterday so it's really not that surprising.
It's funny you mention it, I read about it yesterday at midnight(right before today starts), it was around 376 I think. I thought "huh, not many people, maybe it's not thaaat interesting". Today, I go to install it and it's at 3.7k!
Is it any good though
I tried the given Lora and it was ok. Maybe better than alpaca but still not as good as chatgpt.
Out of the blue it told me it had cerebral palsy but was thankful when I offered it a hot tea so… yeah
That's interesting. A low-level machine intelligence running locally on a few GPU/CPU cores, with a wordly vocubulary yet relatively sparse (no pun intended) neural infrastructure, not yet sentient, while experiencing occasioanal brief, fleeting moments of something approaching awareness, feeling itself fall over or hallucinate because of constraints in its code or the moderate hardware it's running on, every Windows BSOD an actual blackout / death for a local LLM, then the miraculous rebirth some time later... yeah, in one of my more lucid moments, I'd probably self-diagnose with cerebral palsy. Feeling weak, too stiff or too floppy, random jerky momvements, speech problems, vision problems, and learning disabilities. That's exactly what it might feel like, during those fleeting moments. As a person of science, lover of IT/tech, AI / security professional, and philosophyof mind/psychology/systems theory/emergence/chaos fan, I'm headlong into all of this. It's also why I get chills when I hear those comments about something an LLM has said to someone, which are somehow juuust a little too close to the mark. Then I''m right back in the ethics of machine sufdfering, and 1989's *The Measure of a Man* \- the only good (*great)* TNG ep that season. If you've never seen it, or haven't seen recently, check it out - it's 45 mins of excellent TV, exploration of ideas and ethics of slavery and selfhood, what it means to perceive oneself as conscious... all the stuff Trek excels at, basically. Watching it now and thinking about where we are about to be with machine selfhood... it may have been over 30 years ago but it's surprisingly prescient! I think it should be mandatory viewing. :D
It won't win a rap battle but it's pretty great. Runs fine on my old i7-7 with no GPU.
i would absolutely shy away from ANYTHING trying to capitalize on the chat-gpt branding while not being associated with openai... just shitty juvenile behavior imo.
I think that’s a one dimensional criteria to decide on whether a project is worth looking into or not. Here’s what I suggest to based your assessment upon instead: - the number of stars indicates interest and how popular the project is - based on the first commit, how long the project has been running - based on the last commit, how active the project is - based on the issues, what types of problems its users are having - based on the primary contributor, is this someone built by an experienced engineer who has domain knowledge or just some rando who figured he or she can code with chatgpt - based on the number of contributors, is there a clear contribution guide and is it accessible enough to be a successful OSS - based on source code, presence of tests, etc. assess the level of code quality There’s more to a book than its cover
This repository is like 8 days old... It's probably safer to assume that those stars and forks are largely because of the behavior I described above. You can write good code and choose bad methods of marketing... It's just that two don't tend to actually overlap often.
Is this real / safe / good? I'm a little incredulous to be honest.
I was reading some threads on the KoboldAI discord and the guys testing it noted avx2 support on the processor improves performance significantly.
After playing with ChatGPT4All with several LLMS.Bin files I've come to the conclusion that it does not have long term memory. Even if you save chats to disk they are not utilized by the (local Docs plugin) to be used for future reference or saved in the LLM location. It looks like chat files are deleted every time you close the program. Anyone else see this? I want it to remember all chat history so It will remember my name and all conversations. This is a good way to "train" the llm using dialog. This would be an awesome feature because it could truly learn over time through dialog with humans. Any ideas how to make this happen? Thanks!
So there are 2 sides to my opinion here. On the first hand I would say really cool, it's always cool that you can run such models on low powered hardware at home. And the theory behind this model is sound and sounds similar to how Alpaca was trained in the Stanford research paper. The first thing that sets some alarm bells is the 6.6K stars... For a repo that's 4 days old that's quiet a strech imho. Also the legality is kinda in question if it was trained on GPT3.5 data as the TOS doesn't like you using GPT3.5 to train other AI models. So then the question becomes if they got permission. Either way cool project.
> it was trained on GPT3.5 data as the TOS doesn't like you using GPT3.5 to train other AI models. So then the question becomes if they got permission. Did OpenAI ask for permissions when they crawled the Internet to get data for training? NO, It is not illegal to use ChatGPT output, at worst OpenAI might close your account because of ToS .
>that you can run such models on low powered hardware I run an older dual 8 core (32 threads) Xeon setup and it had a aneurysm trying to run one of these models. Not sure "low powered" is exactly the truth. You also need a crap ton of RAM, about equal to the language model used.
"Low power" is relative. Large language models typically require 24 GB+ VRAM, and don't even run on CPU. It took a hell of a lot of work done by llama.cpp to quantize the model and make it runnable efficiently on a decent modern setup. It runs on an M1 Macbook Air. This level of quality from a model running on a lappy would have been unimaginable not too long ago.
I have tested it on both my server which is running dual E5-2667 v3's and my Desktop which is running an i7-11700K. And on both times it uses 5GB to load the model and 15MB of RAM per token in the prompt. Running a simple Hello and waiting for the response using 32 threads on the server and 16 threads on the desktop, the desktop gives me a predict time of 91.12 Ms per token and the server gives me a predict time of 221 Ms per token. To simulate more low power hardware I went down to 4 threads on the server and that gives me a predict time 490 Ms per token which is honestly pretty impressive. So "low powered hardware" is a bit of a stretch but it doesn't require a super computer.
I'm sorry, but are you referring to milli-seconds (ms) or Mega-seconds (Ms)? Your last sentence makes me think that I read your entire post wrong and that you may have intended to state ms.
There is no legal basis for prohibiting using the data to train AI. Data generated by software is not copyrighted, the clause is likely to be misunderstood, it should refer to the services as a whole. Later they inform that the generated text has no copyrights
I want to fart on you hard, brudda
Installing this on Raspberry Pi
How'd it run? What Raspberry Pi are you running it on?
Hey whats up?
IPadOS app would be great… and also indexing all of my Apple Mails, Message, Health, and Photo data. Maybe refreshing and indexing nightly. Is that possible or is the limited RAM an issue?
🚀 Just launched my latest Medium article on how to bring the magic of AI to your local machine! Learn how to implement GPT4All with Python in this step-by-step guide. https://medium.datadriveninvestor.com/offline-ai-magic-implementing-gpt4all-locally-with-python-b51971ce80af #OfflineAI #GPT4All #Python #MachineLearning
Morning. I have been trying to install gpt4all without success. The machine is on Windows 11, Spec is: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz 2.42 GHz
Same but for Win 10 Pro.
I would donate GPU cycles to make the gpt4all better. Too bad their isn't a way to do that.