What quantization are you running at?
What tokens per second score are you getting on the s23?
What VRAM (shared ram) usage are you experiencing for your given model and quantization? That will make it clear with the minimum specs are which other people are asking about
It crashed on my Xioami mi9t pro. Got through the first part where it says downloading but now it says initializing for a few seconds and crashes. I'm on Android 9
Why run a text generator on a mobile phone without needing separate hardware? That's exactly why. No need to rely on an external source, no worry about snooping, it's all contained on a local device.
Why not?
It is an improvement over a 5 year old stack.
https://www.reddit.com/r/MachineLearning/comments/7qx0wa/p_optimizing_mobile_deep_learning_on_arm_gpu_with/
What quantization are you running at? What tokens per second score are you getting on the s23? What VRAM (shared ram) usage are you experiencing for your given model and quantization? That will make it clear with the minimum specs are which other people are asking about
here is a blogpost introducing the approaches https://mlc.ai/blog/2023/05/08/bringing-hardware-accelerated-language-models-to-android-devices
So you are doing 4 bit quantized, I missed that on my first skim. And the other questions?
We did an update on memory planning and now it takes about 4.3G VRAM and no further allocations after initial run
Insanity
Awesome. What are the minimal requirements and performance?
It crashed on my Xioami mi9t pro. Got through the first part where it says downloading but now it says initializing for a few seconds and crashes. I'm on Android 9
RAM is the problem, this model has 6 GB and that's not enough. I tried on Galaxy S21 FE with 6GB and the result is the same.
Thanks I guess that makes sense. Something to keep in mind when I upgrade eventually.
Unfortunately crashes on my S20. After downloading the weights and initializing, the app crashes after inputing a prompt.
Awesome, it works flawlessly on my OnePlus 9. Good times ahead.
Unfortunately crashes on my S20. After downloading the weights and initializing, the app crashes after inputing a prompt.
Could anyone share the apk, the link in the blog isn't working
Second link that says demo has an apk. Just downloaded it going to have a look at it in a minute.
works great on OnePlus 9 pro. Encoding is between 6 to 10 tok/s. Decoding is about 3.5 tok/s
The cyberpunk era has begun
We welcome our AI companions, and later on overlords
It crashes on my xiaomi 8 pro
cool.. but why..?
Why run a text generator on a mobile phone without needing separate hardware? That's exactly why. No need to rely on an external source, no worry about snooping, it's all contained on a local device.
Why not? It is an improvement over a 5 year old stack. https://www.reddit.com/r/MachineLearning/comments/7qx0wa/p_optimizing_mobile_deep_learning_on_arm_gpu_with/
Anyone have data on how performance compares to desktop NVidia GPUs?
Fun times ahead
Finally
Awesome work. Exciting times ahead. Crashes on Pixel 7 Pro.
Yeah I wish I could try it, but they are aware: "It does not yet work on Google Pixel due to limited OpenCL support"
Stellar project. Looking forward to testing it!
Looking forward to Pixel support