Which seems sufficient for mortals who are not running gas town, but want a local model.
khalidkhair 7 minutes ago [-]
Not sure if the measurement method used for this is actually valid or reliable enough, but for reference, Codex 6.1 Sol Standard runs at 19.9 tok/s according to a benchmark I recently saw: https://www.reddit.com/r/codex/s/b4mUszSOv1
kolinko 37 minutes ago [-]
60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.
serf 2 minutes ago [-]
>.. nor any sort of agentic work.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
brcmthrowaway 33 minutes ago [-]
Good luck when you need to prefill context..
InsidiousWombat 1 hours ago [-]
I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
kadoban 1 hours ago [-]
I think what we're going to see first is essentially gpus with nvme (or at least flash) bolted on, not sure if that's also what you're saying or not.
purplecats 58 minutes ago [-]
would this work on a virtual disk, say w s3 or dropbox
trollbridge 13 minutes ago [-]
Sure, it’ll just be absurdly slow.
Maybe next you can run an LLM off of some old MFM drives
doublerabbit 32 minutes ago [-]
Not online services. The overhead of TCP and latency I imagine would really lag performance. A SAN over SFP, maybe.
KerrAvon 58 minutes ago [-]
There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.
Which seems sufficient for mortals who are not running gas town, but want a local model.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
Maybe next you can run an LLM off of some old MFM drives