Wait—Jev has only been hot for a week, and a local build under 1GB is already out???
A developer took the open-source alternative Laya and natively ported it to Apple MLX as Laya-MLX—
The 421M-parameter version peaks at only 943.6MiB of MLX memory.
The 322M multilingual version drops further to 687.6MiB.
No cloud API, no PyTorch or Transformers runtime—download the model onto a Mac and run.
And small does not mean slow.
On an M3 Max, Laya-MLX’s P50 for a short decision is as low as 7.39ms; the 421M version is still only 13.42ms.
Stuff in 50 questions at once and the 322M multilingual build hits 395 questions/s throughput.
Open-source Jev
By now everyone has seen Jev.
Its biggest difference from ordinary LLMs: it cuts the most time-consuming part—text generation. No chat, no code, no articles… it focuses on one job: making judgments.
(Bert: Hmm? Have I heard this job description before?)
Without token-by-token generation, inference speed naturally pulls ahead of classic LLMs.
That blunt approach is why Jev blew up so fast after launch.
Then the open-source alternative arrived.
Laya—a multilingual, non-autoregressive System 1 decision model.
Same idea as Jev: do not spend compute generating text; output decisions on structured questions. It already has 10.4k stars on GitHub.
It mainly does three things:
choice—pick among options; score—grade against a given rubric; noul—estimate the probability that something is True.
Feed in a state plus the questions to judge; one forward pass returns the result.
Laya currently has three main checkpoints.
The English version is based on ModernBERT-large at 421M parameters;
The multilingual version is based on mmBERT-base at only 322M, supporting 100+ languages;
Plus a 421M version tuned for typed-decisions workflows.
With only a few hundred million parameters, it is already fast.
Original Laya on a Tesla T4 is about 33ms per question; in batch mode it averages 7.2ms/question, with single-GPU throughput in the hundreds of questions per second.
Third-party Jev P50 latency cited by the project is 236–276ms; Laya’s own single-question measure is 32.8ms—roughly 7–8× faster.
On a 2,000-decision typed-decisions test, the fine-tuned Laya version hit 76.6% accuracy (Jev’s published figure was 72.7%).
Open source is here, and it is not slow.
Next people asked: can it get even smaller?
Enter Laya-MLX.
Runs locally under 1GB
Laya-MLX reworks Laya’s inference stack.
Developers could already self-host original Laya, but mainly on PyTorch and Transformers.
mizorewww reimplemented Laya’s full neural architecture on Apple MLX so it runs natively on Apple Silicon.
PyTorch and Transformers runtimes can go; no cloud API either.
After swapping stacks, results still have to match.
Laya-MLX validated all three checkpoints in FP32 and FP16; on 63 validation questions, every answer matched original Laya.
Three models × two precisions = 378/378 comparisons passed—no inference drift.
The test machine: an M3 Max with a 40-core GPU and 128GB unified memory.
For a short question, the 421M English build peaks at 943.6MiB MLX allocation; the 322M multilingual build falls to 687.6MiB.
Speed does not shrink with it.
On the same M3 Max, P50 latency for a short question is 13.42ms (421M) and 7.39ms (322M multilingual).
P95 is only 13.92ms and 7.79ms respectively.
Batch 50 questions and multilingual throughput hits 395 questions/s.
And these times are not bare kernel latency.
Timing covers prompt prep, tokenization, tensor build, synced inference, calibration, and final formatting—only model load is excluded.
In other words, 7.39ms is end-to-end latency for one short decision from input to result.
Laya-MLX can decide Snake moves at 60 times per second.
Each step actually calls Laya again from the current state; a cycle safety layer then corrects dangerous actions.
With the project’s tested compile and prefix-reuse optimizations, a continuous 2,400-step run on the same M3 Max hit—
75.40 moves/s, 0 deaths.
The safety layer visibly intervened twice; only a few scenes needed action fixes.
Three lines to run
Anyone interested can deploy locally on an Apple Silicon Mac.
Requirements: Python 3.11+, macOS 14+.
After installing deps, import the library and load a converted checkpoint:
> pip install laya-mlx > import laya_mlx as laya > agent = laya.load(“aac6fef/laya-mlx”)
Then pass state and questions to agent.predict().
The first run downloads the checkpoint; after that inference stays fully local.
Smallness has a cost, of course.
The 421M English version has only 512 tokens of context;
The 322M multilingual and 421M typed-decisions builds top out at 1,024 tokens.
That 1k context must hold state, instructions, and options together—so it fits short inputs, high-frequency calls, and clearly structured decision tasks.
Gaps are real, but that has not stopped open-source folks shipping overnight—yes, we are already dreaming about the next version!
Jev: I’ve only been viral for a week…
相关文章
