Lily: The Local AI That Beat Apple on Its Own Macs

Lily is a key part of Perplexity Engineering’s local AI on Mac strategy. The idea is straightforward, but commercially meaningful: move some inference onto the device so private files and applications can be handled without sending that data off the machine, while the cloud remains responsible for deeper research and more complex reasoning. In business terms, this creates a split between sensitive execution and heavier compute, which can matter when privacy, latency, and operational control are priorities.

What changed with Lily

Lily is a local inference engine built for Macs in Rust with handcrafted Metal kernels. According to the source, it bypasses traditional stacks such as PyTorch or MLX in its execution path. That matters because it signals a shift away from general-purpose tooling toward a highly specialized implementation. Lily is not presented as a universal engine; it is designed around a specific architecture and workflow.

Why local AI on Mac matters

Local AI on Mac can be useful when a company needs to work with internal documents, personal data, or private applications without uploading them to the cloud. A conceptual example: a legal team may want to summarize contracts on-device; an operations team may need to query internal files without exposing them externally. The value is not only speed, but also control over where data is processed.

Performance and practical meaning

The source says Lily outperforms Apple’s MLX-LM on the same hardware with the same model. In the cited tests, it averages 1.23x in prefill and 1.35x in decode, with 4,156 tokens per second for reading versus 3,388, and 170.0 for writing versus 126.4. At a 4K prompt and 4K context point, it also shows higher token throughput. Still, these numbers should be read as evidence of specialized optimization, not as a promise for every model or workload.

Limits, risks, and evaluation

There are important constraints. The reported results were achieved on a MacBook Pro with M5 Max, a 40-core GPU, and 128 GB of unified memory, so they should not be generalized to a standard 16 GB Mac. The 35 billion parameter model used in testing is not feasible on that class of machine. The source also says the public GitHub release is a standalone demo inside the pplx-garden repository under an MIT license, which means it is exploratory rather than a finished product.

Evaluation checklist

• Confirm whether the use case truly requires local data handling.
• Check whether your hardware is comparable to the demo setup.
• Determine if the task is narrow and repeatable.
• Compare results against MLX-LM in the same environment.
• Weigh performance gains against operational complexity.

How to apply it in your business

Local AI on Mac may fit organizations that handle sensitive information and need low-latency inference. Before adopting it, decide whether your main goal is privacy, speed, or specialization. If your work depends on broad compatibility, limited hardware, or very large models, this approach may not be the right fit. If you need controlled on-device execution for specific tasks, Lily shows one possible path within a hybrid architecture.