The source explains that the price of using machine‑learning intelligence is dropping by several orders of magnitude each year, with no sign of slowing. It states that large language models (LLMs) are likely to become part of computing infrastructure rather than a standalone product within the next year or two, and that frontier‑quality LLMs could run locally on commodity hardware within three to six years. The source adds that, going forward, the limiting factor for AI use will be quality and access rather than the sheer number of tokens that can be processed. To support these claims, the source cites evidence across three areas: hardware, model design, and inference software. Regarding hardware, it notes that GPU power efficiency follows a logarithmic trend where efficiency doubles roughly every two years, a rate comparable to the historic Moore’s Law improvements of the 1960s. On the model side, the source describes how the cost to complete a given task—measured on a pareto frontier of quality versus cost—has fallen sharply. It references charts showing that, between the start and end of 2025, models became both smarter and cheaper on a per‑task basis, and that by 2026 the cost axis had shifted two orders of magnitude toward lower expense while intelligence remained roughly flat. The source also points out that smaller models may have a lower per‑token price but often need more tokens to finish a task, so the overall task cost is the relevant metric. Inference engines are highlighted as another source of gains. The source says vLLM, an open‑source inference engine, improved its energy efficiency by about 40% over a 15‑month period from version 0.5.4 (September 2024) to 0.11.1 (December 2025). It also notes that NVIDIA’s MLPerf stack showed up to 50% efficiency improvements from version 2.0 to 2.1, and Intel reported a 2.4× throughput increase solely from software updates between MLPerf 6.0 and 6.1. Architectural advances such as Mixture‑of‑Experts (MoE) are cited for reducing compute needed for a given quality level, allowing models to be far smaller in parameter count while maintaining benchmark performance. The source cautions that MoE’s memory demands limit its benefit on local devices unless techniques like on‑demand layer swapping are used. Finally, the source mentions Mamba‑based architectures, which replace the transformer’s full‑context memory with a lossy summary, cutting the RAM required to run LLMs locally by five times or more. Hybrid Mamba‑Transformer models are said to achieve large reductions in memory footprint while preserving the ability to recall details such as URLs.
Key facts
- GPU power efficiency doubles about every two years (source)
- Cost to complete a task on the pareto frontier fell two orders of magnitude from 2025 to 2026 (source)
- vLLM improved energy efficiency ~40% from v0.5.4 to v0.11.1 over 15 months (source)
- NVIDIA MLPerf stack showed up to 50% efficiency gains from 2.0 to 2.1 (source)
- Intel MLPerf throughput rose 2.4× from 6.0 to 6.1 via software only (source)
- MoE architectures can cut model size 7× while keeping benchmark performance (source)
- Mamba‑based models reduce local RAM needs by 5× or more (source)