NanoMind-S3: Running a 15.2M Parameter LLM on a $4 Microcontroller

Overview

The question NanoMind-S3 starts from is not whether a language model can run on a microcontroller. It is whether a real one can. Not a lookup table. Not a pruned toy approximation. Not a hand-crafted response generator dressed up as a neural network. An actual dense transformer, with full weight matrices, full attention computation, and a real vocabulary, running a genuine forward pass on every token it generates. The answer is yes, but getting there required solving a set of problems that do not appear in standard machine learning literature because the standard literature assumes a GPU. The hardware is an Espressif ESP32-S3 DevKitC-1 N16R8. It has two Xtensa LX7 cores running at 240 MHz, 8 MB of PSRAM, and 16 MB of flash. There is no dedicated floating point unit powerful enough to run inference at any useful speed. There is no memory bandwidth to speak of by server standards. The total cost of the development board is approximately four US dollars. The model running on it is the stories15M variant of the LLaMA-2 architecture, trained by Andrej Karpathy on Microsoft's TinyStories dataset. The full architecture is a 6-layer dense transformer with 6 attention heads per layer, a model dimension of 288, a hidden dimension of 768 in the SwiGLU feedforward layers, a 32,000 token SentencePiece BPE vocabulary, a maximum sequence length of 256 tokens, RoPE rotary position embeddings, and RMSNorm normalization throughout. The total parameter count is 15,206,400. Each token generated requires approximately 30.4 million floating point operations. None of that is approximated or stripped down. The first engineering problem is memory. The original FP32 model weights occupy approximately 58 megabytes. The ESP32-S3 has 8 megabytes of PSRAM. Those two numbers cannot coexist without serious compression. The solution is INT4 packed quantization, implemented in a custom Python script called export_esp32_int4.py. Every weight matrix in the model is quantized from 32-bit floating point to 4-bit signed integers with a per-row FP32 scale factor stored separately. Two 4-bit values are packed into a single uint8_t byte, with the low nibble holding the first weight and the high nibble holding the second. This reduces the model binary from 58 MB in FP32 to 14.74 MB in INT8 and further to 7.49 MB in INT4. The actual binary shipped in the repository is 7.85 MB, which fits within the available PSRAM with room left for the runtime buffers and the attention cache. During inference, each packed byte is unpacked back to two individual INT4 values using single-cycle bitwise operations in C, dequantized to FP32 using the stored per-row scale factor, and used in the matrix multiplication. The dequantization cost is minimal because the bitwise operations are single instructions on the LX7 core. The second engineering problem is throughput. A single Xtensa LX7 core processing a 288-dimensional attention layer against a 6-layer transformer with 32,000 vocabulary lookups per token is not fast. The solution is to distribute the workload across both cores using the ESP-IDF FreeRTOS task system, with Core 0 and Core 1 each handling separate portions of the weight matrix multiplications in parallel. The synchronization overhead is managed through a lightweight semaphore scheme that keeps both cores busy without introducing significant idle time. The third engineering problem is the tokenizer. A standard SentencePiece BPE tokenizer requires a vocabulary data structure and a merge rule table that together consume significant memory. The implementation here uses a single-pool allocator approach where the entire tokenizer state lives in one contiguous memory block, eliminating heap fragmentation and keeping allocation deterministic. This matters on a microcontroller where fragmentation can cause silent failures after extended operation. The result of all of this is a system that generates fluent, grammatically coherent short stories about characters and animals directly on the chip, outputting tokens over UART at approximately 2.96 tokens per second. The INT8 version of the model runs slower due to the larger memory footprint affecting cache behavior. The INT4 version achieves the higher throughput through better data locality and the reduced pressure on PSRAM bandwidth. The model weights, the quantized binaries, the tokenizer binary, and the full firmware source are all included in the repository. The project is MIT licensed and fully reproducible from the provided scripts. The broader point of NanoMind-S3 is not the story generation task. It is the demonstration that the full LLaMA-2 forward pass, with no shortcuts, can execute on hardware that costs four dollars, runs without an operating system, and consumes a fraction of the power of a laptop. That has implications for where AI can be deployed, what it can run on, and what it costs to put intelligence into a physical device.