AllocateWhat are GGUF, AWQ, and FP8?
GGUF, AWQ, and FP8 are common formats for storing quantized model weights. GGUF is a file format built for CPU and consumer-GPU inference; AWQ is an activation-aware 4-bit method that protects the weights that matter most; FP8 is an 8-bit floating-point format served natively by modern accelerators.
The format decides where a model can run. GGUF targets local runtimes on laptops and workstations; AWQ and FP8 target production GPU serving stacks. The same open model is often published in all three.
Memory math is the constant underneath: FP16 spends 2 bytes per parameter, 8-bit formats about 1, and 4-bit formats roughly half of that. A 70B model is about 140 GB at FP16 and around 39 GB at 4-bit before the KV cache.
Catalog entries with FP8 or FP4 in the name are served in that precision. Quality loss is small but real; validate on your own tasks before switching a production route.
Try it: GPU VRAM calculatorRelated terms
Allocate is the cloud inference platform for companies that want to train and run their own models.