Inference for PROs

Inference for pros means optimizing the cost per token, plain and simple. Most people just throw compute at it without looking at the actual hardware cost. If you're running models on cheaper hardware, you need to figure out if the latency savings outweigh the increased complexity. We're talking about optimizing quantization techniques, maybe using 4-bit precision, to squeeze performance out of whatever GPU you have.