Scaling-up BERT Inference on CPU (Part 1)

Scaling BERT inference on CPU isn't magic; it's managing expectations. You can get 4-8 requests per second on a decent CPU, but try to push for serious throughput and you're hitting real bottlenecks. We're talking about memory bandwidth and context window limits, not just raw clock speed. Trying to scale up large models like BERT inference on consumer-grade CPUs is a painful exercise in optimization, not a simple lift-and-shift.

We need to stop treating the CPU as an infinite resource. Implementing quantization techniques—like INT8 or even FP16 where possible—is non-negotiable if you want anything more than toy demos. Otherwise, you're just wasting cycles waiting for garbage.