Mixture of Experts (MoEs) in Transformers

Here's the thing, MoEs in Transformers is just a fancy way of saying 'we're using multiple models to make predictions'.

I need to see some actual benefits from this, like improved performance or faster training times. If it's just a bunch of marketing fluff, I'm not interested.

That being said, if they can actually deliver on this, it could be a useful tool for developers. I'm talking about the potential to create more accurate and efficient models.