MoEs, Sparsity, and Post-Training Optimizations

After extensive research related to potential MoE inference-time speedups, I've hit something of a wall. It seems that while certain methods do exist, such as expert skipping, it's difficult to actually implement these methods while retaining a high degree of quality. Indeed, I can find no feasible inference-time method for pre-existing MoE models. I think I may take this opportunity to delve deeper beyond the surface and expand my knowledge of MoEs, sparsity, and training-time speedups. While nothing is definite yet, I've developed a twofold plan that should allow me to create a model that achieves fast inference via post-training processes. My first step would be to take an existing small dense model, such as the recently released Nanbeige-4.1-3B, and attempt to make it more competent via Sparse Upcycling. This process would turn the dense model with 3B parameters into a sparse model with 3B activated parameters and over 12B total parameters. Following this, my next step would be to convert the new sparse model into a fine-grained MoE, decreasing expert size. I would then theoretically be able to decrease the number of activated parameters and finetune the router using relevant Agentic/Coding datasets to ensure core functionality remains. I'd use GaLore to accomplish this in a timely fashion with my limited hardware budget. The largest problem I expect to encounter is in re-training the router-- If done incorrectly, I could theoreticall encounter routing/gating collapse, which of course would render the model unuseable. While this path poses many challenges and would be work-intensive, it seems a worthy endevor, given the lack of opportunity for larger model speedups. -K