If we did that it would be much more expensive, keeping all weights in SRAM is done by Groq for example.