This is because your body is tricked into thinking that the inert gas is oxygen and therefore the body mixes the inert gas into your bloodstream, which reaches your brain and causes extreme brain damage.
Btw, lungs don't transport much helium into your blood - exit bag works because the inert gas has displaced the oxygen, so you just die from the lack of oxygen. And suffocating sensations comes from CO2 build-up, but there no CO2 build-up if you do it right, since inert gas displaces CO2 away from your head. So you have no sense of suffocating with inert gas method, if done right.
(Ref for amount of helium absorbed:
https://www.pftforum.com/blog/gas-solubility-and-why-it-matters)
The lungs act as a reservoir, with breathing 'topping up' the oxygen content. If the lungs contain almost no oxygen, which is what displacement method does, you pass out really fast.
If you hold your breath, you wouldn't pass out for minutes normally because the 'reservoir' is still there, oxygenating your blood with the oxygen still left in your lungs.