According to dermatologists, you should toss your puffs in the trash immediately. J. Matthew Knight from the Knight Dermatology Institute says that after you use the puff for skin exfoliation, the dead skin cells are trapped in the layers of the puff.
Once you leave the loofah in the shower after using it, the moisture which remains of the warm water causes bacteria to fester within the poofy layers of the loofah, leading to growth of mold and yeast. The massive breeding ground grows within just a matter of hours. Also, using the poofy colors after you shave can cause infections as the bacteria gets into the nicks, leading to skin issues which look and feel awful.
If you don’t want to part with it, then you should better rinse it off after you use it on your skin and take it away from the shower so that it can dry.
Replace the puff with a new one every couple of months in order to make sure nothing hides inside. Smell it and if it is unpleasant, then you should get rid of it. It probably comes from the bacteria. Just buy a new one.