Whisk the cocoa powder and confectioner's sugar together in a medium sauce pan.
Slowly pour in the milk, whisking the whole time to mix the dry ingredients completely into the liquid. Stir in the vanilla extract.
Cook the cocoa over medium heat until it starts to boil, turn off the heat. Let the cocoa cook.
Put a few mini marshmallows into an ice pop mold, fill with cocoa (do this in the sink). Insert the tops to the molds.
Freeze overnight. To remove the pops from the mold, run them under warm water. Enjoy!