In a medium saucepan, whisk together cocoa powder, sugar, water, and salt over medium heat until smooth and combined.
Bring the mixture to a gentle boil, whisking constantly for about 2 minutes until it forms a thick chocolate syrup.
Reduce heat to low and gradually whisk in the milk, stirring constantly to prevent lumps.
Add the chopped chocolate and continue stirring until completely melted and the cocoa is hot but not boiling.
Remove from heat and stir in vanilla extract.
Top mugs with whipped cream and marshmallows; add cocoa or a candy cane if desired.