It's actually not. That scene happened when they were running from the bees that came out of the hive Charlie broke. As the ran into the cave, Jack and Kate took off their shirts to get the bees out, and to swat them away, and as the scene progressed, they put their shirts back on.
That scene was the source of one of the best-ever "Lost lines" in my opinion (loose translation):
Hurley: What are you guys doing?
Kate: (embarrassed) My shirt had a couple of bees in it.
Hurley: Looks more like a couple of C's to me...

(Or maybe it wasn't Hurley, maybe it was Charlie. Whomever. Still funny.)