I've never heard of cannibalism in whelks, but I'm sure it's possible. However, I think it's much more likely that these whelks were mating. Higher water temperatures in late spring and summer trigger the females to release pheromones that attract males. Dog whelks are polyandrous, which means that each female mates with multiple males. I suspect that you saw a female and her attendant males jostling for position.
