Potatoes grow in dirt, and occasionally some gets trapped in the shallow eyes. When they are brought out of storage and sorted for shipping one of the first steps is to flume the potatoes in water from one step to another and this removes most of the dirt. When I open a carton of foodservice potatoes in a cardboard box, I always wash the potatoes before using. Usually there is some dust or sand in the bottom of the box.

What I anticipate that you are saying is a taste of dirt, is usually some sort of mold on one of the potatoes in the bag. Sometimes, rinsing the potatoes off and placing them in a bowl of water with a little concentrated lemon juice will take away the smell or taste when baked or boiled later. Be sure to cut off the mold before washing, if you find any.