The purpose of an anchor is not to weigh a ship down. In terms of total tonnage, an anchor is pretty small compared to the overall mass of the ship. The purpose of an anchor is to drag along the bottom and wedge itself in the soil on the sea floor. This is what prevents the ship from moving.

The weight of a fluke-style anchor is relatively small compared to the overall weight of a ship, and it would have to be to be hoisted out of the water after all. The anchor works both by catching a fluke in the sea bottom and also by providing a spot to attach a line from the anchor to the ship; this line (technically known as the "rode") is generally set in a ratio of 5:1-10:1, that is, to be 5-10 times the water depth at anchor. The ratio is the "scope." And a greater scope means more of the rode is at the seabed, forming a less acute angle with the anchor.

The rode itself is quite heavy, and can under normal circumstances extend along the seabed for some length before rising to the ship.

The whole system of anchor and rode is what provides holding power, and can be quite strong assuming the anchor finds a good holding bottom (sand or clay/mud).