We still have both(and others) because they have different advantages for different applications. The Phillips, is designed to 'cam out', meaning that it pops out of the head when resistance is met. This is essential for drywall hanging, for example. Also for drywall, a Phillips head is easier to fill with mud.

The Robertson, on the other hand, is designed to be self holding on the head, and provide large torque. This is useful in mechanical applications, and for when you have to place screws in difficult places.

Other screws are optimized for things like ease of use in dirty conditions, ease of access (head fits in in more angles), tamper resistance, one way operation, ease of manufacturing, etc.

As for the history, we have the slotted screw because that's what we've always had. No patent, cheap to make, and everywhere. This head is probably the least useful design. They were (and still are) used in clock and watch making.

The ubiquity of the Phillips over the Robertson in the U.S. (and the reverse in Canada), is due to Robertson being Canadian, and refusing to licence the design to Henry Ford. Ford had wanted to use Robertsons, but needed to ensure supply. Too bad for the U.S., because the Robertson is superior for most applications. (No, seriously. I have run into USians who've argued the reverse, but they're on crack.)

I remember reading somewhere that the earliest attempt at standards for screw pitch were Ductch windmill makers, but I can't find that on the web.

This wiki page has a large list of heads, along with some info on each. Most others seem to have been made just to be different (tamper resistance), or for small gains in torque or stripping resistance.