First, this is partially an /r/askscience question - the decision is at least as much of an issue of physics as it is of history.
Just to get the physics out of the way first, there are 2 main reasons Florida is a good launch site:
Cost. Proximity to the equator = more rotational energy to work with/less fuel needed to launch. This is why the European Space Agency launches from French Guyana, and not somewhere closer like the Canary Islands or Madeira or something. More rotational energy means you need less fuel to reach orbit, which means smaller rockets for the same payload. Orbital math has a LOT of variables, but to use a quick and dirty and not at all accurate example, a rocket launched from New York City might need 1000lbs of fuel to put 1 lb in orbit, a rocket launched in Florida might need 100lbs of fuel per pound of payload, and a rocket launched on the equator might need 10lbs. Those numbers are insane, but that's the general relationship of latitude: cost.
Safety. Launching rockets is famously difficult and dangerous. Proximity to the ocean = no worries about rocket parts dropping on inhabited areas/causing a forest fire/whatever when something goes wrong. Launches go east, so a coastal location in Florida ensures that any catastrophes would happen over the ocean. Imagine a rocket launched from SoCal breaking up over Phoenix, for example.
Now: with that being said, Cape Canaveral was not in fact the US government's first choice for a launch site. Early US rocket testing took place at White Sands, New Mexico. That range was insufficient for larger rockets, because it was only 135 miles long, and quite close to El Paso/Juarez, Albuquerque, and Amarillo, so the search was on for a new place.
The initial desire was to establish the permanent US launch site in Southern California, for more or less the reasons you described in your post. However, this was eventually abandoned. First, the two issues described above were already beginning to become apparent. Then, a wayward US V-2 launch wound up nearly crashing into Juarez Mexico. The missile impact created a hole 50 feet wide by 24 feet deep, and while no one was injured, it caused a minor international incident and the US government had to settle damage claims.
Mexico rather understandably didn't want to allow US rockets to overfly its territory after that, so other locations had to be sought out. Hawaii and Puerto Rico were both rejected for logistical reasons, but the UK was willing to allow overflights of the Bahamas (then a Crown Colony).
And thus Florida became the location.