One of the major considerations in picking a launch site is how close to the equator you can put it. The angular momentum of the Earth adds a fair bit of oomph (that's the technical term) to the launching of a rocket, and the angular momentum increases as you get closer to the equator. To illustrate this, take a look at a spinning basketball (or any kind of ball) -- for the ball to complete one rotation, the areas near the midpoint have to travel much faster than the areas near the top or bottom of the ball. Similarly, as the Earth rotates, the areas near the equator spin faster than those at the poles.

Secondarily, the Cape is also on the east coast of Florida. Since rockets use the Earth's rotation to help boost them, they are generally launched on an eastward trajectory. That means that launches from Canaveral and the Kennedy Space Center (adjacent to, but separate from, Cape Canaveral per se) will have an initial trajectory over the ocean, so if they fail any debris will fall into the ocean. (Also, the reusable parts of the now-retired space shuttle, the SRBs, could be recovered after their splashdown.)

Thirdly, the space launch complexes offer easy access to the Intracoastal Waterway, which means that a lot of large equipment can be floated to the site.

Interestingly, the Soviet Union's space program, facing similar considerations, wound up building their launch program at Baikonour, in what's now Kazakhstan. That's essentially a steppe/high desert environment, but it launches over land. The Soviet program (now Russian) also recovers cosmonauts on land, generally also in Kazakhstan. The cosmodrome there remained under Russian control when the Kazakhs declared their independence in 1991.