San Diego has a good natural harbor, it borders Mexico, and was the first city founded in California yet SF/LA seems to have always overshadowed SD in everything from population size to arts and culture to being the economic center of California. What drew people to those two cities?
Without answering your question precisely, I'd like to point out that being the first city doesn't always make it important. For example, Jamestown, though the first English colony in America, no longer even exists even though it was the capital for 80 or so years.
Terrain and resources make a larger impact on the success of a colony. Oil was also found in Los Angeles leading a a growth boom in the 19th century. Sand Diego appears to have suffered population-wise as well when it was part of Mexico, not sure if that's because of political reasons, but it happened.
I'll link these two sources which corroborate what I say, but I'm not saying I'm wrong. I found them after a quick Google search so I hope they're decent.
"City" is a very generous term prior to 1849 for San Diego, Los Angeles, Yerba Buena (aka San Francisco) or any other community in what is now California. They were communities based around Spanish missions, but hardly cities. Under Spanish and Mexican rule, Alta California was a rural backwater carved into large "Ranchos", whose landowners sought to pattern themselves after Spain's landed gentry.
After the US won control of California in 1848, there was the famous discovery of gold in Northern California, leading to the Gold Rush and the extremely rapid growth of San Francisco. Southern California was far away from the gold fields and saw limited growth, while San Francisco, with its massive natural bay and huge influx of immigrants, was the catalyst for California statehood.
In 1876 the Central Pacific railroad was built to connect Northern California with Southern California. Los Angeles, as opposed to San Diego, was chosen as the southern terminus of the railroad, and this gave Los Angeles a head start over San Diego in terms of growth. The discovery of oil in Los Angeles in 1892 led to more financial wealth for the city, and by that time the ball was rolling for Los Angeles to eventually become a major city.
San Diego, on the other hand, was left out of most of these developments. Largely agricultural and rural, it wasn't until World War II that San Diego experienced major, sustained growth. San Diego became a major Naval base for the war against Japan, and is to this day the largest US Navy base in the Pacific. During the war and in the decades after, it was a major defense manufacturing city, and grew to be the second largest city in California, after Los Angeles.
For sources, this is basically what every 4th grader in California learns. For more depth and understanding, I would look to something like Kevin Starr's California: A History
I took a seminar on the history of Los Angeles that included an examination of its success relative to San Diego and San Francisco. L.A.'s overshadowing of San Diego rests on a number of factors. First, the fight over the terminus of the intercontinental railroad, and which towns in California received regular service; L.A. won this battle. Next, the two cities battled over becoming the port destination for import and export; L.A. won this battle, as well. Finally, when William Mulholland stole the water rights from Owen Valley and diverted it to L.A., the lushness it brought to the nearby California valleys made the city more attractive than arid San Diego.
Sources:
Avila, Eric. Popular Culture in the Age of White Flight : Fear and Fantasy in Suburban Los Angeles. Berkeley: University of California Press, 2006.
MacGirr, Lisa. Suburban Warriors : The Origins of the New American Right. Princeton [u.a.]: Princeton Univ. Press, 2001.
Nicolaides, Becky. My Blue Heaven : Life and Politics in the Working-Class Suburbs of Los Angeles, 1920-1965. Chicago: University of Chicago Press, 2002.
Scott, Allen. The City : Los Angeles and Urban Theory at the End of the Twentieth Century. Berkeley (Calif.): University of California press, 1998.
Los Angeles had the oil boom and then Hollywood and then everything. (In the early 1920s perhaps 1/4 of the world's oil was coming from areas around LA.) San Francisco had the gold rush and shipping (the bay is one of the best natural bays in the world). Both had the central valley agricultural boom as well. Los Angeles additionally benefits from a very large area of fairly flat land which is perfect for development.
San Diego may have been founded earlier but development requires impetus. For SF and LA there were enormously strong early development cycles in the mid 19th and early 20th centuries which caused the cities to grow large enough to where they could fuel their own growth. San Diego didn't have the same advantages to the same degree so it grew slower.
San Francisco has a better harbor and was near the Gold Rush.
Los Angeles has oil and was easier to build a railroad to from the East.
Back in the developmental era it was all about what was in the news; what was attracting people and development - and qualities of life that make it appealing now weren't really an issue back in history. San Francisco had the gold rush; the LA basin was an agricultural and oil mecca. I can't speak specifically to San Diego's development, but these two things were national and world issues in the 19th century.