There is undoubtedly a vast number of sites that have not yet been identified. There are also a vast number of sites that have in fact been identified, but have not yet been excavated in anything resembling an organized manner.

The process of finding a site can take many forms. For most sites with mentions in written records (this includes cities mentioned in holy books such as the Hebrew Bible), there are often clues about their location (other cities near them, mentioned distances to these sites, geographical features, oral traditions, etc). For sites without these sorts of textual trails or written clues, sites can be found in other ways.

In the part of the world I work in, most ancient cities are in the form of manmade mounds, or tells. These tells have a distinctive shape, and often do not fit into the landscape geologically (i.e. there is a mound in the middle of what should be a flattish valley). Examples of tells include Tel Hazor in northern Israel, Tel Barri in Syria, Tell Begum in Iraq. So it is often possible to tell where a site is just by walking through the landscape, and seeing the distinctive flat-topped slightly out-of-place hills made up of cities on top of each other.

In cases where tells are not the primary form of occupation, such as in the Neolithic period or in Europe, archaeologists use archaeological surveys (they use them in areas with tells too). This involves basically walking through an area looking for artifacts, and recording their concentrations. An area with a high concentration of artifacts could be a site.

In the US, and presumably in Europe, often times an archaeologist will do a brief geographical study, and analyze areas where it would be logical to found a settlement (near a water source, along a trade route, etc) and they will conduct another style of survey, where they dig small holes with a shovel at regular intervals (called shovel test pits) to test for the presence of archaeological remains. Again, the concentration of archaeological remains among the network of test pits can inform as to the location of a site. I participated in an excavation of a very small site in New York state which was hinted at in colonial texts, and narrowed down using shovel test pits, and I helped out with the next step of excavation.

So it is certainly possible to set out to locate new sites. In addition advances in remote sensing (Dr. Sarah Parcak, currently at the University of Alabama, has done a lot of excellent work on identifying new sites in Egypt using satellite imagery) have also proven useful in discovering hitherto unknown archaeological sites.

To be honest, we know the locations of many, many unexcavated sites, but have not had the available funding/manpower/a compelling enough reason to excavate them.

One example of a site that has been known for a very long time but has not yet been excavated is Tel Shimron, a site that has been a nature preserve, and has been surveyed, but has not yet been excavated. Even in a country like Israel, that has hundreds of active archaeological excavations every year, many of which are sponsored by foreign institutions, there are many unexcavated sites.

Fortunately, Tel Shimron is set to be excavated beginning in the summer of 2017 by a team of archaeologists who are currently finishing up excavations at Tel Ashkelon (Hopefully I will be one of them, but we will see).