There are a couple possibilities that show Jerusalem, specifically the site of the temple, was important in Judaism:

1. Abraham almost sacrifices Isaac there (Genesis 22)
2. Jacob has a dream of a ladder going up to heaven with angels ascending and descending

But there are problems with this:

1. Much of what takes place is Genesis is considered mythic^1 and late in terms of writing - 1st millennium (vs the traditional 2nd millennium), so this might be backward retconning of the Genesis story into a much more pragmatic choice by David.
2. Jacob's ladder is an interpretation given much later on by Jewish interpreters of the text, not within the biblical texts.

Text-wise, there's not much beyond point 1 until we get to 2 Samuel 24/1 Chronicles 21, where depending on which passage you take, Yahweh/Satan, incites David to do a census, and as a result of that sin, a plague sweeps through Israel for 3 days killing 70,000 people. Eventually Yahweh tells the avenging angel to stop killing everyone, and the angel obliges by the threshing floor of Araunah/Ornah, which David purchases and builds an altar in it. He then moves the Ark of the Covenant there. Much later, his son Solomon builds the first temple over the spot.

That's the biblical text, what you make of that depends. There's a few commentators who suggest that this is an older story (Angry god -> Plague -> theophany of the plague god) that existed in Canaanite or Jebusite circles and was incorporated and turned Israelite by later writers. The arguments are that there was a pre-existing altar there, and there needed to be a way of explaining it and connecting it to David. However, as McCarter points out, there's already an existing connection between plagues and Yahweh, so we don't need a previous existing story to make this happen. Yahweh causes a plague, David has a vision from Yahweh, builds an altar and so stops the plague, saving the Israelites.

So there's nothing historical about Jerusalem that makes it holy. The site becomes holy because of the plague story and the association with the altar and then Solomon's temple completes the sanctification of the site. The story is an aetiological explanation of Solomon's temple site, not David's.

^1. In terms of giving meaning, not in terms of historicity