This is actually a pre-Christian belief, most likely. It's not an idea that's ubiquitous in the Hebrew Bible (G-d dwells specifically in the Temple, He walks in the garden, etc), it's very clearly established in some places. G-d's discourse from the whirlwind at the end of Job, for example, emphasizes his omnipotence (as does the whole book of Job). For specific instance (that's part of a long list), G-d is even presence in "she'ol", "the Pit", which is used to refer to the abode of the dead in several Jewish texts (Job 26:6). These "omni-" aspects of G-d come up repeatedly in Psalms. As a good example, check out Psalm 139. It's hard to date Psalms exactly, but it was certainly written a few centuries before Christian Jesus's life. Since the Hebrew Bible was canonical from the first canons we have (Ireneus's), it's hard to argue this wasn't a part of Christianity from the beginning.

It's also separately present in the New Testament. There are great lines in the Gospels about how G-d knows how many hairs are on your head (Luke 12:7, Matthew 10:30).

Luke:

12:4 I tell you, my friends, do not fear those who kill the body, and after that can do nothing more. 5 But I will warn you whom to fear: fear him who, after he has killed, has authority to cast into hell. Yes, I tell you, fear him! 6 Are not five sparrows sold for two pennies? Yet not one of them is forgotten in God’s sight. 7 But even the hairs of your head are all counted. Do not be afraid; you are of more value than many sparrows.

Matthew:

10: 26 So have no fear of them; for nothing is covered up that will not be uncovered, and nothing secret that will not become known. 27 What I say to you in the dark, tell in the light; and what you hear whispered, proclaim from the housetops. 28 Do not fear those who kill the body but cannot kill the soul; rather fear him who can destroy both soul and body in hell. 29 Are not two sparrows sold for a penny? Yet not one of them will fall to the ground apart from your Father. 30 And even the hairs of your head are all counted. 31 So do not be afraid; you are of more value than many sparrows.

Similar things are present in Paul, for instance

Hebrews 4:12 Indeed, the word of God is living and active, sharper than any two-edged sword, piercing until it divides soul from spirit, joints from marrow; it is able to judge the thoughts and intentions of the heart. 13 And before him no creature is hidden, but all are naked and laid bare to the eyes of the one to whom we must render an account.

It's mainly G-d's omniscience that's emphasized, but I think the omniscience implies a clear omnipresence. The omnipotence I'm having a harder time finding places where the text explicitly emphasizes this aspect, but there aren't really limits to what G-d can do anywhere in the Hebrew or Christians Bibles (other than little things, like in Titus, Paul says God cannot tell a lie).