There's a few different factors involved, and yes, distance is one. I'd argue that economics is the principal factor, however. Until the late 19th century, Russia's economy was extremely centralized. To quote Glorious Misadventures by Owen Matthews:

"The merchants of Irkutsk, or even the wealthy Demidovs, were good for financing single voyages, but, unlike Britain or Holland, Russia's merchant class was neither wealthy nor bold enough to finance larger-scale, longer-term ventures."

Now, Matthews' book has some problems, and he's talking about merchant voyages in the Pacific, not Siberian expansion, but his point holds true: Without state support, eastern expansion moved in fits and starts.

Furthermore, Siberian growth was hampered by geography and climate. In addition to being absolutely vast, the rivers in Siberia flow north-south, not east-west. That's a big deal in the era before steam. Riverine travel was the most efficient way to get things from place to place before the 19th century, and the trans-Siberian Railroad wasn't finished until the late 19th century. That meant sailing around the world, through the Arctic, or hauling things overland for years sometimes.

In accounts of Russian America, there are stories about sledges and wagons spending years hauling anchors and cannon across Siberia to Okhotsk -- Vladivostok wasn't founded until 1860, after the area was seized from China -- only to have all that work go to waste as a newly built ship was lost in the Pacific or wrecked.

Then there's the climate. Take a look at a map of the currents in the northern Pacific Ocean, and you'll see what I mean. In North America, the currents benefit shipping -- warm water flows from south to north, keeping ports free. Even Anchorage, way up north, needs only minor icebreaking to stay ice-free the whole year, and Seward, a bit farther south, doesn't even need that.

Contrast that with Okhotsk, which is on the Sea of Okhotsk and gets currents from the north. That is emphatically not an ice-free port, and even Vladivostok had problems until technology and civil engineering projects fixed its issues for the most part in the middle 20th century.

One final factor: There was no real reason to be there. Before the California gold rush, California was mostly agrarian. After the gold rush brought hordes of immigrants westward, it grew native industry and could support itself before the establishment of a rail line connecting it to the eastern United States.