The premises of this question are half-right. Russia isn't normally my specialism although I happen to be read up on a few relevant areas so I'll give this question the best stab I can.
###Africa
There was actually an attempt by Cossack adventurers to establish a Russian presence in or near Ethiopia, ostensibly to assist the 'Orthodox' Ethiopian Empire against British and Italian predations. In January 1889, Nikolai Ivanovich Ashinov led a band of just over 160 men, women, and children in seizing the fortified village of Sagallo on the coast of Djibouti, which was rather ostentatiously renamed New Moscow. Ashinov had been a wanted criminal in Russia and his involvement in the region was never formally sanctioned by the Russian government (though it received considerable support from several officials acting on personal initiative), which latterly disavowed any connection with his expedition, and within less than a month of its establishment the Cossack colony was assaulted and dispersed by the French navy.
The Russian government's relative disinterest in the African continent was, all told, quite comprehensible. For one, most of its geopolitical interests simply lay elsewhere, but for another, the logistics of trying to establish an African empire from Russia would have been quite problematic. Now, it's not as though the Russian Empire had no maritime presence, as it did after all have four coastlines: the Arctic, the Baltic, the Black Sea, and the Pacific. But to get to Africa from any of these would have been a bit of a problem.
The Arctic is pretty obvious.
Getting out from the Baltic would have meant sailing through the Danish Straits and/or the Kiel Canal, and then either the North Sea or the English Channel. That meant that Russia would have to be on at least terms with Britain and also either Denmark or Prussia/Germany to sustain a maritime link between Africa and its Baltic ports such as St Petersburg and Riga. Moreover, Russia's naval power in the Baltic had been curtailed by the 1856 Treaty of Paris at the end of the Crimean War, which demilitarised the Åland Islands.
Getting out from the Black Sea meant sailing through the Bosporus and Dardanelles, i.e. past the Ottomans, whose territorial integrity had also been guaranteed by Britain and France. Moreover, the Treaty of Paris had mandated the demilitarisation of the Black Sea and Russia would not begin rebuilding its naval presence there until after the conclusion of the 1877-8 Russo-Turkish War, and even with that remilitarisation Russia was still bound by the 1841 London Straits Convention which prohibited the movement of non-Ottoman warships through the Turkish Straits. Even if you could send a civilian ship out via the Straits, its options out of the Mediterranean were either via the Strait of Gibraltar (which meant sailing past Italian, French, and British territory) or the Suez Canal (owned originally by France and latterly by Britain). So, as with the Baltic, you had to be on decently friendly terms with a number of major, potentially rival powers to keep your sea lanes open.
The Pacific at least was a decently plausible option, but there were some obvious limitations. Russia did not have a blue-water port until the establishment of Vladivostok in 1860, with previous bases such as Petropavlovsk and Nikolaevsk being white-water (that is, they were iced in for several months of the year). White-water ports, simply put, are substantially limited by those climactic conditions. But while the Russians did eventually gain year-round maritime access to the Pacific thanks to Vladivostok, the extent to which goods and people could be moved to and from the region from European Russia was, for a good while, quite limited, thanks to the absence of a railway. The Trans-Siberian Railway, which would eventually provide that link, would not even begin construction until 1891, and would only be complete by 1904.
In effect, even if Russia did have ambitions in Africa, there was simply no practicable way of realising them from a simple logistical standpoint. But also, and I would argue this may be more important: why go overseas for empire when you could just look out over your land borders?
###East Asia
To put it bluntly, Russian expansion in East Asia most certainly did happen, especially if we include Manchuria in that definition. Russia exploited Qing weakness during the Arrow War (1856-60) against Britain and France, itself a result of Qing military weakness resulting from the ongoing Taiping War (1851-64) against domestic rebels, to expand its territory in Siberia to regions that had been officially within Qing borders since the Treaty of Nerchinsk in 1689. The Nerchinsk treaty had affirmed Qing control of the entire Amur watershed, although actual Manchu rule in the region was relatively weak if it was apparent at all beyond a series of border markers – indeed, Nikolaevsk would be founded in Qing territory in 1850. The 1858 Treaty of Aigun, which preceded the signing of the Treaty of Tientsin with Britain and France, saw the Russians gain control of all formerly Qing territory north of the Amur, and the subsequent Convention of Peking also saw Russia gain control of the land east of the Ussuri, with an additional border drawn south of Lake Khanka, giving Russia a small border with the Korean Joseon kingdom as well as its long-desired blue-water port at Vladivostok, formerly the Manchu town of Haišenwai.
While Russian territorial control lay mostly beyond the Amur and Ussuri, there was an expansion of Russian influence in the Manchurian interior as well, especially in the wake of the establishment of a naval base at Port Arthur in 1898 (on which more in this answer). Russia's position in the region would be leveraged following the First Sino-Japanese War in 1894-5, when it issued a diplomatic protest to Japan along with France and Germany in what was known as the 'Triple Intervention' to limit Japanese gains, and it would also gain a hand in Korea thanks to the Russian embassy becoming a refuge for King Gojong from both pro-Qing and pro-Japanese factions. Gojong's declaration of the Korean Empire in 1897 (on which more in this answer) was undertaken on the implicit understanding of Russian support. And, in 1900, it would be Russia that dispatched the largest intervention force during the Boxer Uprising, launching a major invasion of Manchuria with around 100,000 troops, only a portion of which were withdrawn following the signing of the Boxer Protocol. In the wake of this, an extension of the Trans-Siberian Railway, known as the Chinese Eastern Railroad, would be established across Manchuria in 1903 as a shortcut to the Trans-Siberian route that followed the Russio-Qing border, and garrisoned by the Russian forces left over after the 1900 intervention.
Russia's long-term interests in the Pacific would be somewhat curtailed thanks to its utter defeat in the Russo-Japanese War of 1904-5, but although Russia lost its lease on Port Arthur, its Manchurian railway concessions, and its protectorship over Korea, it did still retain the main part of its Pacific coastline, which, when the Russian Empire became the Soviet Union, served critical strategic purposes in both the Second World War and the Cold War. The existence of Russian imperial territory in East Asia isn't just a 'was', it's an 'is'. The bulk of those imperial holdings still exist as part of the modern-day Russian Federation.
###Other Imperial Expansions
I think it's worth adding a little coda here by pointing out that we tend to conceive of imperialism as a maritime endeavour when it really isn't. Imperialism is about control of land, and the fact that our colloquial images of colonial and imperial powers happen to be ones whose land holdings were separated by water is arguably just coincidence. Russia's imperial expansion in the latter half of the nineteenth century was considerable, and extended across multiple regions. The most territorially substantial was its conquest of Central Asia from the 1850s to the 1880s, but there were also considerable operations against the Ottoman Empire and Iran. These conquests saw Russia extend its control over Dagestan, Chechnya, Georgia, Armenia, and Azerbaijan, just to name the most substantial of the Caucasian polities, and Russia would also force the Ottoman Empire to cede independence to Romania, Serbia, Montenegro, and Bulgaria. That Russia's expansion was over contiguous landmass doesn't make it any less of an imperial power, and ultimately the aims and modes of that imperialism were entirely analogous to those of other imperial powers: presumptions of racial and/or civilisational superiority mixed with pragmatic desires for natural resources and strategic territory.