Your question is as much a matter of science as it is history and culture. First off, your premise is a bit off. The cereal grains you note may be ground now (though oats are a definite exception-- there's plenty of oatmeal eaten today), but historically were often not. Frumenty, for example, is a medieval dish of whole wheat, barley, or other grains boiled in milk (like porridge). Whole or cracked grains were also commonly added to soups and stews (pottages) to bulk them out and thicken them. And boiled whole or cracked grains are of course used in making alcohols of all kinds.

In Asia there are also staple grains that are commonly ground; chickpea flour is a staple of South Asian cuisine that is turned into everything from doughs to fritters to soups (try making kahdi if you have the patience, it's excellent). Millet is also a staple cereal grown widely in South Asia and ground into flour fairly commonly. You also find rice flour used in Japan for a wide range of uses-- Mochi, for example, is made out of rice flour. And you can find rice noodles across all of the cuisines of southeast Asia.

Grinding grains has advantages: you can sift out some of the harsher bran, making a cleaner product that is more easily transportable, accepts water more readily and that you can turn into a smooth paste. And even just cracking, let alone grinding, grains makes them cook much faster and more evenly. But grinding grains takes a lot of work, and historically your ability to buy a loaf of bread had a lot to do with how much money you had and your proximity to good millers and bakers. In many cultures, good white bread was a status symbol since it meant you had access to millers who would grind it fine enough and sift the bran out of the grains, as well as the money to pay for a product that had more waste.

As for the question about bread from rice flour, to some degree it depends what you call "bread". But for the classic loaf shape you're likely thinking of, you need a key ingredient: gluten. Despite its recent fall from grace in the public eye in the 21st century, gluten is a massively important building block for many of our cuisines; it creates a strong, stretchy network into which yeast can blow bubbles of CO2 during the fermentation process of bread. Without that (or some other contemporary scientific jiggerypokery to replace it) you simply don't get leavened bread. And wheat (and to a much lesser extent, barley and rye) is the grain where gluten is found (or rather, wheat contains the proteins that form gluten when you add water to it).

Rice simply doesn't have gluten in it. So while rice flour has been used in modern times as an ingredient in bread, without wheat, the best you can likely do is crackers or noodles. That's not to say that ground rice doesn't have its place in many, many dishes. But it doesn't behave the same way because it doesn't have the same chemistry.

For more on bread, I recommend Bread: A Global History by William Rubel

Searching for information on gluten gets you a million results breathlessly telling you how bad it is for you, but if you want to learn more about the science of it, you could do much worse than this article from the good people at Serious Eats.

Edit: Oats don't produce gluten. Thanks, u/etherizedonatable