I have seen it in movies, cartoons and on figurines. Here is an image to visualise what I mean.
If they did wear flags, did they actually wear them into battle? Did the flags serve a purpose? Wouldn't having a flag sticking out from your back be a huge disadvantage in combat?
They are called sashimono. And yes people actually wore them into battle.
Whether battle screens here and here or deployment screens here and here, we can clearly see they are visible. And they are well attested in written sources as well, whether mobilization orders or field manuals. For example, the zōhyō monogatari specifically recommends ashigaru using yari to aim to knock down the sashimono of the enemy foot. That would be a bit weird to say if the enemy didn't wear them in battle.
It's quite clear usually not everyone wore one. Ashigaru don't wear them in most sources and depictions, though it seems the commander of an ashigaru team (5~10 men?) usually did. Depiction differ on whether all cavalry wore them. The commander didn't wear one. He had his weird helmet and was surrounded by so many flags it's no needed.
Obviously they were for identification purposes, preventing accidentally attacking the wrong side and make both friends and foe better able to see where whose men are. Maybe it also as a local focal/rallying point for the small units.
As to how much a bamboo pole with a flag on it attached to your back effected combat, you'd have to ask reenactors. Personally, I'd think if wearing them hindered combat movement more than they're worth, then people wouldn't wear them. At least the horo that bodyguards and aids wore might have had some protective qualities against arrows.