If I understand the history correctly (and feel free to disabuse me of any errors), the rise and continued strengthening of the shouen estates is often implicated as a significant factor in the loss of power held by the Japanese imperial court in the centuries following the end of the Nara period, culminating in the establishment of the Kamakura shogunate some four hundred years later.
The tax-free nature and essential independence from the imperial government of the shouen directly empowered local daimyou and temples, enabling them to consolidate ever stronger military and financial power while simultaneously undermining the power of the imperial government in Heiankyou. Land slipped further and further out of imperial hands over the centuries, to the point that by the end of the Heian period there was almost no imperially-owned land left.
My question is, why did the imperial government allow this to continue for so long? Were there no attempts to rein in the system before it led to a complete loss of power by the imperial court? Was it not recognized as a potential long-term problem?
It was a trade-off. Every law comes with the cost of enforcing it and if enough people circumvent or ignore them they can become untenable. An Emperor might want to permit it but due to forces outside their control they cannot. This is true in every century. What changed?
It is up to debate how extensively the ritsuryo system was applied, whether it was all that different from other land tenure systems and how much actually changed, though I'm sure it was a part of life where the central government had influence.
From around 200 to 700 the population about quadrupled before reaching an equilibrium.
https://en.wikipedia.org/wiki/Demographic_history_of_Japan_before_Meiji_Restoration
Land was at a premium and agriculture became more sophisticated, also Japan was hit by smallpox and possibly changes in climate associated with the dark age cold period.
https://books.google.co.uk/books?id=5dmxY_HIWp8C&pg=PA75&dq=land+clearance+in+the+law+codes
These might have been factors in ritsuryo being viewed as too rigid and falling out of favor. It is possible that arbitrary short term political events like smallpox sweeping though the imperial court or a power struggle might have triggered changes in policy away from the confucian idealism of its original architects.