Yes they did but keep in mind that similar comments were said of all time periods in the roman era. Cicero for example thought Rome was in decline in 50 bce. Plus many others said things along the lines of things were better in the golden age which changed as time wore on. With Cicero saying the end of the Punic wars was the high time of Rome and others saying the time of Cicero was. Now as for primary sources on the fall of the empire there are two major ones.

The first is an account of Priscus an ambassador to the Huns talking about a person who had lived in Rome and now lives with the scythians.

"He considered his new life among the Scythians better than his old life among the Romans, and the reasons he gave were as follows: "After war the Scythians live in inactivity, enjoying what they have got, and not at all, or very little, harassed. The Romans, on the other hand, are in the first place very liable to perish in war, as they have to rest their hopes of safety on others, and are not allowed, on account of their tyrants to use arms. And those who use them are injured by the cowardice of their generals, who cannot support the conduct of war. But the condition of the subjects in time of peace is far more grievous than the evils of war, for the exaction of the taxes is very severe, and unprincipled men inflict injuries on others, because the laws are practically not valid against all classes. A transgressor who belongs to the wealthy classes is not punished for his injustice, while a poor man, who does not understand business, undergoes the legal penalty, that is if he does not depart this life before the trial, so long is the course of lawsuits protracted, and so much money is expended on them. The climax of the misery is to have to pay in order to obtain justice. For no one will give a court to the injured man unless he pay a sum of money to the judge and the judge's clerks.""

The second is called On Military Matters by Vegetius which talks a great deal on the decline of the legion. As seen in the following section.

"The name of the legion remains indeed to this day in our armies, but its strength and substance are gone, since by the neglect of our predecessors, honors and preferments, which were formerly the recompenses of merit and long services, were to be attained only by interest and favor. Care is no longer taken to replace the soldiers, who after serving their full time, have received their discharges. The vacancies continually happening by sickness, discharges, desertion and various other casualties, if not supplied every year or even every month, must in time disable the most numerous army. Another cause of the weakness of our legions is that in them the soldiers find the duty hard, the arms heavy, the rewards distant and the discipline severe. To avoid these inconveniences, the young men enlist in the auxiliaries, where the service is less laborious and they have reason to expect more speedy recompenses. Cato the Elder, who was often Consul and always victorious at the head of the armies, believed he should do his country more essential service by writing on military affairs, than by all his exploits in the field. For the consequences of brave actions are only temporary, while whatever is committed to writing for public good is of lasting benefit. Several others have followed his example, particularly Frontinus, whose elaborate works on this subject were so well received by the Emperor Trajan. These are the authors whose maxims and institutions I have undertaken to abridge in the most faithful and conclse manner. The expense of keeping up good or bad troops is the same; but it depends wholly on You, most August Emperor, to recover the excellent discipline of the ancients and to correct the abuses of later times. This is a reformation the advantages of which will be equally felt by ourselves and our posterity."

So to conclude many Romans thought they were in a great decline by the 400s however many had thought that before. Romans could see that things had got really bad in the late era and wrote like Vegetius how they thought things could be improved.