If Augustus conducted such a census, he surely would have mentioned it in his Res Gestae. He brags about censuses he took of Roman citizens in 28 BC, 8 BC, and AD 14. The largest of those recorded just under 5 million Roman citizens. A census of the entire empire would be a much greater logistical feat. The idea that Augustus would have conducted such a census and not mentioned it when he did mention these smaller ones is difficult to swallow.

The normal practice was for censors to set themselves up on the Campus Martius, and citizens would present themselves and declare their tribal affiliation, wealth, etc. As the Roman citizenship expanded geographically beyond Rome, and Rome fell into a system of civil wars, this system obviously didn't function anymore. Censuses normally happen in five-year intervals, but they stopped after 115 BC. There was an attempt at a census in 86 BC, but it wasn't completed. The next full census was in 70 BC, and involved the whole Italian peninsula up to the Rubicon. That was the last census until Augustus in 28 BC. For these later, larger censuses, it seems likely that (as provided in the tabula Heracleensis) magistrates in the towns of Italy conducted their count locally and forwarded the results to Rome, but we don't know how the count was conducted before that.

Censuses of the provinces happened through the office (for a lack of a better term) of their governors or other officials for tax reasons, and included assets (land, slaves, etc). These are ad hoc and local.

The OCD mentions a census of Judea in 6 AD for tax purposes, causing some unrest and leaves it at that.

It's not very likely in the sense that there's no evidence for a one-time empire-wide census that involved returning to one's ancestral home. However, very few interpreters understand the passage that way.

Certainly Augustus did conduct census operations throughout the empire, and this was part of program to complete a census of the whole empire, though it appears more likely that they occurred on a rolling basis in different regions (i.e. it was not 'one census', but a series of censuses conducted more locally.)

Much more problematic is the association of this census with Quirinius. We know that Quirinius was legate in Syria from 6-9AD, which is too late for the birth of Jesus (4-6 BC).

Judea at the time is a client kingdom, not a province, and so there are questions about whether a census would have taken place there.

The return-home issue seems problematic, but in other censuses (Egypt in 104) it was practiced, and where a property owner held property they often had to turn up (since censuses are about taxation as much as other things). It is possible, then, that a Judean census involved returning home, particularly if this was seen as an accommodation to local, Jewish practice. Furthermore, for a mostly rural, agricultural society, the number of people travelling is much more limited than I think you are imagining.

The main historical difficulty remains the dating. Conservative attempts to explain why Luke has a census under Quirinius in (4-6) BC, whereas no evidence supports this over against 6AD, have been varied, but none of them has emerged a very convincing. At best, the conservative case for historicity makes it possible that Luke is accurate, and our information incomplete.

The Luke 2 census is one of the most historically 'difficult' texts in the New Testament for claims of historical reliability, mainly because of verse 2, not because of the empire-wide census efforts of Augustus.

I'll just quote a couple of relevant paragraphs of a comment by u/jasoncaspian on this issue:

Two of the gospels deal with the birth of Jesus. Without going into too much detail, it's easy to make the argument that both Matthew and Luke did not get their information for this narrative from the same source. They are constantly at odds with each other over many specific areas of this story (example: in Matthew, Mary and Joseph already lived in bethlehem and then had to move to Egypt and then, years later, move to Nazareth. In Luke, Mary and Joseph lived in Nazareth, traveled to Bethlehem for a theoretical tax registration, waited there for 32 days after Jesus was born, and then returned immediately to Nazareth).

Most historians believe it is likely that both of them made up nearly all (if not all) of the parts to their stories because they were trying to fulfill the prophecies from the Old Testament. See, in the book of Micah, it was predicted that a savior would be born in the city of David (Bethlehem), so these writers wanted to make sure that Jesus fulfilled this prophecy. But wait, they had a real issue to deal with. It was probably well-known that Jesus was from some small town called Nazareth, thus he didn't fulfill that part of the prophecy. So, to deal with this, early gospel writers created these narratives to explain how this person from Nazareth could have still been from the city of David.