"First" questions are difficult to answer, because as soon as somebody posts an example, someone else immediately tries to find an earlier example. But let's break this question down a little.

There was never one single "Confederate flag." The Confederate States of America had several national flags, state flags, battle flags, and naval ensigns during its brief run; u/sowser discusses this in What is the history of the Confederate flag and what does it ACTUALLY stand for?

The first such flags that appeared in Washington, D.C. were probably those that had been captured in battle and sent back to the War Department. An 1864 report lists rebel flags captured and returned to the War Department, including numerous battle flags and "national colors." Other rebel flags would have become personal possessions, and some of them became accessions to Washington, D.C.-based institutions like the Smithsonian. For example, the 1926 Annual Report for the United States National Museum includes as one entry a battle flag captured in 1861.

After the defeat of the CSA in the American Civil War, public display of the various flags dropped off. The use of the flag (predominantly the version of the battle flag of the Army of Northern Virginia which we now think of as the "Confederate flag" today) was revived to support the revisionist Lost Cause mythology, at meetings of organizations like the United Confederate Veterans, Sons of Confederate Veterans, and United Daughters of the Confederacy. Units of the United States Army from Southern states sometimes incorporated the flag into their unit insignia, and there were any number of other ways that a "rebel" flag might be displayed, in public or private capacity, in Washington, D.C. after the war in the early 20th century.

I have not been able to locate a specific instance when an actual captured rebel flag was displayed in the Capitol building, there is one flag which was certainly displayed in the Capitol building for a number of years: the state flag of Mississippi, which in 1895 was changed to incorporate the "Battle flag" into its upper quarter. This was ultimately removed from one tunnel in 2015, although I am not sure how long it has been hanging there.

I wish I could provide a more definitive answer to when the first such flag was displayed at the capital. As a point of law, no flag other than the United States Flag is to be flown at the Capitol building, so no rebel flag has ever been flown at the Capitol, but it is possible that a captured rebel flag may have been displayed as early as the American Civil War (the dome was under construction from 1855 to 1866), or at any time thereafter, and there have certainly been variations on the Confederate flag on display, in the form of various state flags like that of Mississippi that incorporate elements from the Confederate flags, or unit patches from military units that do the same, etc. etc. in the early 20th century.