Most of nobles weren't oligarchy. They should be considered sth like modern middle class.

According to Tadeusz Korzon [1] in late 18th century 8%-10% of Polish-Lithuanian Commonwealth population were nobles, and in some regions (like Mazowsze) 25% of population were nobility. They all had voting rights, but most of them were poor, and lived similary to peasants, working for others.

Obviosuly you can't have oligarchy with 25% or even 8% of participation ;)

Oligarchy was much smaller group inside nobility - families like Radziwiłł in Lithuania and Zamojski in eastern Poland/western Ukraine had huge amounts of land and private armies.

At different points relative power of poor nobility and oligarchs were different, there were times when magnates bought votes of poor nobles to get more priviledges, and times when poor nobility forced oligarchs to give up some power by using their voting rights (see https://en.wikipedia.org/wiki/Executionist_movement ).

There are also many more reasons for the fall of Polish-Lithuanian Commonwealth, it can't be simplified to one sentence. Among other reasons - Cossacks' uprisings, Swedish Deluge, economic problems caused by falling prices of grain in the most important export destinations. One of the important reasons was - nobles didn't pay taxes unless they agreed to - so the royal army was small compared to the size of the country, and each time there was emergency - there were negotiations about temporary taxes and other ways to boost the army - and magnates (rich nobility) usually gained more priviledges in the result.

[1] Tadeusz Korzon, "Wewnętrzne dzieje Polski za Stanisława Augusta, 1764-94, badania historyczne ze stanowiska ekonomicznego i administracyjnego", t. I, Kraków, Warszawa 1897