How did the United States determine to stop use a gold standard, why, and how does valuing a non commodity backed currency even work? Was this a beneficent plan or a scheme to help a certain group?
Many of these questions are better suited for r/Economics, but I'll try to tackle the econ side of things out of Gregory Mankiw's introductory macro textbook and my admittedly hazy recollection of undergraduate level macro.
how does valuing a non commodity backed currency even work?
I think a more answerable question is what gives a fiat (non commodity backed) currency value, and this is something that u/Village_People_Cop addresses. Fiat currency is basically as flimsy as u/Village_People_Cop notes, but there are actions institutions (e.g. governments) can take to make it more "credible". First, they can declare it to be legal tender, which can help resolve coordination problems. Second, they can require that taxes be paid in currency.
One way to think of currency (fiat or otherwise), is as an IOU, a debt that the backer of the currency (central banks in the case of governments) owes to the holder of the currency. For gold-backed currencies back in the day, these could be literally redeemed for gold as u/Village_People_Cop notes, while for fiat currencies it's a promise that it can be exchanged for goods or services (or at the very least to pay your taxes).
More broadly, you can think of currency in the same terms as other exchangeable assets in that its value will be driven by supply and demand.
The supply side is relatively simple to understand. Analogous to the currency debasing/devaluing that u/Village_People_Cop discusses, if nothing else changes except that you increase the supply of currency, each unit of currency will be worth less. The easiest way to think of this is if you double the amount of currency without changing the amount of stuff available to buy, then price of everything will double (in a highly simplistic world).
The demand side can be thought of in terms of the demand for the function of money. Money serves three purposes:
So, in a world where everyone is a subsistence farmer and lives solely off their own labour, there's not much need for money. Conversely, as workers become increasingly specialized, one would expect the demand for mediums of exchange and hence money to increase. The more people want money, the more valuable it becomes.
Another factor that drives the demand for mediums of exchange is product variety and innovation. The more cool things there are to want, the more we need money to trade for those things.
You may have heard of the value of a given currency discussed relative to another currency in the context of exchange rates. This follows the same logic of supply and demand.
If you've ever gone shopping you probably have noticed that most stores don't accept every type of currency. For example, if I were to visit China I wouldn't expect local merchants to accept Canadian dollars because they wouldn't be able to spend it without exchanging it at a bank or flying to Canada. So the demand for a sovereign currency is also driven by demand for goods that country produces. If everyone wants to buy Chinese goods they need Chinese yuans to buy those goods and so the demand for and value of Chinese yuans will increase if nothing changes on the supply side.
A special exception is the international reserve currency which is the US dollar (for now). This acts as a sort of 'world currency', making it especially demanded because it can be used around the world.
This gets to your second question. Why stop using a commodity backed currency? In order for money to function well, it needs to have a relatively stable value over time. Imagine a world in which today $1 buys you an apple, tomorrow $1 buys you 1/4th of an apple, and on next Thursday $1 buys you five apples. This makes it really hard to plan and budget. With fiat currency, central banks can stabilize the value of money through the supply side. This can't be done readily with commodity backed currency. Of course, if your government is corrupt or incompetent it can really mess up a national economy by screwing around with the money supply.
Fiat currency also allows governments to use monetary policy to manage the economy. By manipulating interest rates and the supply of money, central banks can encourage individuals and businesses to save more if interest rates are high, or spend more if interest rates are low. There are of course trade offs that I won't get into, but you can see how encouraging people to spend can be a very useful tool to keep the economy running during severe negative aggregate demand shocks, such as the pandemic.
As u/Village_People_Cop alludes to, different people can benefit more or less (or even lose) from having fiat vs commodity backed currency. However, mainstream macro and monetary economics is of the opinion that fiat currency has more benefits than drawbacks compared to commodity backed currency for countries as a whole.
Mankiw, N. G. (2003). Principles of macroeconomics.
The entire economic system is based on 1 thing: trust that something will still have a certain value tomorrow.
Non-commodity based currency is literally based on air, there is no physical value to a dollar bill and I can only trade it for a loaf of bread because we all collectively agree that that piece of paper has the value of a loaf of bread. Without that trust and agreement a dollar bill is no better than a piece of toilet paper.
When adhering to something like a gold standard there is still the underlying fact that at any given moment I can go and exchange my dollar for a set amount of gold, something that can be processed into a useful item. Gold or silver are usually used for this because there is a certain rarity to the gold/silver and almost universally gold has had a certain influence and impact on all societies.
The biggest profiteers of dropping the gold standard are financial institutions be it banks or governments because before there needed to be a physical amount of gold in stock in order to distribute money. Without this standard a government can theoretically print more money whenever they need it, this in turn devalues the currency if done to an excessive amount but it can be done.
Lets say the federal government of the US wants to build a new road, they could literally turn on a money printer prints stacks of hundred dollar bills and pay workers and companies for goods and labor to build said road. With a gold standard they need to check is there enough gold in stock and can we print more money, or even worse devalue their currency in order to distribute more money (so instead of 1 gram of gold per dollar I can suddenly only get 0.8 gram per dollar)