I'll just speak to the coinage, but I think it'll be a really good jumping off point.

So the most important distinction is that coinage is fractional, whereas bills are not a fraction of anything but simply multiples of the whole. It makes more sense, when talking about their history, to talk about coinage as fractions of that whole rather than decimals- so a penny is 1/100, nickel- 1/20, dime- 1/10, and quarter- 1/4 (or 2/8 as will be important in a moment).

The reason 2/8 is the most relevant denomination for quarters, which are the coin really in question here is that there seems to be at least some linkage between the Spanish milled dollar and the way that coinage was denominated once the US started minting their own. The colonies weren't allowed to print any money of their own to my knowledge, and so the most common currency, rather than English, were actually Spanish coins. The Spaniards had split their EDIT: REALES into milled dollars - what you might have heard called pieces of eight. Two bits is just two pieces of eight, or a quarter of a real. I don't actually know of a direct textual link between this and the fact that we have quarters today, except that the US mint law that brought our own coinage into existence pegged the value of a US dollar to the value of a Spanish dollar originally, so I think it's not a huge leap to assume the commonality and notoriety of the Spanish dollar impacted how we split up our own dollars at that time. I've actually not the slightest idea where the dime and nickel come from given the early importance of pieces of eight though. Maybe someone else can chip in.

EDIT: Adding sources directly here after a couple requests down below. A Short History of US Coinage... by Rowland Hazard and the Coinage Act of 1792 itself, the text of which can be found here in a not so legible format. You might also check out /u/tleilaxan's sources below.

I'll give a bit of background as this is a question best approached chronologically.

The cornerstone of U.S. coinage is the Coinage Act of 1792. Notably, it was enacted four years after the ratification of the Constitution which includes the clause:

The Congress shall have Power To...coin Money, regulate the Value thereof, and of foreign Coin....

^(Article I, Section 8, Clause 5)

This Coinage Act established the US Mint and defined a decimal system of denominations for bullion-based coinage. The primary coins created by the act were a $10 (gold) eagle, a $1 (silver) dollar, a $0.1 (silver) disme (now known as a dime), and a $0.01 (copper) cent. Also enacted was the terminology by which monetary quantities should be referenced.

And be if further enacted, That to be expressed in the money of account of the United States dollars, etc. shall be expressed in dollars, or units, dismes or tenths, cents or hundredths, and the milles or thousandths, a disme being the tenth part of a dollar, a cent the hundredth part of a dollar, a mille the thousandth part of a dollar, and that all accounts in the public offices and all proceedings in the courts of the United States shall be kept and had in conformity to this regulation.

^(Coinage Act of 1792, Section 20).

The US Mint never minted a mill themselves, but several state governments have at various points in history. These other base units are defined to a reference value in the bullion (precious metal) from which they are minted. Silver dollars were to be the value of current Spanish milled dollar, containing 371.25 grains of pure silver. Gold is defined as being proportionally worth 15 times as much as silver. Cents were to contain 11 penny-weights of copper (264 grains).

Besides these primary coins, some coins were to be minted in fractional values. Namely, Eagles were minted also as Half Eagles ($5) and Quarter Eagles ($2.50). Dollars were minted as Half Dollars ($0.5) and Quarter Dollars ($0.25). Dismes were minted as Half Dismes ($0.05). Cents were minted also as Half Cents ($0.005). These fractional coins had the same primary bullion type as their "whole" counterpart. This helps explain the oddities in the varying size and shape of modern coinage. A disme (dime) had a tenth the silver as a dollar making it quite small. Even though a cent was worth ten times less than a disme, the ratio in the value of copper-to-silver was about 71:1. This meant original cents were around 7 times heavier than dismes!

The history since the Act has a lot of small details, but I'll hit on the major ones. In the mid-1800's the Double Eagle was added ($20). Several changes were made to the cent but by the end of the Civil War the cent was smaller, thinner and was now bronze. Due in large part to silver shortages and hoarding, the silver half dime was effectively replaced in 1865-6 by the Nickels (actually composed of 3:1 copper to nickel). Two-cent and three-cent pieces came first and the five-cent piece we know today came the year after. The silver half dime was made officially obsolete by the Coinage Act of 1873 in favor of the nickel.

Federal Reserve Notes were first issued in 1917, beginning the shift toward paper for larger values. Until 1933 these initial Notes were backed by gold and could be exchanged at a bank for gold coins upon request. In 1933, by Executive Order 6102, gold coins were forcibly replaced by paper notes and production of silver dollars soon ceased (though extant silver coins remained in circulation). Until 1964, dimes, quarters, and half dollars still primarily consisted of silver.

To summarize, the first system of denomination after the Constitution was based on a decimal system (with primary coins at $0.01, $0.1, $1, and $10). Those coins all had half-value versions and the Eagle ($10) and Dollar each had quarter-value versions. From this starting base, many of the denominations were removed over the years, and a few added, taking us to what we have today.

The most important distinction between US currency and all currencies in the world at the time (except the Russian ruble) was the use of decimal in US currency and the use of non-decimal fractions, especially in British currency. The pound was based off 4 crowns, 20 shillings, 240 pence, and 960 farthings. One could easily divide the shilling by 1, 2, 3, and 4 since the shilling was made up of 12 pence. The crown, too, was divisible by 1, 2, 3, and 4, and was a 1/4 pound. The UK had also released guineas of 1, 2, and 5 pounds (and 1 shilling more per guinea, but rounded for practical reasons), which represent whole amounts.

By introducing the quarter, the United States had continued the tradition of dividing by 4 that the crown had fulfilled. The United Kingdom in the mid-1800's had tried to practice decimalizing the pound, and introduced the florin of 2 shillings or 1/10 pound, much like the US dime.

To answer the question, another user explains the difference between fractions and whole amounts, which is why low-denomination paper currency issued in the US were called "fractional currency." However, the US had released gold "half eagles" and "quarter eagles" worth 5 dollars and 2 1/2 dollars, respectively. Odd enough, the Russians had released gold coins worth 3, 7 1/2, and 15 rubles in the late 1800's.

You are missing the $2 bill as well. There was, at one time, a 2¢ coin (as well as other coins no longer in use).

Interestingly, for gold coinage, there was the eagle ($10) and then the double eagle ($20), which mimics our bills today.