The Bakamatsu era is the only time where we get to see Edo period military forces in any significant action, and the armies that appeared during this period were much like you suspect - feudal vassals of the Daimyo in question, varying widely in skill, equipment, and size. The size of a force a Daimyo could field would roughly follow the size of his domain (known as Han) - a more productive Han would support more and presumably better equipped samurai.
The Bakamatsu era provides us several examples of forces being fielded to get an example of what an Edo period army may have looked like. The Mito Rebellion in 1864-1865 saw a group of ronin in the Mito han - a wealthier domain traditionally close to the shogun - spark an insurrection with roughly 2,000 men. Initial efforts to quell the rebellion involved 700 men from the Mito han supported by 3,000 men from the Shogun, complete with about 1,000 firearms and several cannons.
That same year, a rebellion in Kyoto took place, pitting forces loyal to the Shogun against ronin and Choshu. This incident gives us a good look at what a typical force for the era may have looked like, as the presence of such forces in Kyoto at the time was the result of orders from the Emperor that the Shogun come to the capital. In mid 1864, a force of roughly equal parts Choshu troops and ronin attacked the Kinmon Gate of the Imperial Palace with the intention of "rescuing" the Emperor and bringing him back to Choshu. The assault was opposed by the forces of the various Daimyo also in the capital at the time - Satsuma, Mito, Aizu, Owari, Kii, Kuwana, Echizen, Hikone, Yodo, and Asao - totaling roughly 50,000 men. The anti-Choshu forces in this incident also included the unique special units that had been created by the Shogun such as the Shinsengumi and Mimawarigumi - units that amounted to secret police forces manned primarily by commoners and samurai, respectively.
The First Choshu campaign that followed this incident would see the participation of most of the han loyal to the Shogun at the time, but in the sources I've read there doesn't seem to be much mention of numbers beyond the fact that they were clearly overwhelming. However, the Second Choshu expedition of 1866 saw an even larger coalition of loyalist han coming together with a force of at least 100,00 men against forces numbering less than 5,000 on the Choshu side. However, the disaster of that campaign gives some idea of the disorganization of the Tokugawa loyalists. In addition to the typical disorganization one would expect from a large force of feudal levies, the Shogunate was by then beset by grave concerns over its legitimacy. While some Daimyo like Aizu Katamori remained fiercely loyal to the Shogun, others offered only nominal support. Many nominally loyal daimyo such as Satsuma and Tosa remained out of this second expedition, and of those that did participate with their forces, many did so unenthusiastically.
Meanwhile, the Choshu forces had learned their lessons from their successive defeats at the hands of Tokugawa loyalists, and, more importantly, the foreigners who had come to bombard Shimonoseki not just once, but twice. They had begun importing modern firearms, smuggling them in through a Scottish merchant under a secret agreement with Satsuma, and they began organizing their men in a more Western manner. Significantly, they had also organized groups like the Kiheitai - a force that stood out among Edo-period units as one made up of commoners. Thus, the more motivated and modernized Choshu forces were able to put up enough of a fight to convince the unenthusiastic loyalists to withdraw.
Coming after the Choshu expeditions, the most significant battle of the Boshin War that brought down the Shogun was perhaps not the best indicator of the Edo period armies. By this point, the Choshu-Satsuma alliance was open, and the forces had thoroughly modernized. While the Shogunate had taken on a French military mission with similar aims, they had done so much less quickly, and the ensuing battle of Toba-Fushimi showed just how critical that was. Despite fielding a force three times the size of the combined Choshu, Satsuma, and Tosa forces, Shogunate forces were easily repulsed. Though not a decisive victory in terms of damage done to either army, it resulted in the Emperor throwing his support behind the anti-Shogunate forces and the Shogun being declared an outlaw. This saw the breakdown of much of the loyalist coalition - those Daimyo that didn't immediately jump ship to the Emperor's side would quickly do so as the forces approached.
During the brief Boshin War, Shogunate forces made attempts to adapt to the new tactics and technology, but the efforts were far too late to make a difference. The Shinsengumi, for example, hastily re-equipped with rifles and made an ill-fated attempt to seize Kofu castle before the Imperial forces arrived, only to be outgunned and routed in a mountain pass on their way there. The tenuous alliance that held on after the fall of Edo continued to fight on into 1869, but they fared little better, despite retreating to a modern star fort in Hokkaido with the remnants of the French military mission.
All the Edo era han 'clan domains' kept samurai retainers. The ranks varied substantially, and were hereditary, with mobility for performance or the lack thereof.
Low level samurai acted as physical guard forces - gatekeepers, reaction forces, local patrol.
Higher level samurai usually acted as administrators for the daimyo 'big name' aristocrats, collecting taxes, planning public works, teaching, etc.
The shogun in Edo had direct ˆsamurai retainers called hatamoto, primarily divided into two classes. The upper class again provided administrative services, running the city, acted as inspectors, while the lower class gokenin provided armed security, corvee' labor, etc. Total numbers were below 20,000.
Another group not well known today are the hyakunin gumi - the '100 man units'. A number of areas outside Edo were given to samurai retainers and their responsibility was to maintain and be able to field '100 men' armed and capable of fighting on short notice; the shogun planned them as a sort of quick reaction force of musket-armed fighters. Some of the tales of the 'ninja' in service of the shogun come from these groups, which mostly were strung west of Hanzomon, Hanzo Hattori's area of responsibility at the western gate of Edo castle. Iga, another 'ninja' area, also maintained a hyakunin gumi in the area.
The lack of a central standing army was a feature of the Tokugawa bakufu. The shogun made certain that Edo was ringed by friendly, rich daimyo that could field substantial armed samurai, so every contingency in the land meant that the shogun had to call on those friendly daimyo to field forces at great expense. But they had no modern weapons, drills, tactics, anything standard.
When Commodore Perry and other foreigners started sailing into the heart of Tokyo Bay and into Osaka Bay, it showed the weakness of the system. The Tokugawa bakufu had limited the size of ships to counter the possibility of attack by sea, so it had no defenses against such an approach.
We'll get into the military system and the lives of some of the key hatamoto that play a role in Meiji Japan in The Kanô Chronicles. www.kanochronicles.com