One of the earliest firearms or proto-guns, the Chinese fire lance, was essentially a spear attached with a bamboo tube of low nitrate gunpowder. The fire lance was first documented at the siege of De'an, circa 1127 to 1132. The Chinese military treatise Huolongjing (part 2, chapter 2, page 240) has this to say:
A pear flower tube is bound tightly to the end of a long spear, and ignited when face to face with the enemy. As it burns the flames shoot forth as far as several dozen feet. Anyone that gets in the way is inevitably burnt to death; and after the fire has ended, you can still use the spear-point to pierce the enemy through. It is the best of all fire weapons.
While this fire lance is more akin to a flamethrower, later fire lances would fire pellets made from lead. The boldfaced line from the quoted passage above demonstrates that early firearms could be used as a melee or a ranged weapon.
Source: Science and Civilisation in China: Military Technology: The Gunpowder Epic Volume 5 Part 7
Not that before, but in XVIIth century Russia before bayonet came - berdysh-ruzhyo (бердыш-ружьё): https://upload.wikimedia.org/wikipedia/commons/e/ec/01_086_Book_illustrations_of_Historical_description_of_the_clothes_and_weapons_of_Russian_troops.jpg
And how original berdysh was used: http://spiculo.ru/wp-content/uploads/2013/07/4.jpg
My answer is: Even if you didn't have a bayonet, you were trained to act like you did.
Some armies or more specifically army regiments that were not equipped with bayonets (such as militias) would be taught to brace their muskets/flintlocks/rifles into their hips as if they did have bayonets. The thought behind this was that enemy units at a distance may mistakenly believe that your regiment was armed with bayonets. Horses would also behave similarly to an infantry square whether they were bayonet tipped or not. Furthermore, what is still taught today in military training is that you sometimes do not have time to fix a bayonet and must thrust with the "blunt" (I use that term loosely) end of your weapon and melee fight using it as a bludgeoning weapon. The ends of most barrels are actually thin and durable enough to at least crack ribs if you stick someone hard with it. If someone thrust this into your chest it probably wouldn't immediately kill you but it would definitely injure you. Lacking a bayonet didn't mean they couldn't still jab with their barrel. http://www.warrelics.eu/forum/attachments/world-firearms/43028d1245528320-1860-s-english-musket-musket-5.jpg Modern Army Technique for muzzle jabbing. Page 16: http://www.bragg.army.mil/18abc/FBCS/Documents/Basic%20Combatives%20Course%20(Level%20I)%20Handbook.pdf Older Army manuals with similar depictions can be found on Archives.gov
There are quite a few examples of this. Knife-guns. Shield-guns. My favorite is the mace-gun of Henry VIII. I don't think any of these were intended as infantry weapons, no necessarily that effective in battle.
The musket could already be turned around used as a club though. This is how a foot soldier would have fought hand to hand before the introduction of bayonets.
Firearms have doubled as clubs almost since their conception, not to mention that soldiers would have armed themselves with a sword or other sidearm to use in close combat.
The widespread adoption of the bayonet really had more to do with the increasing reliability and effectiveness of firearms. A short, 10 lb matchlock with a plug bayonet would not have been very effective in melee against a 10 foot pike.
Pistol swords were a thing as well.
A combination of a sword, and wheellock or flintlock pistol, with the muzzle pointed in the same direction as the blade.
I believe the Polish^Hussars? cavalry at one time used an Axe-Gun.
http://1.bp.blogspot.com/_59GYpEVAu0U/TRhPlMZmZqI/AAAAAAAAApM/IIvitC0dhlo/s1600/flintlock-axe.jpg
