An important thing to know is that reenactors will make it a rule to not reenact with the bayonet attached. They do this for safety since these people are not professional soldiers and things happen.
But generally, yes the bayonet was always attached. Rarely will soldiers fight without bayonets and the rare instances will be for weapons (like the non-Baker rifles) that didn't have bayonet slugs or socket bayonets. The importance for this is as you've noted, anti-cavalry and bayonet charges.