Well, this is not a very complete or in-depth answer, but the Oxford English Dictionary suggests, just in terms of timeline, that the fastener sense is much older in English (14th century or so). From there you get a sense which is in reference to those, "buttons" as anything that is a very shallow cylinder shape, such as nail-heads or even some parts of plants. It is only in the late 19th century that the "push-button" comes out of that, which is to say, button-shaped things that you push to complete an electrical circuit of some sort.

I'm not sure there's a deeper history there; it seems like the secondary sense, where "button" is just a metaphor for the shape, that seems to have rather quickly been applied to another use.

The more interesting question, I guess, is why the shape of the push-button was so relatively standardized, how that came about, why push-buttons displaced other kinds of switches and levers, etc. That I don't know.