It has to do with on/off being the easiest thing to represent using various systems. For example, punch cards most easily used binary by the presence or not of there being a hole in the card. Switches are either open or closed. It's easiest to compare high and low (or no) voltage than spitting the voltage levels into three or more levels (important for transistors).

Additionally, classical logic is basically a choice between true or false, which is easily transformed into binary. Logic was an essential building block of modern computing.

At each stage of computing (except for some of the first electronic computers, which used decimal because of its familiarity), using binary was easier and more reliable.

Non binary systems have been tried. Most famously, the Soviets built the Setun computer at Moscow State which used a balanced ternary (1, 0, and -1, rather than 0, 1, 2 in ternary) system and logic. Theoretically, balanced ternary has some computational advantages over binary, but has not been implemented on a large scale.

My question would be, when exactly did scientists come up with the concept of binary computing? Was there a problem with analog computing that led them to the concept of binary?