Many people were too scared of what would happen to them if they took them off. If you were caught to be a registered Jew without a star on your chest, you were either imprisoned or killed. Plus, in the beginning, most people thought nothing of the stars. They believed it to be just another way to identify people. Then, of course, they became a way of the Nazis being able to tell who to ship off to the concentration camps.
In the holocaust memoir "Paper Love," the author describes how one of her interviewees bravely attached hook and eyes to his star. Once he was a sufficient distance from his home in Berlin (and away from any neighbors or citizens who might recognize him as a jew) he would sneak into an alley and remove the star so he could go to the movies or buy groceries. This was an EXCEEDINGLY dangerous practice, as your papers clearly identified you as a jew. Any checkpoint or inquisitive officer would discover your lack of compliance with the law and arrest you, or worse. I'd highly recommend the book if you are interested in what life was like for the Jews within Germany or nazi-controlled areas who were not transported early in the war.
The practice of marking Jewish people as "other" and separate from the rest of society was not new within Europe. During the 13th Century, Jewish peoples were required to were distinguishable clothing which would prevent Christians from consorting with them accidentally. An example of this, quite similar to the practice of the Nazi German government, would be of the Synod of Narbonne, who in 1227 issued the following: "That Jews may be distinguished from others, we decree and emphatically command that in the center of the breast they shall wear an oval badge, the measure of one finger in width and one half a palm in height..."
Within German speaking regions during this era, Jews were required to wear a "Judenhut", a cone shaped hat which would distinguish them from the rest of society. This would come about once again following the Nazi invasion of Poland, where Polish Jews were required to wear a mark of some kind, although the regulations on what they should be were lax, but the threat of death still loomed over those who did not comply with the order. This continued to evolve to a regulated mark which would identify on sight a Jewish person, and this brings about your question. Why didn't they take it off? Well, the simple answer is fear. Nazi German record keeping was extremely efficient, and although on sight no one could immediately tell that you're breaking the law, any checkpoint, which were common throughout Nazi Germany, could prove that you're breaking the law, and the punishment was severe. Children were required to attend school, although Jewish children were excluded. Many jobs excluded Jewish people as well, so if you were Jewish most people around you were aware of it already, and not wearing a star would raise some alarms.
Essentially, there was severe punishment for not wearing one when you were required to, and the Nazi government employed extensive efforts to search out for Jews within Germany, such as "catchers" who would collaborate with the Gestapo to root out underground Jews within German cities. If you're asking what stopped them? Nearly every facet of Nazi German society stopped them, from the school systems, to the police, to the government, to the everyday person would would not think twice before turning them in to the Gestapo. Any amount of travel was also regulated through checkpoints, and stops within most of Germany.
sources: Twelfth Ecumenical Council: Lateran IV 1215
Medieval Jewish Civilization: An Encyclopedia (Routledge)- "Jewish Clothing", Norman Rose.
The Pity of It All: A History of the Jews in Germany, Amos Elon.
Last Train From Berlin, Howard K. Smith, Alfred A. Knopf.