The Pythagoreans were pretty much a religious cult, but they didn't worship numbers. We're told that they "honoured the study of numbers" more than all other forms of knowledge; to modern eyes, most of this study looks like numerology, or number games. They believed that music, the cosmos, and humanity were ordered in a harmonious way based on arithmetic proportions. The study of numbers came to be particularly associated with the school of Pythagoreanism known as the mathēmatikoi or "advocates of the path of learning", who saw themselves as expanding on "pure" Pythagoreanism by means of ongoing investigation and learning; they were an offshoot from the akousmatikoi or "advocates of the path of hearing", who relied on tenets and aphorisms supposedly passed down from Pythagoras himself. The break between the two seems to have taken place in the 5th century BCE, well after Pythagoras' lifetime, though it wasn't exactly a schism: the mathēmatikoi always recognised that the akousmatikoi represented "orthodox" Pythagoreanism.

It's difficult to gauge the exact importance of numbers in each denomination, or the theological relationship between them, because we have hardly any intact Pythagorean texts, and none within three centuries of Pythagoras' lifetime. But one example is the teaching of the tetraktys, or "universal harmony and music of the spheres" -- literally, the name means "system of four": this seems to have been a common doctrine: this was a form of numerology that seems to have been based on the observation that basic musical harmonies are built on ratios involving the numbers one to four (i.e. a perfect fifth possesses the relationship 3:2, a fourth is 4:3, and an octave is 2:1). Just as musical harmonies are based on these proportions, supposedly cosmic proportions were too. This led them to posit elements of the cosmos that could not be directly observed: since 1+2+3+4=10, they concluded there must be ten spheres making up the cosmos: but because the earth, moon, sun, Mercury, Venus, Mars, Jupiter, Saturn, and the stars only account for nine, they inferred there must be a tenth for an "anti-earth" (whatever was meant by that). (Aristotle reports this snippet of Pythagorean doctrine, and he is very displeased by it. On the one hand, he's justified, because it's clearly nonsense; on the other hand, a prediction that can be tested by experiment is a central pillar of the modern scientific method, so it's hard to know whose side to take!)

In some ways Pythagoreanism seems to have foreshadowed Plato's theory of forms, in the sense that they regarded number as a necessary property of all things, but something that has an independent existence in the abstract as well. That is, you can add two apples to two apples and get four apples; but you can also add "two" and "two" and get "four", without any physical objects involved. But basically they saw numbers as a cognitive representation of the proportionality -- literally harmonia -- that underlies the universe.

There was no worship involved in the sense of sacrifices. They did have tenets that to modern eyes look like more purely religious beliefs, especially the Pythagorean doctrine of transmigration of the soul, and they did observe rites of one kind or another; but it looked much more like a mystery religion than like Olympian cult. They had restrictions on their diet; advanced initiates (at least in the akousmatic school) had to observe a ritual silence for five years; and there's an early tradition that Pythagoras made a point of teaching in underground chambers, since only the practices of mystery religions would allow initiates to gain access to secret knowledge (this last tidbit comes from Timaios, FGrHist 366 F 131 = Iamblichos Life of Pythagoras 143).