There are quite a few.
The Samaritan religion comes to mind. According to Wikipedia there are about 800 Samaritans, how many of these still practice their traditional religion I don't know but I guess it's safe to say that in comparison with Judaism Samaritanism was not very successful.
In the 5th and 6th century there was a series of rebellions by the Samaritans against the Eastern Roman Empire. Procopius claims that the Samaritan religion being outlawed already sparked the revolt of 529 CE but at least after everything was said and done it was suppressed which probably contributed to its niche existence.
Samaritanism is nevertheless very interesting to biblical scholars, though, as the Samaritans maintain their own Torah tradition.
So I will talk about what I know from undergrad days, but I would love it if another scholar who specializes in this,or perhaps a person of the Bahai faith, could add some nuance to my explanation of the Bahai faith's connection to the Abrahamic religions.
I believe the Bahai faith considers itself connected to the Abrahamic tradition, although it also considers itself connected to various "Dharmic" traditions as well, and, to most religious expressions on earth. Unity of religion (all religion) is an important part of Bahai doctrine. Each religious expression is in connection to the same God in this understanding. The differences in expression come from differences in time/place.
The Bahai faith believes/affirms the prophethood of those mentioned in the other Abrahamic faiths, but also asserts that there have been two more recent messengers to humanity from God (in the 19th c.), the Bab and Baha'u'lla. Because of this, there is a strong connection to both Islamic/Persian influences in the aesthetics and, in some of the elements, of the faith. The writings of Baha'u'lla were considered scripture, and his teachings the message of God. In my understanding, that would be the Abrahamic God, but also, the God behind all faiths.
Here is a great reading list/intro to get you started: http://www.h-net.org/~bahai/rg/rg.curricula.html#1
Oh! And as to how its doing. Anecdotaly, I hear it is growing, and that there are approximately 7 million adherents. There are I think 7 major temples in major cities all over the world. I saw the one in Kampala, Uganda. It was breathtaking. http://www.transindiatravels.com/wp-content/uploads/bahai-uganda.jpg
I know you were excepting the various denominations of the three major Abrahamic religions, but you may want to look again, particularly at the early sects and heresies of Christianity, as some of them had ideas that were extremely far from the norm, such as the Sethian Christians who believed in a spiritual universe before the Genesis creation, and that the God of the Old Testament was actually the Gnostic demiurge, and the snake that tempted Adam and Eve was actually the true god granting the gift of knowledge to Adam and Eve.
I don't have the proper material to cite at the moment, but perhaps someone could discuss the Muslim concept of the Mahdi, namely figures such as Muhammad Ahman, a Sudanese Muslim who essentially proclaimed himself the redeemer of the faith in 1881, gathering a notable religious following at the time.
Any Mormons on here? Would you consider LDS to be a Christian denomination, or its own religion?
edited: was a statement, changed to a question.