Certainly they were effective. However, they became obsolete midway through WWII, so they tend not to be thought of, today, as a very effective anti-tank weapon (in comparison to shaped charge based weapons and anti-tank artillery which characterized anti-tank weaponry post about 1942).

The very first tanks such as the British Mark I were so lightly armored they could be taken out with nothing more than special armor piercing rounds in guns already in use, such as the German "K" bullet which could be fired from a standard issue Mauser rifle. This rapidly became obsolete as the Mark IV tank came on the scene, leading to the creation of the Tank-Gewehr Model 1918 anti-tank rifle. The Tank-Gewehr definitely could penetrate a Mark IV's armor and it's use was responsible for many tanks being rendered inoperable. Notably, the Treaty of Versailles forbade Germany from possessing the Tank-Gewehr specifically, an indication of its military effectiveness.

At the start of WWII many anti-tank guns existed, and many were fielded in battle, to varying degrees of effectiveness. The Allies et al made use of the Boys Anti-Tank Rifle, which saw heavy use in the Finno-Soviet "winter" war of 1939-1940. Against the lightly armored T-26 and BT Soviet tanks of the early war the rifles proved effective. So much so that the Finns ordered more of the rifles, ironically from the Germans, who had captured many of them at Dunkirk. Which brings up the fact that the rifles were found to be decidedly not very effective against the Panzer Is and IIs used in the German invasion of France (nor were they effective against Japanese tanks in Malaya), though the Boys rifle is credited with some tank kills in that engagement.

One of the reasons that I suspect anti-tank rifles don't get much recognition for their effectiveness is their method of employment. More modern anti-tank weapons are devastating and typically do their job with one hit. A shaped charge against the hull of a tank will knock it out pretty reliably, large caliber anti-materiel guns will do so as well. Anti-tank rifles generally don't work that way, however. They can penetrate tanks, but they are not one-shot-kill weapons. They are designed to be made use of in concert with heavy rifle and machine gun fire in an ongoing fusillade designed to open up and exploit holes in the armor of the tank in order to take out key systems and render the tank inoperative. And against pre-WWII tanks such rifles were effective in that role. With a couple of machine guns and a couple of anti-tank rifles even a Mark IV or V tank was in for a heck of a slugging match, and one that the tank was far from being guaranteed to win.

If you think about Cold War era tanks as being at a fully "tank-like" point along an armored vehicles spectrum, then looking at WWI era through 1930s tanks you'll see that they are a lot closer to the "armored car" point of that spectrum. They were often riveted together, they had innumerable joints, hatches, and openings. Tanks of those eras often struggled against ordinary rifle and machine gun fire, if you add in several folks punching holes through and cracking/spalling your armor at a rate of once every few seconds, well, things get pretty grim.

During WWII armored vehicle development proceeded so rapidly that weapons were becoming obsoleted at a break neck pace. And armor developed so rapidly that it quickly outpaced the ability of anti-tank rifles to keep up. The roughly .50 caliber rifles of the late WWI and inter-war era were already near the limit of the size of rifle that could be feasibly carried and operated by infantry without being mounted on a specialized vehicle. And such weapons were pretty much useless against more heavily armored tanks like the Panzer IV, T-34, or Sherman. Such armor was not only thicker but sometimes sloped (in the case of the T-34), vastly reducing the effectiveness of anti-tank rifles, and in general much better made. Tank chassis and turrets from mid-WWII were more often forged and/or welded instead of being riveted or bolted together, making them much more resilient to small arms fire. It took big guns or substantial amounts of explosives to destroy such tanks, so anti-tank weapons moved from being hand-held rifles to being much larger guns (field artillery, anti-materiel guns mounted on "tank-destroyer" vehicles), explosives lobbed at the tank (panzerfaust, bazooka, PIAT, etc.), or both (HEAT ammunition). But anti-tank rifles had a solid run up until then.

Additionally, I should point out that anti-tank rifles have continued to see use up through the present day. There are still many of them around, they are effective against armored vehicles other than top-tier MBTs of developed nations, and they are often much easier to acquire than other anti-tank systems. They saw use in the Korean War, the Chinese Civil War, and the Lebanese Civil War (among others).