Shouldn't Columbus have been able to figure out he wasn't in the indies using the maps of the time a sextant, and a compass?
I've written about Columbus' voyage before, so let me gently correct a couple of misconceptions in your question. First off, "the maps of the time" assumed there was no such thing as the Americas, as the continents weren't known. The maps and globes that would have been available to Columbus would have shown him a much smaller earth by circumference (although he also miscalculated the circumference of the earth) and an "Indies," what we'd call Indonesia, stretching much further east than they do..
To quote myself a bit from a previous answer:
Yes, Columbus appears to have thought that the earth's circumference was about 25% smaller than it actually is. We know that he corresponded with Paolo dal Pozzo Toscanelli, a Florentine scholar who believed that sailing west would be a feasible way to reach Asia. Toscanelli had a map, possibly based on Martin Behaim's globe, that showed Asia extending much further east than it does, and the Azores, Canaries and Cape Verde islands extending much further west than they do (the Azores would end up right off the coast of Labrador in Behaim's projection). Still, Toscanelli thought the Atlantic (from Asia to Europe) was about 5,000 miles wide; Columbus on the other hand interpreted Asia as being much bigger, thinking the Atlantic was only about 3,200 miles wide.
...
Yes, Columbus seems to have been influenced both by Toscanelli and also Behaim, and by a general contemporary thought that the Indies (that is, what we'd now call Indonesia) were a larger island chain going eastward, and that there were even more islands to the east of that -- particularly, it was thought that Japan was larger than it is, further east, and on the equator.
Ptolemy did estimate that Eurasia spanned 180 degrees of latitude (it's closer to 130).
Now as to your specific question:
using the maps of the time a sextant, and a compass?
There are a few misconceptions there. I dealt with the maps issue above, but one thing that's crucial to keep in mind is that finding one's location on the globe only works inasmuch as you understand the size of the globe and that coordinates correspond to known points of land. One can use navigational instruments to find one's spot on the globe, but if the maps you're using are not correct, that location won't correspond to what you think it does. Land being mapped incorrectly was a major problem for early navigators.
Secondly, there was no such thing as a sextant in 1492; the principle of the double-reflecting navigational instrument (what would become an octant or a sextant) doesn't show up until the writings of Isaac Newton, and isn't put into practical use until about the 1730s or so.
Furthermore, a sextant only finds your latitude, that is north-south location on the globe. What Columbus was concerned with is longitude, or the east-west location on the globe; the "discovery of the longitude" didn't happen until around 1760 or so, when methods of calculating longitude using lunar observations, observations of the moons of Jupiter, or other methods, became widely available.