Seabirds would definitely be one indication they were near land or rocks. Another indication was the sound and sight of breakers crashing against rocks or reefs, the white foam of which could be more easily seen in the dark. Other indications of land could be lights on shore and seeing deeper blackness in one direction than the surrounding blackness, indicating land. Another precaution was to measure the depth of the water (called sounding) to help judge whether they were close to land. Still, it was indeed very dangerous sailing at night and a lot of horrific shipwrecks happened as a result of it throughout history.

Ships would usually try to put distance between themselves and land, especially at night, but depending on the period and location they could sometimes hit an uncharted reef or landmass or iceberg, or be wildly off course from where they thought they were. This was especially common before the invention and widespread use of the marine chronometer in the late 18th and early 19th century. Before then, navigation was much more crude because it depended on unreliable timekeeping devices (I wrote a little more about this here) which made it difficult to accurately measure longitude, although they could accurately measure latitude (basically this meant they could tell how far from the equator they were but it was more difficult to tell exactly where along it they were). Obviously that's really dangerous and it was so worrisome that ships at sea would often hail each other just to compare longitude estimates.

I can list many examples of shipwrecks occurring under circumstances like this. In the early morning of June 4, 1629, the Dutch East India Company ship Batavia sailing from the Netherlands to the Dutch colony in Indonesia was wrecked on the Houtman Abrolhos islands about 50 miles off the uncharted western coast of Australia. Not only were these islands uncharted but the ship was doubly off course when it wrecked because 1) the skipper Ariaen Jacobsz seems to have deliberately steered it off course in order to separate it from the rest of the fleet so he could launch a mutiny and 2) because even he may have steered it further than he intended since it unexpectedly wrecked with him on it, thus ruining all his plans, although he didn't know about the islands. This is a translated excerpt from the captain Francisco Pelsaert's journal:

Fourth of June, being Monday Morning, ... with a clear full moon about 2 hours before daybreak during the watch of the skipper [Ariaen Jacobsz], I was lying in my bunk feeling ill and felt suddenly, with a rough terrible movement, the bumping of the ship's rudder, and immediately after that I fell out of my bunk. Whereupon I ran up and discovered that all the sails were in top, the wind southwest, that during the night the course had been northeast by north, and that lay right in the middle of a thick spray.

Round the ship there was only a little surf, but shortly after that I heard the sea breaking hard round about. I said, "Skipper, what have you done that through your reckless carelessness you have run this noose round our necks?" He answered, "How could I do better? I did not sleep, but watched out very well, for when I saw the spray in the distance I asked Hans the gunner, 'What can that be?' Whereupon he said, 'Skipper, it is the shine of the moon;' upon which I trusted.

I asked him, "What council now? Whereabout do you think we are?" He said, "God knows; this is a shallow that must be lying quite a distance from the unknown land [Australia], and I think we are just on the tail of it.

As you can see in the bolded text, the lookout mistook waves breaking on the reef as merely reflections of the moon but he was severely mistaken. Of the ~320 people onboard when the ship wrecked, approximately 70 drowned while ~50 of the sailors and officers sailed away in the longboat to get a rescue ship. Of the ~200 survivors who remained on the islands, about 115 were murdered over the next three months by a gang of mutineers before a rescue ship finally arrived. There is a great book about this horrific incident by Mike Dash called Batavia's Graveyard: The True Story of the Mad Heretic Who Led History's Bloodiest Mutiny.

A century and a half later, another deadly shipwreck occurred in very similar circumstances. In the early morning of August 4, 1782, the British East India Company ship Grosvenor sailing from India to England was wrecked on the coast of South Africa when sailing around the Cape of Good Hope. As in the case of the Batavia, the ship was wildly off course (though more because of incompetence than deliberate malice) and likewise when indications of land were sighted, they were dismissed with disastrous consequences. The captain John Coxon and the helmsman that night did not have a marine chronometer and they estimated they were about 300 miles from land. When the lookout William Habberley said he saw strange lights in the distance that kept disappearing, these were dismissed by the officer of the watch Thomas Lewis as "something akin to the Northern Lights," although in reality they were land fires lit by natives that kept being obscured behind hills as the ship moved.

When the watch was changed at 4:00 AM, Lewis warned the new officer of the watch Thomas Beale about what had been seen, but Beale disregarded it. Evidently both Lewis and Habberley were beginning to be worried and they both stayed up with the new watch to keep a lookout until suddenly Lewis thought he clearly spotted land. The author Stephen Taylor writes:

Soon afterwards Lewis, who had stayed with the watch and was now at the top of the foremast, came swinging down the ratlines in a rush. He told his shipmates that he had seen a great mass of land on the starboard side. They peered into the murk, across the foaming crests. Some wanted to see the looming blackness as merely another squall rolling in from the north-west; but Habberley, who had also remained up, was no longer in any doubt, and he and Lewis together persuaded a quartermaster, William Mixon, of their peril.

Beale persisted in completely dismissing Lewis and Habberley in addition to William Mixon as ignorant and paranoid, but Mixon was convinced and defied Beale to wake up the captain. They then tried to turn the ship's course but it was too late and they hit a reef at 4:30 AM. Of the ~150 people onboard the Grosvenor when it wrecked, only about 25 drowned but almost all of the rest ended up dying of starvation or being killed or adopted by natives as they tried to walk 400 miles to the nearest European settlement, with only four confirmed to have made it back. There is an interesting book written about this by Stephen Taylor called Caliban's Shore: The Wreck of the Grosvenor and the Strange Fate of Her Survivors.

These were just two examples but I would say the majority of shipwrecks I know about that were caused by the ship hitting something occurred at night due to circumstances like this. Another more modern example I can think of but which I know less about is the Titanic in 1912 which seems to have collided with the iceberg mainly because there was no moon that night making it very dark and the sea was very calm resulting in no large breakers hitting the iceberg that could be spotted soon enough.

Sources:

Batavia's Graveyard: The True Story of the Mad Heretic Who Led History's Bloodiest Mutiny by Mike Dash

Caliban's Shore: The Wreck of the Grosvenor and the Strange Fate of Her Survivors by Stephen Taylor

Voyage To Disaster by Henrietta Drake-Brockman

The Buccaneer's Realm by Benerson Little