During the Age of Exploration how did mariners find out if their ships would be able to go up new rivers without sonar or modern devices to tell the rivers depth?

The simple answer is a sounding line with a weight at the end. The weight takes it straight to the bottom, note how much rope went down and that's depth. If you want to see if a bay or river is navigable, send a boat out and measure with a sounding line to see if the area to see if the ship can pass.