Before nails and other modern bonding materials, how were ancient ships, such as the Trireme, constructed in a way that they wouldn’t simply break when waves hit them?

The most common methods are sewing, lashing, mortise-and-tenon, and treenails.

In sewing, rope, cord, or similar is used to sew the planks together. Holes are drilled in the edges of the planks, and the cord passed through. Often, the hole runs from the inside of the plank to the side, so that there is no hole on the outside. This protects the cord from damage from the outside; this is very common on bottom planking, so that the sewing isn't damaged when the boat is dragged onto the beach. Side planking is often sewn with the holes running from inside to outside.

The planks can be lashed to framing timbers. This is often combined with one of the other methods for attaching the planks to each other. Sewn boats and mortise-and-tenon boats are usually built planking-first, with the planks joined to make the outer shell of the boat, and then the internal framing is fitted. With such construction, the internal framing is often quite light - the planks themselves provide most of the structural strength.

A reconstruction of a Norwegian sewn boat, with the planks sewn together and the framing lashed to the planking:

http://www.vikingskip.com/halsnoybaaten.htm

and another view showing the cleats carved on the planking for lashing to the frame: https://commons.wikimedia.org/wiki/File:Halsn%C3%B8y-brest_2012-3000px.jpg

Sewing and lashing were common in many parts of the world, and were used into modern times. Boats made in this way are relatively flexible, and will flex with the waves, which reduces the chance of breakage (but note that this does increase the chance of leaking).

A more rigid, and thus not always as suitable for heavy seas, but more leak-resistant construction is mortise-and-tenon. Basically, the planks are joined to each other by tenons fitting into mortises. Mortise-and-tenon construction can be combined with sewing to actually hold the planks together - the sewing holds them together, and the tenons stops them from moving along each other. Combined with sewing (or attaching the planks firmly to the framing), there is no need to lock the tenons in place. If the mortise-and-tenon construction is to hold the planks together, then the tenons need to be locked in place. This is traditionally done by drilling though the tenon and the plank with the mortise, and putting a wooden peg through the tenon. A diagram of locked mortise-and-tenon construction: https://cdn-images-1.medium.com/max/1600/0*N8XNcEo3dYNkAiCY.png (from https://medium.com/the-bronze-age/the-bronze-age-ancestry-of-ancient-sailing-vessels-ffa9b7effd17 which has an excellent series of photos of a replica being constructed). The tenon can be part of the edge of one plank, or it can be a separate piece, in which case, mortises are cut into both planks.

Locked mortise-and-tenon construction was probably used for triremes. In the absence of actual remains of a trireme, our best guess is that they were made in the same way as other Greek ships of the era, and locked mortise-and-tenon construction was typical in the Mediterranean at the time.

Just as the tenons are locked with wooden pegs, wooden pegs can also be used to attach planking to frames, or overlapping planks to each other. Such wooden pegs - treenails, trenails, etc. - are wooden nails. They have the advantage of rust resistance over iron nails, and cheapness over bronze nails. Metal nails, however, can be clenched (the nail driven completely through the planks/frame, and the pointy end folded over), and a treenail cannot. Other methods can be used to fix treenails in place, such as splitting the end and driving in a wedge (as used to expand the ends of hammer and axe handles to keep the heads on). Wedging treenails is shown on pages 26 and 27 of http://www.fao.org/docrep/019/i3282e/i3282e01.pdf

Treenails were probably use to attach trireme planking to the framing, with metal nails used as well.

There are other types of construction that were used, such as basketry, with the hull of the boat woven like a basket, and then waterproofed by covering it with hide or a coating of bitumen. There are also one-piece designs such as dugouts and pottery boats, where no fasteners are needed (but dugouts sometimes have sewn side planks added, to increase freeboard).