I feel like Poland did nothing to prevent Germany's rearmement and even helped them (like during the partition of Czechoslovakia). Were they even aware of the threat that Hitler could be? What did they do to prevent german remilitarization before war broke out?
Yes, there was efforts by Józef Piłsudski (the de facto leader of Poland until his death in 1935) to end Hitler's reign. Concerned about Nazi designs on Danzig, which was an independent "Free City" but closely tied to Poland and with a overwhelming ethnic German majority, he proposed to France an invasion of Germany months after Hitler came to power. However the French declined the offer, and concerned for the short-term security of Poland, Piłsudski signed a non-aggression pact with Germany in 1934.