Men's Style: Is linen clothing wearable, or should I go back to Cuba?

I have a few articles of linen clothing: a pair of beige Express pants, an off-white shirt, and a canary yellow shirt. They’re loud, and I wore them well. Should I even keep them to wear to the beach, or will I look like a tourist/grandpa Tommy Bahama?