Should I drop this Physics course?

I’m currently taking a Physics course called “Spacetime, Quanta and Cosmology”. Now, I like science – or rather I like learning about the discoveries science makes. However, I have rapidly learned that the actual doing of science, or at least physics, is much less enjoyable.

The class so far has involved reading a very dense textbook in an attempt to understand relativity, followed by rather mind-bending math. Occasionally, I’ll get a flash of understanding (when I got my head around the idea that there’s no difference between saying that a boat moves past the water and that the water moves past a boat, it blew my mind a little).

I don’t want to give up on something simply because it’s difficult, I’m just wondering whether the 10% of the class that’s interesting is worth the 90% that’s not.