Don't you think that there should be a "Stop following this question" tab where the "Answer this question" box is?

I scroll to the bottom reading the answers. Decide I am done with the question. Then need to to scroll back to the top so I can “Stop following” the question.

Would be great if I did not have to, scroll back to the top.
Seems simple enough.