Should I inform farmers before I take big rocks out of the ditch by their field?

I am a rockhound….I love hunting for rocks…mainly the bigger ones..I use them in landscaping…..Rare ones, unique ones and sometimes I am hunting the ditches between the road and a farmers ground,I get people driving by looking at me funny….Sometimes I think they think I am doing something I shouldn’t.The ditch is not their property but I wonder in the areas I go to or would like to,,,Should I look up who owns the land next to it?I do go right up to the fence line but do not cross it.Should I always inform the owner of the land?