Why do I have this reaction when I drink caffeine?

Ok so I am 25 and have been drinking coffee for probably almost 10 years now on and off but for at least the past 5 years I’ve drank it every morning and throughout the day. I have always had a low tolerance for it and almost as time goes by it seems my tolerance for caffeine becomes even lower. For example: A half a mug of coffee is usually ok with me as long as I eat something with it, and even then if I’m not constantly eating throughout the day, I will “feel” the coffee usually like 4–6 hours later, get terrible jitters, become hot and sometimes irritable, and this will generally last until I put something else considerable in my stomach. If I have a full cup of coffee, god watch out, my days are numbered. I’m semi joking but not really, I accidentally drank a little more coffee than normal a few days ago and was at yoga probably 6 hours later and thought I was going to pass out.

Now the question you ask yourself is WHY DON’T YOU JUST STOP DRINKING COFFEE? Well, I do like the boost it gives me, although I feel mine is far delayed compared to most people. Getting up and drinking coffee does not immediately make me feel awake like many people claim it does for them. Normally my energy boost from the coffee comes hours later, but before the jitters and irritability. I also still drink it because on days I have tried to forego coffee, I have ended with terrible headaches which I can only assume are from not drinking it? On any of those occasions I haven’t done anything else out of the ordinary. These headaches are bad and so I figure ok, I’ll just keep drinking coffee then, geez. Sounds pretty stupid when I write it out, but regardless that’s what I’ve done.

So, lately, it’s been kinda getting worse it seems. Today, I drank yerba mate for the first time since I heard it gives you caffeine but is healthier and wouldn’t result in all the negatives that the caffeine in coffee gives you, or whatever. (If it matters I did not use the bombilla and gourd method but just had a teabag) So, I drank it, felt normal but not caffeinated for the 2 or 3 hours after I had finished it. In fact I felt tired. Then, I began feeling really really hot. It was a hot day out to begin with but I love the heat and can usually handle 110 degrees no problem. It was probably only 90 but I had a wet face and I was just sitting in the shade, reading! I’d had plenty of water and had eaten, and then I became extremely irritable. I was sort of having trouble focusing and I was just so hot, and so irritated about everything. My boyfriend was playing the guitar in our living room and I was so annoyed by it I seriously had to leave the house.

WHAT IS GOING ON??????????????????????
I assume the answer is JUST STOP DRINKING CAFFEINE! But I wonder if there is anything more to it, anything I can do to lessen these bad effects, or if anyone else has this reaction to caffeine? Please, any insight would be really helpful! Thanks!