Overall I really liked this. It didn't feel too cliché, which is hard to do in an angry poem like this one, so I commend you for that.
The first thing I noticed about this is a lack of punctuation. People don't realize it, but it really does help. It doesn't have to be perfect, but at least put a period after complete sentences. Quite honestly, I think it feels pretty weird without it.
"Though I try to brush off your effect on me"
This struck me as very cliché, and the rest of this poem shows me that you can do much better than that. Not only is that line over-done, but it doesn't seem as sophistocated vocabularywise as the rest of the piece.
"Emotions I'd rather avoid"
I also have an issue with this line. You've already told us that these emotions are "the worst." By that point in the piece, you've actually said it twice already. It seems obvious that you'd rather avoid them. I'd change this line completely. Maybe to say something about where those emotions are when not being stirred up by "You."
Fix those lines, and I'd call this a pretty good poem.