The most common and most likely answer to this crossword clue is the 4 letter word NARK
I've seen this crossword clue in The Guardian
N
A
R
K
More clues leading to the result NARK
Buddy explains!

The straight part
We know that "annoy" is the straight/definition part of the clue. The straight part is often a synonym or definition, in this case annoy = nark.The cryptic/wordplay part
In cryptic crossword there's often an wordplay/cryptic part as well.This part involves a more complex hint, such as an anagram, homophone, hidden word, etc.
Sadly, We don't have an explanation for this specific crossword clue yet.
What does NARK mean?
NARK noun- A police spy or informer.
- An unpleasant person, especially one who makes things difficult for others.
- To watch; to observe.
- To serve or behave as a spy or informer.
- To annoy or irritate.
- To complain.
- (often imperative) To stop.
- A narcotics squad police officer.
- A narcissist.
- A police spy or informer.
- An unpleasant person, especially one who makes things difficult for others.