The most common and most likely answer to this crossword clue is the 4 plus 1 plus 3 plus 2 plus 5 letter word OPEN A CAN OF WORMS
I've seen this crossword clue in The Guardian
O
P
E
N
A
C
A
N
O
F
W
O
R
M
S
More clues leading to the result OPEN A CAN OF WORMS
Buddy explains!

The straight part
We know that "Invite trouble" is the straight/definition part of the clue. The straight part is often a synonym or definition, in this case invite trouble = open a can of worms.The cryptic/wordplay part
In cryptic crossword there's often an wordplay/cryptic part as well.This part involves a more complex hint, such as an anagram, homophone, hidden word, etc.
Sadly, We don't have an explanation for this specific crossword clue yet.