In many [possibly most] UI situations, the UI hows something fairly obvious. Partner makes a slow penalty double, and it is clear he would be happy to hear it removed with an unsuitable hand - and the reverse is that a quick, happy double is one that partner would like you to leave in whatever.
In this case, the first question is: What does a slow 5

show?
It could be that he was considering defending 4

, perhaps taking it out because of fear that it might make. Alternatively it could be that he is wondering whether 5

is enough, considering a more encouraging bid.
The problem as far as deciding what to rule is that these two views work in different ways. If he was considering passing, then this suggests that 6

now is wrong: if he was considering doing more, then this suggests that 6

is right.
So I agree with the TD. South is on a guess, and the hesitation from partner does not really help [apart from the fact that he is not allowed to use it].
Suppose South had passed rather than bid 6

, and North had a hand that was borderline whether to defend 4

, so 6

does not make. Now the defence would be asking for an adjustment to 6

-1.
While I think this ruling fairly clear, others might disagree on the bridge judgement. For that reason, an appeal is not frivolous.