"Why would they? Surely you must have done something to bring on an attack."

It doesn't make sense otherwise. Perhaps the elves simply don't realise when they've done something wrong, something that might make humans angry. It's not as though the Dalish spend a huge deal of time among them.