Exclude Object areas dynamically generate a map that is used to include/exclude geometry. It could be with a large area like this that the resolution of this map is too low and needs to be increased to improve accuracy at the edges. To do this use the Resolution
spinner found in the Areas rollout. You can also increase the Scale
of the exclude are here which might also help to remove those trees.
I hope that helps, please let me know if you have any further questions.