Are you using high-poly trees ? How many models and what size ?
The problem is that Max 2009 was the first version including BSP2 in Mental Ray, and it was very unstable. Further, the 32 bit version is very limited handling large tree models.
I suggest that you convert your models to mrProxies, that would help to save memory. But, if possible, upgrade to Max 2009 64 bit, even using less than 4Gb. RAM the Max performance should be better and more stable.