Unfortunately Mental Ray is much more limited in the number of render instances that it can handle, compared with VRay. For our tests, the limit would be around 500.000 items. Above this value, the render times grows exponentially and (depending of the Max version), it would be unstable:
http://www.itoosoft.com/forestpack/reference/rendering.phpAnyway, with 500.000 items you can create huge extensions of grass, and Mental Ray has been greatly improved in the latest Max releases. So, the problem would be other. In any case, you can see the number of estimated render items at General->Forest Stats.
What version of Max and Forest do you use ? If possible, please send us a test scene so we can reproduce the problem here.