This feature it is not supported.
Full list of supported/unsupported renderer features can be found in our documentation
I'm afraid there are several limitations to implement features like Forest Color
or the Edge Boundary Checking
mode for this render engine.
The reason is that these features work at render time, and require a custom shader for each renderer (a piece of code which runs directly in the render engine). But V-Ray RT doesn't support any third-party shaders (there is no SDK, so we cannot write code for it).
Regarding OP case,
If it is really big terrain You should try to minimize the amount of objects for ForestPack to evaluate if edge mode hiding is needed.
In both cases, I would suggest you to use the work-flow shown in the Grass Edges