To avoid slicing hedges I would create a pre-formed 90 degree angle as well as your regular hedge segment. You can then use this in the Corner input.To prevent bending and slicing and therefore maintain instancing, follow these steps;
1 - Turn off Bend and Slice for the segments
2 - Turn on Adaptive Mode
3 - Make sure the corner geometry's pivot is set correctly so that when it is placed on a corner vertex it aligns with the rest of the hedge.
4 - Change the corner segment's X and Y Alignment to Pivot
5 - Turn off the generator's Corner > Align to Path setting.
6- Use a conditional operator set to check if the Angle is less than 180 degrees. Rotate the corner accordingly using Transform operators for the True and False inputs.
Your final style should look like this.
I've attached a sample file is case it is of use. I'm afraid Octane are responsible for the integration of RailClone and Forest Pack into their engine so we can't give you any more information about that. It's best to request that on the Octane forums.
I hope that helps, please let me know if you have any further questions.