By default, the Library Browser inserts the trees in the scene as XRefs, so they should not take space.
Please, check in the library settings that it is working in XRef mode (see the screenshot). You also would test it in a empty scene (i just checked it now, and the Max file size remains low).
About the VRayProxies, they could be used as well, but really there is not great benefits over XRefs. Unfortunately we can't offer it as an option for HQ Plants, since 3D Mentor doesn't have their libraries in this format.