Really there is not any downside, except that XRef files are needed to edit or render the scene. But likewise as using XRefs in your scenes for other purposes.
Ragarding render or viewport performance, if that is your worry, there are not differences.