Hi,
Enabling the Respect System Units in Files option in the System Unit Setup window & marking the Rescale the XRef Objects to the System Unit Scale checkbox during the Create XRef Record from File process should do the job (when using any geometry from a scene with a different Scene Unit Setup, it will rescale the geometry accordingly). I hope that helps.
Best regards,