In case anyone else thinks of monkey patching this to work BEWARE it causes the loading of the belongs to record to get mucked up Recursive Sideloading
I really do wish I knew the correct way to accomplish this task. My current solution is to create a new record with just the ID on it and store that but it seems pretty wrong, not to mention it will probably break something else later.