Currently you can’t directly do what you want.
The workaround for this is to create a copy of the objects you are subtracting with, and hide those objects / groups. Do the subtract and unhide the previously copied object.
Here’s a quick video about it: