Hi Stephen,
We had a similar issue a while ago (luckily not with 50.000 contacts, but still...). At the time I asked CD if my suspicion was correct (http://support.clickdimensions.com/entries/67477610-Does-ClickDimensions-send-to-Contacts-that-I-m-not-allowed-to-see-but-that-are-part-of-a-dynamic-Mar) and they confirmed it. Seems it is based on the Created By field of the Email Send record. In my opinion, it should be based on the Owner of the Email Send record.
Best
Joris