How do I approach the subject of certain family members not being allowed around my baby?

I understand your concern. The main focus is your Grandmother. If I were in your shoes I would go about your day as normal for the sake of not upsetting your Grandma. If the "Douche" tries to touch your baby just politely say no. Try not to show your Grandma that there is an issue between the two of you because this may stress her out.