Would it make me look bad in court not letting my ex husband speak to our 4 year old on the phone

I am surprised to learn your  divorce does not mention phone calls. Parenting Plans should all include the following language:   Under T.C.A. § 36-6-101 of Tennessee law, both parents are entitled to the following rights: (1)  The right to unimpeded telephone conversations with the child at least twice a week at reasonable times and for reasonable durations. The parent exercising parenting time shall furnish the other parent with a telephone number where the child may be reached at the days and time specified in a parenting plan or other court order or, where days and times are not specified, at reasonable times; It is literally the first right listed in a long list of parental rights in TN.   If you are not allowing at least two unimpeded calls a week, then you are likely violating the other parent's rights, which is never a good look in Court. I would say that by itself, that is not grounds for him to get more parenting time, but if he can convince the court it is a part of an overall attitute on your part to minimize his involvement, that will be a strong factor in a modification case. Depending on your county, and many other factors, it may be the Court will lean towards increasing his time to a 50/50 schedule, or something with at least 120 days, if he does not have that already.  You h=ay very well have a good defense to his action, of course.