Can a father get custody of a child by claiming neglect when the custodial mother leaves the child with HER mother 90% of the time?

Unmarried parents have joint custody, Father pays child support to mother, and babysitting money to mother's mother, for 2 year old. Mother claims she is always at work but can't afford to pay bills or support herself. She is always begging to borrow money, never pays it back. She is not blatantly neglectful when she is WITH the child but she often does not return her mother's calls or messages, says she can't pick up the child (10 minute drive) because she has no gas, etc. Father wants sole custody, has a steady job, currently lives with HIS parents because of financial problems caused by mother over past 3 years. Child's mother lies, cheats, steals and uses drugs, but very little can be proven, so father doesn't think he would ever get custody. Also, the mother has a 12 year by a different father (who is not in the picture), who has been raised, and lives with, the mother's mother. Would the 2 year old's father be able to get custody of his child, given these circumstances?