can my ex husband be taken legal custody away /

The story is a bit convoluted -- if I understand the history: For 13 years, you and the dad have not lived together. During that time, he has had physical custody, on paper, and the child has mostly lived with him, but for months or years at a time, you and he have divided custody, in one way or another, or the child has lived mostly or entirely with you. On your questions: If you cannot continue making informal agreements satisfactory to both of you, you can file a motion with family court to have a custody schedule established.  See: http://willicklawgroup.com/child-custody-and-visitation/ And while the question is a bit vague, I don't see who your ex could get "in trouble" WITH for leaving your child in your care, with your consent.  Finally, yes, under the Rivero case, which is posted at the same place, if you go to court anytime soon, the court will do a "one year lookback" to see who has really HAD custody of the child, and the person with "de facto" custody would presumably have an advantage in any further court proceedings, since the parties "voted with their feet" as to where the child should live.