Undisclosed fence encroachement on 2nd owner's property

Being neighborly doesn't mean you have to give a neighbor some of your property.  The short answer to your question is no, you are not bound by the agreement made by the prior owner.   An easment a right of one person (the neighbor) to use the property of another person (you) for a particular purpose without actually having ownership of the property.  An easement needs to be either identified in your deed or a separate easement agreement that is recorded with the property records in your county.  Unless the neighbor's right to put his fence on your property is recorded either of these ways, then you can rightfully demand that he remove the fence.  I would suggest that you do so in writing, although it seems overly formal.  Send it via certified mail so that you have proof.  If he refuses, then I would not recommend self-help.  Rather I would suggest that you file an action in quiet title and obtain a court order requiring him to remove the fence.     Good luck -- tough situation to have trouble with the neighbor.  I'm sure you can understand why the neighbor thinks he's in the right, given that he had permission and then spent money to build the fence.