Even if you agreed on everything, that doesn't mean you don't need to consult your own attorney.
Divorce 101: don't listen to the X (esp. one who doesn't have a job, unless he's independently wealthy and can afford an attorney ... in which case you consult attorneys who are willing to petition the court for an order that your wealthy to-be-ex pay your legal fees). :)
"I had agreed to joint custody and I'd get no child support."
Er, why would you agree to no child support? I'd change my tune on that. Being nice gets you nowhere. You two are free to agree to a modest sum, but you don't need to negotiate "joint custody" by giving up anything. That will be the default, and the default in reality is that the mom will get primary physical custody regardless. In any event, until judge puts pen to paper (or you two do), you're free to change your mind about whatever you like.
"I'm just nervous when it comes to my son."
I get it, but psychological weakness ... there's no room for it in this stuff, I'm afraid ... not once you get a whiff of the irrational/unreasonable from the other side. (People give into intimidation tactics from car sales people for crying out loud, so I understand why when your kid's involved, it's far worse.)
Still, I'm never clear why folks think they could lose custody of their child just because a spouse threatens to (or actually does) petition the court for such. It would take very careful planning for even the most crafty parent to arrange for sole custody of a kid even temporarily. (Any law student would be able to successfully counter what would be a (necessary) assertion that you're a monster or cannot take care of your kid. "So ... the father's been ok with this 'monster' or neglectful woman taking care of the kid despite no court order being issued as to custody only until ... she decided she wanted to move 40 min away? You're so worried about your kid's well-being that you ... never contacted CPS? Didn't take the kid and refuse to give him/her back (absent a custody/visitation order, either of you is free to take the kid and not let the other see him/her until there is an order). Yeah, judge this sounds terribly credible."
If you're in a one-party consent state (google "can we tape"), feel free to start recording every conversation from here on in without telling him. Just in case he makes threats/insinuations or you want to get him to repeat them (by asking a leading question or a leading statement, like "John, why is it that you were thinking it was okay to threaten me with seeking full custody just because ... when you know there's no basis for doing so and that you don't even WANT it?"