Even though I don't have a book coming out (yet), I'm already doing the conference circuit because of the From the Mixed-Up Files blog and #MGlitchat. And even though I'm comfortable speaking in front of groups (and don't even mind--heck, I might actually prefer--the non-structured Q&A part), I am a lot like Nova when it really comes down to it. I worry and stress and lose sleep and can't get any work done before the presentation and I usually crash afterwards, too. It probably isn't as bad for me as it might be for the rest of the introverts here (I am right on the border of introvertdom/extrovertdom), but it's still a problem. I'm okay DURING the presentation, it's the before and after I struggle with.
And many times I have asked myself why I keep saying yes. I'm not that great of a speaker, so it's not like I'll be depriving the world of anything spectacular if I stop. There are so many people who could share what I have to say much better than I ever could.
But in the end, I decided it was better to get the practice now, when the stakes were pretty low (no books on sale, no pressure to perform for a publisher/publicist, no push to get up in front of an audience other than for purely altruistic reasons, etc), then to find myself in a situation like Betsy describes.
So I understand where you're coming from, OP. And I agree with the idea that, if this makes you too uncomfortable, don't do it. But I also think you should start looking for small ways you can speak in public. Because Betsy's right: you never know what life's going to throw at you, and it's best to be prepared. And it's always better to make those preparations on your own terms, too.