I've never ran into that error, but then again I don't have a ReplayTV (But would love to have one!) The sync problem may have something to do with the codec Replay uses verses the VdubMod codecs. Have you checked any of the Replay forums?
A few links from my files.
Here is the link to the Doom9 Forum and there is a section on Vdub and VdubMod.
This link is to various articles @ Afterdawn.
Nicky's Guide has a page on mpeg2 to avi.
I am very interested in your problem(s) dealing with the ReplayTV because I have been thinking seriously on purchasing a unit. Keep us informed of your progress.
hth,
Jeff