I just purchased a Windows VPS here at HMS to use as a SmaterMail email server for websites I am hosting on a dedicated server at another provider. I have added several domains to SmarterMail on my VPS, but I do not understand what to use for the MX Record on the DNS server?

Just point your MX Record to whatever server is serving your mail

example: mail.domain.com

and Use Priority: 10

Actually, I just got my answer from support. The step I missed was setting up an 'a' record for mail.mydomain.com using the IP address of my VPS.

Thanks for your help though.

OK, I have another question about SmarterMail settings. In General Settings > Server Info, there is a warning message that I must add a valid Host Name (i.e., mail.mydomain.com). Since I will be hosting multiple sites and there is no primary domain assigned to the VPS, does it matter which domain I use?

Doesnt matter but reverse DNS should be set up for what ever domain you choose.

While most email servers typically do not check that the domain actually resolves I know DNS stuff shows an error when the A record for the domain doesn't match that of IP you're looking up.

But if you plan on sending mail to the likes of AOL, Yahoo!, etc. you will definitely have to have support setup Reverse DNS for your VPS IP address.

If you're a reseller it's probably the best to use your professional domain for the hostname ie: mail.domain.com and point that to the primary IP address of your mail server.

You should setup the RDNS entry to resolve to that hostname because there are some finiky MTAs out there that check.

Check RDNS here, http://neranjara.org/rip.php !