I've been using my new Home Server as a massive network storage device whose primary purpose is to stream audio and video to my Apple TV box.
The video conversion tool I've been using is Handbrake, mostly because it has an excellent and powerful command line utility, as well as a GUI that you can use to inspect your DVDs and get the exact right command line. It's also cross-platform, so I could use the same tools and knowledge on both my Windows and OS X machines.
My goal was to be able to take video from a variety of sources, and ensure that I had great quality for playback on the Apple TV, but also support taking the video with me on the iPhone and Zune. As an added bonus, the video will play directly from as well as stream to the Xbox 360.
One of the mistakes I made was failing to realize that once I had video files, I was going to need to ensure that they were deinterlaced.
Interlacing is the process that older CRT TVs used where they would display half the picture at a time (every other line). A lot of video stored on DVDs is stored in this interlaced format. When you play it back on a DVD player, either they and/or your HDTV is designed to recognize this and deinterlace the picture when appropriate. Unfortunately, if you play back a video, this logic doesn't kick in, and the picture is distractingly jerky.
During playback, you'll see three normal frames, like so:

Followed by two interlaced frames, like so:

It's incredibly visually distracting. So now I'm going to have to go back and re-encode the video with de-interlacing enabled. I thought it might be useful to share this knowledge and show people what switches I'm using to drive my encoding, in case they want to do the same thing for themselves.
I use the following command line switches for handbrakecli.exe:
-i "InputSource"
This is the input filename or input source (you can point at the VIDEO_DS folder of a DVD to use the DVD as your source). If you're using a DVD, you should use
-o "OutputFilename.m4v"
This is the output filename. Handbrake makes MPEG-4 videos.
-t title#
-l
If your input source is a DVD, you can use "-t #" to specify the title number on the DVD to encode, or use "-l" which says "auto-select the longest title". If the source is not a DVD (i.e., it's a single video file), then this parameter is not necessary.
-e x264
This tells Handbrake to use the H.264 encoding, using the "x264" encoder. This is the slowest encoder that Handbrake has, but it produces the best quality video with the smallest file size.
-q 0.59
This tells Handbrake what quality level you want. This is the right quality level for the bit-rates supported by all my devices.
-E faac,ac3 -B 160,auto -R 48,48 -6 dpl2,auto
This instructs Handbrake to create two audio tracks for the video file. The first is a 2-channel down-mix, which is used by the iPhone/iPod and Zune. The second is to also include the original AC3 (Dolby Digital) audio track, which will be used for 5.1 surround audio by the Apple TV.
-a audio#,audio#
When a DVD has multiple audio tracks, you can use the -a parameter to specify which one you want to use. It's duplicated here twice, because you could choose a different audio track for the AAC and AC3 audio tracks; in practice, you will use the same one. If you don't specify -a, Handbrake chooses the first audio track, which is usually the right choice.
-X 720 -P
This instructs Handbrake to make an appropriately sized video, automatically figuring out if the video is widescreen or not.
-m
This tells Handbrake to make chapter markers in the video for each of the chapters in the video. Usually only useful if you're encoding a DVD.
-d slower
This is the all-important flag that says to de-interlace with the best quality available.
-x level=30:cabac=0:ref=3:mixed-refs=1:analyse=all:me=umh:no-fast-pskip=1
These are specialized instructions to the x264 encoder. Leave them as they are. :)