File Formats: Uncompressed image formats such as TIFF and PNG will yield the best quality (compared with compressed formats such as JPG).
Resolution: 300 dpi at the given output size will yield the highest quality. For example, if you want an image to be 10 inches wide, you would need that image to be at least 3000 pixels wide.
Many photos will still reproduce well at resolutions down to 150 dpi. Images below 150 dpi will cause Motif to issue image quality warnings at checkout. Warnings can be ignored if you choose, however.
Please note zooming images requires they have enough resolution to support the enlargement. For example, take the same image discussed above and zoom it such that half the original image is now full frame. Those five inches of the original image also need 300 dpi (i.e., the original would have needed 6000 pixels of width).
RGB vs CMYK: Either mode is fine. Both are ultimately converted to CMYK by the RIPs on the printing presses (which do an excellent job).