The main difference between these 2 massages is that The Khmer massages are not as vigorous as the Thai massage. There are also some minor stretches at the end taking no more than a minute to complete. Khmer massage is most of the time laying on your back – maybe 60% of the total experience and 40% lying in your stomach.
Whereas Thai massages work the entire body using a sequence of movements that are similar to yogic stretching. Your therapist will use their palms and fingers to apply firm pressure to your body. You'll also be stretched and twisted into various positions. You can wear loose, comfortable clothing during the massage.
The 2 main differences between these 2 Massages are:

1/ the pressure strength usage
The traditional Khmer massage always asks the person whether they should put more or less pressure on their body first, whereas the Thai massage always goes with a harder pressure because of their various "technique" even if you ask the therapist to lessen their pressure, you can still feel a little bit of pain within the techniques.

2/ The Techniques
The main technique of a non-oil Khmer massage is a type of 'kneading', working deep into the muscles and leaving you feeling relaxed and sleepy afterward. Pressure points are stimulated throughout your body, and it is not uncommon to drift away into a deep sleep.
Thai massage works the entire body using a sequence of movements that are similar to yogic stretching. Your therapist will use their palms and fingers to apply firm pressure to your body. You'll also be stretched and twisted into various positions.
Is the Khmer traditional massage worth it and where to find the best place to get it in Phnom Penh?
Overall, Cambodian massage is one of the most comfortable and soothing massage styles ever. It secures each one with the experience to help your body feel the 3Rs (Revitalized, Relaxed and Recovered). There are few travelers who have tried and attested the superb experience.
So yes it is worth it, and the best place to get it in Phnom Penh is at:
AQUA SPA - Massage and Sauna
Street 478, #35
Tuol Tompoung
Phnom Penh 12310