Hi,
you have right, we need a SQL Where a bit more complex to make this, you can use something like:
user_id IN (SELECT user_id FROM #__user_usergroup_map WHERE group_id IN (1,2) GROUP BY user_id HAVING count(user_id)=1) AND group_id IN (1)
We need something like this because there is a INNER JOIN between Joomla User Table and Joomla Usergroups Table.