I think... if I didn't need f(x,y) == f(y,x), and assuming only positive integers where x != y, I could generate a unique number for each pair by doing (x*y)+y ...which gives a hyperbolic paraboloid.
However, that is dependent on the order f(x,y) and doesn't help my situation because I need a key by which to group both ways.

I've gone ahead and solved my problem by using CONCAT(MIN(userID),',',MAX(userID)) to generate a unique identifier, but unfortunately this is rather slow.