One solution could be to (i) use the sutherland hodgman algorithm to clip one rectangle within the other, and then (ii) use greens theorem/the surveyors formula to compute the area. The second step is easy, but the first step requires a little bit of care to program correctly.

There might be a simpler solution, so be sure to ask around at some other places.