In case anyone was interested, I figured out an alternative solution. I never could find out what was causing my game to slow down, although I believe it was tied to the tiles and the uncertainty of garbage collection.

In the end I changed my code to recycle objects and reuse them rather than remove them and create new ones. Works like a charm this way.