I figured it out, always as soon as I post a question.

Instead of storing all items on stage I just store the two items being swapped in and out which means only two items are ever on the Main sprite :d