I moved all my *Test.cls into a subfolder called Tests\
After doing this, when I open the run tests prompt, I see all the classes that I moved to the sub folder doubled up in the list of classes to select. If I select all and run, the tests for these classes are run twice at salesforces end as well.
Clicking refresh list did not fix the problem.
If I close and reopen the IDE after moving the test classes, the double entries in the run tests prompt go away.