You probably just need to disable the "Keyboard Shortcuts" for the test window. When you test your movie, click on the "Control" tab at the top of the window and scroll down to the "Disable Keyboard Shortcuts" option and click it. Then try enter button again.
Then I guess it must be a problem with the function you are trying to activate with the Enter key or the path you are using for the function, because the code itself works (see attachment and remember to disable the shortcuts when testing it).
i have (on main timeline) a button, a dynamic textfield <instance name> - test
and a function. I have disabled keyboard shortcuts and tested. Works fine here.
What is your setup ??
// button action -
on (keyPress "<enter>"){
myfunction();
}
// frame action -
function myfunction(){
test.text = "OK";
}