I think I found out what was wrong. I was trying to install RECOVERY.PBP (which was already in caps when I checked) after loading the XMB by pressing select and going into the recovery menu.
Instead I pressed and held R after loading the Ape Escape savegame and before the screen flashed green, and it went straight to the recovery menu. Then I selected "install RECOVERY.PBP" and it worked.
edit: tried it with PicoDrive and when I try to run it I get a Blue Screen of Death... not sure if I'm using the wrong version of PicoDrive? I got it from this link
https://mega.co.nz/#F!NlYCQITY!MU9WnA7_V98tnJC8zUxWRA