To hold more than 999 gems in your wallet, simply play minigames like the race against Rottytops; you'll continue to earn gems, though note that the game will no longer display your current gem count properly until the number of gems you possess goes back to 999 or under.
Playing Shantae on a Game Boy Advance handheld will allow you to unlock the special "GBA Enhanced" bonuses. The title screen should highlight this in the bottom left when you start up the game.
In addition to enhancing the color palette of the game, you'l also be able to purchase a "secret" from an NPC in Bandit Town for 500 gem. You should then be able to learn a new dance at the "Advanced Genies!" building which allows you to transform into a Tinkerbat. The Tinkerbat plays very differently from Shantae, like being able to charge forward, climb walls and use sabres, making for a unique experience!