I got this figured out on StackOverflow, turns out the accounts I was using were all db_owner's of that DB so they had access to the key by default. Once I created a new account that wasn't a db_owner ...
Some results have been hidden because they may be inaccessible to you
Show inaccessible results