did you do the settings inside too ??
Settings --> System --> Power --> Power Button --> and make sure "Full Power Off" is clicked.
because if you didn't ... all the power button does is to dim the screen ...

and who knows ... not fully implemented to usage it was intended to ... ON/OFF
just a thought to cover the bases :)
see post #2 by Eclet ...
so ... after exchanged the new battery ... does it power on properly now after a
FULL POWER OFF or not ???
~~~
~~~~