Not easily. He didn't get to be an admiral by being weak. Even your grandfather is only a vice admiral -- though rank isn't really that indicative of strength.

[Totally putting up with it. Luffy's kind of not so bad sometimes. If he ever has a son, he wouldn't mind a non-pirate version of Luffy.]