Harry can't peform a torture curse because he is not a sadist at heart. He's shown consistantly that he wants justice, not revenge. He spared Peter's life when Lupin and Sirius were willing to kill him. Harry doesn't want death, just Azkaban. It's suffering enough.
Neville is the same. While Bellatrix would have taken great delight in torturing him, he would not have done the same. He wants her punished for her crimes, but he draws the line at torture and murder.
Neville and Harry are alike. Both of their parents suffered for terrible crimes (Harry, muder; Neville, insanity). Harry is a better wizard (magicallly), but Neville stands up for what is right. He'd have died fighting evil. The sorting hat knew it, and put him in Gryffindor.