As speakers and debaters, I really like both Singer and D'Souza.  The thing is, they both push some pretty scary ideas--not only in my opinion, but in the opinions of many.

From what I can tell, Singer is considered to be pretty "out there" and on the fringe of modern ethical thought, at least by the standard of mainstream philosophical thought.  It's true that many people are turned off by his hardline stance on animal rights, but it goes deeper than that.  He questions the value of human life to such an extent that, right or wrong, it makes lots of people uncomfortable. 

As far as D'Souza goes, he's not as much of an extremist as Singer overall, but he does have some pretty troubling views on women's rights and the history of Christianity. 

But yeah.  Singer busted D'Souza pretty early with the whole Buddhism thing.