It was a dangerous cult. The early Christians would go around doing shit like smashing up the temples of other religions to destroy their "idols," a practice from which the word "iconoclasm" arises. Amazingly, they weren't terribly popular. There's a reason they threw them to lions and it wasn't because they were just poor innocent minorities being persecuted by the evil majority. They were basically complete assholes.
They probably didn't burn down Rome, as Nero accused them of doing, but were unpopular enough that people would believe it.