It's not the ants but the aphids that attract the ants. The young tips that bear the flowers will shrivel and turn black, new leaves become deformed. You can use Malathion if you can bear the smell. I just use an old toothbrush and some water, and sometimes just my bare fingers to rub off the aphids.
Plant them in the sunniest area of your garden. They can never get enough sun or so it seems. No sun, no flowers.