Aphids came with a house plant. No ants.
Anyway ants are predatory for the most part so I would worry about ants. Aphids are the evil ones.