A swarm of large asteroids likely lurking around Venus could one day pose an "invisible threat" to Earth if left unchecked, astronomers have warned.