It's not something that can be taken away from the looks of it, within the basic options, but there are several things that you can do to make it not really have any weight.
If you click on manage site>settings>manage settings>login & registration, you can choose to manually approve new member accounts. That will assure you that no one, invited or not, gets into the site, unless you say so.
Another alternative would be to hide the link with some Css. You'd have to add the CSS to your custom modifiers by clicking on /manage site>appearance>advanced>custom modifiers> and on clear line in the modifier box, you'd put this:
#page-profiles .invite{
display: none;
}
