Social interaction increases significantly the collective action and operation of swarming systems. However, a growing body of empirical results reveals limitations in the number of social links in natural swarms, thereby suggesting possibly detrimental effects associated with excessive social activity. We study the consequences that such excess has on the responsiveness of a swarm facing dynamic perturbations. Using a classical model of collective motion, we show that the capacity of a swarm to perform actions critical to its survival peaks for a number of social links much smaller than the maximum possible number. This relationship between social activity, responsiveness and predator avoidance capacity presents a plausible picture of how natural selection could favor organisms that self-limit their sociality, as is common in many taxa. These results have important implications for the design of swarm robotics systems.