Punch Taverns has seen one of its larger shareholders back out.