Government officials and residents said fighters from al-Qaeda in the Arabian Peninsula (AQAP) were among those who seized Zinjibar, in Abyan province.
But opponents said President Ali Abdullah Saleh had given up the town to stoke fears of a militant takeover.
He is resisting calls to step down, despite months of popular protests and growing opposition from a key tribe.
At least 124 people died in recent days in the capital, Sanaa, in clashes between government forces and fighters loyal to Hashid tribal leader Sheikh Sadiq al-Ahmar.