The clean-up, dubbed Operation Restore Order, comes just days after President Robert Mugabe ordered Harare's streets to be cleared of the vendors.
But witnesses said traders who were evicted by police early on Thursday soon returned to their positions.
Parliamentary watchdog, @OpenParlyZW tweeted a picture of a council vehicle with riot police on board driving past vendors still plying their trade on central Harare's Speke Avenue.
@OpenParlyZw Riot police clearing vendors : Crime trying to survive UniteForChange pic.twitter.com/vxmrprrVlh
- Shared Change???? (@ZimForChange) October 12, 2017
It said vendors outside a supermarket in Harare Street had been ordered to vacate earlier on Thursday, but had trickled back an hour later.
There were no immediate reports of any arrests or violent clashes.
The Zimbabwe Peace Project tweeted pictures of two of the riot police's large silver water cannons parked near the city centre, indicating that the authorities are ready for trouble.
The cabinet minister in charge of Harare Metropolitan province, Miriam Chikukwa on Wednesday ordered illegal vendors and taxi operators to leave central Harare "with immediate effect", according to state media.
'Acts of barbarism'
Pressure group, the Vendors Initiative for Social and Economic Transformation (VISET) said it had been informed that similar operations were happening in towns and cities across the country, though that claim could not immediately be verified.
The group called the decision to move vendors "acts of barbarism".
It accused the Harare city council of failing to provide adequate alternative trading sites for street traders.