Jeffrey Hammerstein, a district chief at Wake County EMS, said three people were confirmed dead in the accident at the glass and steel building called Charter Square, where scaffolding had peeled away from the exterior and fallen into a parking lot below.
Authorities said they were working to identify those who died. The fourth worker was reported to have suffered serious injuries.
All were involved in the construction project, Hammerstein said.
Peter Thurston was working inside the building installing a security card reader system when the accident happened.
He said he ran outside to try to help and saw three men in safety harnesses, leading him to believe that they had been attached to the scaffolding.
"It was just a loud crash and a huge cloud of smoke," said Thurston, 32, of Garner. "I noticed three of the guys and it looked like they were dead. ... That could have been anybody. That could have been me."
He said a fourth man wearing a safety harness was found inside a nearby portable toilet. He was still breathing and had a pulse, but was barely responsive, Thurston said.
It was not immediately clear if the injured man had fallen into the toilet or had been sitting inside when the collapse occurred.
The scaffolding that fell was attached to the side of the new building that allowed it to travel up and down. One of the tracks had snapped off several stories up and fallen into a twisted heap on the ground below.
State Department of Labor spokesman Neal O'Briant said Monday that his agency is investigating the cause. Officials had closed off a wide area around the scene.
A group of men in hard hats and yellow vests, some of which said Associated Scaffolding on them, were talking to an official near the edge of the police line. A woman who answered the phone at the company's Durham headquarters declined to give her name and said the company had no comment.
According to the state Department of Labor, the construction industry accounts for nearly half of all workplace fatalities in the state, with 19 deaths at construction sites reported in 2014.
