Laundry Service and Papa John's didn't immediately respond to CNBC's requests for comment.
Schnatter resigned his post last week at the behest of Papa John's board after confirming the Forbes' report. However, he said the comments were taken out of context and that he was provoked into using the N-word after Laundry Service executives on the call suggested the pizza chain bring on performer Kanye West as a co-spokesman for television spots and promotions.
Schnatter said he refused to work with West because "he uses the 'N' word in his lyrics," according to the letter. It was later on the call that Schnatter said he used the actual word when pressed whether or not he was a racist.
"I then said something on the order of, Colonel Sanders used the word 'N,' (I actually used the word,) that I would never use that word and Papa John’s doesn’t use that word," Schnatter told the board. "Let me be very clear: I never used the 'N' word in that meeting as a racial epithet, nor would I ever."
Glaser, Schnatter's lawyer, warned the board in a separate letter sent Sunday against removing him altogether after several directors questioned whether he should give up his seat, she said. Glaser told CNBC the board doesn't have authority to remove him without shareholder approval.
That would be a tough vote to win since Schnatter owns almost 30 percent of the outstanding shares, according to FactSet. Glaser asked the board to open an investigation into the call and subsequent events.
"Those individuals were acting on rumor and innuendo, without any investigation — let alone a third-party investigation of the facts," Schnatter wrote of the board's actions.
Papa John's announced late Sunday that Schnatter was prohibited from speaking to the press, was being removed from the company's advertising materials and that a special committee of independent directors revoked his office space at its Louisville, Kentucky, headquarters.
Schnatter reiterated his side of events in the letter, stating that Laundry Service was brought in to conduct a "diversity media training" session ahead of a conference he was attending.
He said Papa John's fired Laundry Service the day after the call took place, but still owed the media agency about $1.3 million. Schnatter said the agency requested $6 million "because they claimed some of their people had been offended by what I had said," he wrote.
One their attorneys threatened "a smear campaign" if Papa John's didn't pay up, Schnatter said, adding that the company ultimately paid Laundry Service $2.5 million.
"I will not allow either my good name or the good name of the company I founded and love to be unfairly tainted,” Schnatter wrote.
