The 67-year-old explained she became "good friends" with Cosby, 83, as they worked and traveled around the world together for two years. However, one night, "he tried to kiss me,” Gifford told PEOPLE. "And the one and only time that he did try to kiss me, I said, 'No. Bill, you're my friend.' He goes, 'You're right. Good night.'
"So I found those accusations against him very painful, because that's not the man I knew. This was the most admired man in America for years," she said of the Cosby Show star. "He was called America's dad, remember?"
Gifford added: "He was not the first man during a multi-decade career that came on to me, and I'm just very grateful that I never let any of them take it any further than that."
Cosby was sentenced to three to ten years in state prison in September 2018, five months after he was convicted of drugging and sexually assaulting a former Temple University employee in his mansion in 2004. "I never saw anybody personally being treated that way by him in two years, traveling all over the country, staying in the same house with him whenever we went to Harris, or in Reno," Gifford recalled.
"We had great fun together, and I never thought a thing of it ever. He would make me cappuccinos, he'd make the girls cappuccinos, he would pour me a glass of wine on occasion, and I would read the stories about how he would put drugs in other women's drinks and it was just foreign to me," Gifford confessed.
"I didn't doubt the women's stories,” she added of the women who said Cosby sexually assaulted them. "I don't know why he behaved the way he did with the other women, that's their story to tell." Gifford details the "unusual encounter" with Cosby in her It's Never Too Late memoir. In the chapter about "how you think you know somebody," Gifford wrote she was "surprised" by the attempted kiss but "didn't feel panicked."
"I was as surprised as the rest of the world when, years later, Bill Cosby, at one time the most respected and admired man in America, was accused of all manner of despicable acts with a myriad of women," Gifford wrote.
The former TODAY co-host described her husband's transgressions, which happened about 11 years into their nearly 30-year marriage. "Nobody knows what I went through. You feel hopeless and you don’t want to live anymore," she said of the whole ordeal. However, with counseling, the duo was "able to turn it around. It was the right thing for my life to forgive my husband. And then work at forgetting."
"The person you loved the most in your life is the person that can hurt you the most," Gifford added of her unfaithful husband's actions. The couple — who had son Cody, 30, and daughter Cassidy, 27 — remained together until her husband's death at the age of 85 in 2015.
Gifford previously said she decided to write about her past sexual assault experience to let other women know they are not alone. "There's very, very few women in this industry who weren't subjected to that kind of behavior and that kind of abuse," she told ET of the time she was assaulted by an unnamed producer. While Gifford was grateful to get away unharmed, "you are so demeaned, and feel so dirty," Gifford explained. "You know what you feel more than anything? Stupid. I thought, Kathie, you idiot."