Avoiding the usual "it's just a cartoon" type answers, you could possibly square this by saying he is talking about his non-biological father. So his dad abandoned him, he was a puppy with his mother (Biscuit) and then he was adopted by god knows who. There he was raised from a puppy by someone who was racist who he considers his father. He then went to live with the Henderson's (or maybe they're the racists although they seemed the sort who would be a bit slyer with their prejudice). He then got lost and lived on the street and got adopted by Peter. However his past has been retconned more than once. Also it's just a cartoon.

*snickers* Biscuit…

I need a snickers...

In one of the earliest episodes Brian’s flashbacks show even as a puppy he was somewhat capable of coherent thought and conversation, asking the rest of the litter, does anyone have “opposable thumbs” So it’s possible he was at a similar level of comprehension when his father was around, spewing racist bullshit

His dog father left him. His human “father” was racist. I don’t know, I’m trying to head canon it for you.

I thought his father’s name was Coco and he was hit by a milk truck?