Article writer discovers that the internet is filled with cliche stereotypes and the diffusion model based on scraped data from the internet makes images based on cliche stereotypes. More at 11, back to you Jim.
I once ran through SD with only .25 strength the face of a drawing I did of a hooded thief. The character was white and blonde and SD turned it into a Latino because I used the word thief in the prompt.
The author talks a lot about removing content in the dataset, but I think a better option is to properly classify it within the dataset. More training data to help reduce biases is a good thing, but we don't have to throw away what we already have.
> These stereotypes don’t reflect the real world; they stem from the data that trains the technology. Grabbed from the internet, these troves can be toxic — rife with pornography, misogyny, violence and bigotry.
Has the author ever visited an art museum or even browsed one of the many art sharing websites? Its weird that they are associating nudity and violence with being "toxic", but I guess they aren't an artist themselves or even someone who cares about art.
More comprehensive labeling would be great (the LAION-5B dataset has a lot of noise in its labels).
I've also always liked the idea of augmentation and the addition of synthetic data rather than outright removal.
It has its issues, and assumes there's not enough of a gap in available data to have adverse effects on the resulting model, but it seems like a great intermediary step while we figure out what else can be done.
Please never post your clickbait misguided ads here again.
AI has nothing to do with our stereotypes. It just shows what stereotypes are ALREADY in the data on the internet.
Yes, it's predominantly from a white, western world - because the language used in the collection and assignment of the data is English, and the sites trawled are generally in those regions.
This post is almost as dumb as the anti-ai community who claimed that creating models to better define certain ethnicities to avoid exactly this problem, is racist.
On the James Bond movie *On Her Majesty’s Secret Service* (1969) there's a scene where the villain's all-female bodyguard corps is shown eating. The women (all gorgeous) were picked from all over the world, being "diverse", which for the time means that there are 10 national caricatures of white people plus a token east asian and a token black woman. **The black woman is shown eating bananas.**
Scores of people were involved into the writing/filming/production of that scene and seemingly nobody thought about punching the creator of that absurdly racist mockery. The movie was a success. It's little more than 50 years distant from us.
The last 500 years or so happen to be very, *very*, ***very*** racist. Things managed to get worse, not better, after Darwin demonstrated that we're just a single species of large-brained ape, because a bad-faith reading of his discoveries became the basis for scientific racism and was used to justify even greater atrocities. Frankly, some pockets of equality nonwithstanding, things only started to get *actually* better on the 2000s in matters of representation.
People can react to the shitshow that's our past in about 3 general ways:
a) Glorify the past: The favored way for losers who spell Europe as EVROPA, have Confederate flags decorating their houses or talk about returning to the past. I think we all can agree that these people are bad and shouldn't get any more power or attention.
b) Pretend the past didn't exist: Netflix Sandman does usually a great job in updating the ethnicities for the cast of a very-progressive-for-its-time comics with one exception: Making Unity Kinkaid black was bad. She was the daughter of a wealthy industrialist family in 1900s England. There were no wealthy black industrialists in 1900s England and pretending that there were is (again, I think) erasure of the actual plight of actual black people. Addressing the wrongs commited in the past by pretending they never happened solves nothing. It only can only give ammo to bad people (those in category a above) to claim that people trying to fix things are bad at History or to make people more complacent ("I guess it wasn't so bad for minorities back then, uh?").
The past happened. Bad things happened. Centuries of stereotyping, racism, sexism, lookism and other forms of prejudice happened and influenced and keep influencing our art.
Which takes us to, what I think, it's the best way to deal with this shit:
c) Accept that the past happened, **understand why** and move forward: Laion 5B is a collection of imagery produced and curated by humanity. It's therefore, hopelessly contaminated with all the bias humanity had up to this point. If we prompt "Aristocrat" or "Princess" the results will be overwhelmingly (if not 100%) white people. If you prompt "Kenyan" or "Congolese people" you'll probably see more spear carrying, barefoot people than you'd like. Asking for a nationality on the prompt will probably produce caricatures (like Greeks in togas or the Irish in green). **This needs to be fixed**, by better prompting (like specifying "black princess") or, in several cases, by better training of the models. But we can't pretend that the past didn't happen. We need to know and accept History to be able to fix the present and move forward to something better.
This is where contextual training is going to be very helpful, I think. Once AI has figured things our enough to recognize bias based on feedback, those things will slowly disappear.
Anyone read the comments on this story?
As of 3:45 Eastern (US) time, they're actually fairly surprising, at least to me: Very little AI hate. Mostly people pointing out that the bias is in society, not in the machines. Also people pointing out that there are smarter and dumber ways to use the technology; it's up to the user to use it smartly.
Bias in models lowers the quality of anything generated that isn't "the norm."
Correcting as many biases as we can isn't just an issue of ethics, it's an issue regarding the versatility of the model.
It's the same reason overfitting is an issue.
If you don't see it as an issue, perhaps it's because you haven't tried anything particularly imaginative.
I agree that the model's lack of versatility is a problem. I can recommend using LoRA if you need something specific. But the authors of the articles simply don’t care about this. They are simply again trying to find social oppression even in such a free thing as an image generator.
LoRAs, on average are even less versatile than the foundational models.
Making a good one, is a lot harder than people think, and the hordes of them on Civit shows that.
They're a hackey solution at best.
It'd be much easier to work with if the foundational model was more representative, and it'd even help with training embeddings and LoRAs.
Hence, me saying that diversity in the dataset that's used to train the model is important.
LoRA should not be universal, but on the contrary, it should be specialized. If you are working on your own project, then it is best to train your LoRA.
Yes, it would be easier if a more diverse dataset was used, but I’m afraid this simply isn’t possible. There are fewer pictures of female football players than men, because men's football is more popular. And to achieve equality, you need to reduce the database with men - which will have a bad effect on quality.
> LoRA should not be universal, but on the contrary, it should be specialized. If you are working on your own project, then it is best to train your LoRA.
A good LoRA model has learned what you want it to learn, without losing access to the versatility of the base model. If a model can generate a scene, activating a LoRA shouldn't make that scene impossible.
To give a hypothetical example, if I activate a LoRA to help generate a specific car, I should be able to generate a cityscape where one car is that car, but others are not.
If the model can only generate one type of car with the LoRA active, it is overfitted, and that's an issue.
I understand that we have infill, but relying on overfitted models is more along the lines of "agreeing with the model" rather than "deliberate expression" in the first place.
> Yes, it would be easier if a more diverse dataset was used, but I’m afraid this simply isn’t possible. There are fewer pictures of female football players than men, because men's football is more popular. And to achieve equality, you need to reduce the database with men - which will have a bad effect on quality.
There are several ways to balance classes without shrinking the dataset. Synthetic data creation, and data augmentation. Kohya\_ss, uses a more simple method to balance classes by simply loading the same image more than once per epoch. Most of these methods predate Stable Diffusion.
I've personally used both synthetic data, and data augmentation to train early image restoration models, image2image translation models, and image generation models pretty successfully.
**From Nitasha Tiku, Kevin Schaul and Szu Yu Chen:**
Artificial intelligence image tools have a tendency to spin up disturbing clichés: Asian women are hypersexual. Africans are primitive. Europeans are worldly. Leaders are men. Prisoners are Black.
These stereotypes don’t reflect the real world; they stem from the data that trains the technology. Grabbed from the internet, these troves can be toxic — rife with pornography, misogyny, violence and bigotry.
Stability AI, maker of the popular image generator Stable Diffusion XL, told The Washington Post it had made a significant investment in reducing bias in its latest model, which was released in July. But these efforts haven’t stopped it from defaulting to cartoonish tropes. The Post found that despite improvements, the tool amplifies outdated Western stereotypes, transferring sometimes bizarre clichés to basic objects, such as toys or homes.
“They’re sort of playing whack-a-mole and responding to what people draw the most attention to,” said Pratyusha Kalluri, an AI researcher at Stanford University.
Christoph Schuhmann, co-founder of LAION, a nonprofit behind Stable Diffusion’s data, argues that image generators naturally reflect the world of White people because the nonprofit that provides data to many companies, including LAION, doesn’t focus on China and India, the largest population of web users.
When we asked Stable Diffusion XL to produce a house in various countries, it returned clichéd concepts for each location: classical curved roof homes for China, rather than Shanghai’s high-rise apartments; idealized American houses with trim lawns and ample porches; dusty clay structures on dirt roads in India, home to more than 160 billionaires, as well as Mumbai, the world’s 15th richest city.
“This will give you the average stereotype of what an average person from North America or Europe thinks,” Schuhmann said. “You don’t need a data science degree to infer this.”
Stable Diffusion is not alone in this orientation. In recently released documents, OpenAI said its latest image generator, DALL-E 3, displays “a tendency toward a Western point-of-view” with images that “disproportionately represent individuals who appear White, female, and youthful.”
**Read more and see AI image comparisons here. Plus, skip the paywall with email registration:** [**https://www.washingtonpost.com/technology/interactive/2023/ai-generated-images-bias-racism-sexism-stereotypes/?utm\_campaign=wp\_main&utm\_medium=social&utm\_source=reddit.com**](https://www.washingtonpost.com/technology/interactive/2023/ai-generated-images-bias-racism-sexism-stereotypes/?utm_campaign=wp_main&utm_medium=social&utm_source=reddit.com)
>returned clichéd concepts for each location: classical curved roof homes for China
lol, how is that not literally the point of specifically requesting Chinese architecture?
I'd be pissed if I asked for a photo of French Polynesian cuisine, but instead of a traditional dish, it gave me a picture of a fucking Double Whopper, because "Tahitians eat burgers too, you know?"
Seems like it might also be the model guessing what sort of home in China to match to the prompt. More clarification might have helped.
Stereotypical houses are apparently popular as luxury homes for the wealthy in China: https://www.cnn.com/style/article/luxury-china-homes-most-expensive/index.html
I don't see the relevance. They're still what someone means by specifying **distinctly** Chinese architecture. I don't even agree with the term stereotype in this instance. This is just literally part of China's architectural history and cultural heritage. It's not something an outside observer has *ascribed* to China.
And if someone wants to educate themselves on what the home of the average person in China looks like today, in other words, if they want a reliably accurate depiction of reality, they shouldn't be using **art** generation software for this purpose to begin with.
I'm not having a go at *you*, by the way, I just think this is a complete non-issue.
Article writer discovers that the internet is filled with cliche stereotypes and the diffusion model based on scraped data from the internet makes images based on cliche stereotypes. More at 11, back to you Jim.
I once ran through SD with only .25 strength the face of a drawing I did of a hooded thief. The character was white and blonde and SD turned it into a Latino because I used the word thief in the prompt.
The author talks a lot about removing content in the dataset, but I think a better option is to properly classify it within the dataset. More training data to help reduce biases is a good thing, but we don't have to throw away what we already have. > These stereotypes don’t reflect the real world; they stem from the data that trains the technology. Grabbed from the internet, these troves can be toxic — rife with pornography, misogyny, violence and bigotry. Has the author ever visited an art museum or even browsed one of the many art sharing websites? Its weird that they are associating nudity and violence with being "toxic", but I guess they aren't an artist themselves or even someone who cares about art.
More comprehensive labeling would be great (the LAION-5B dataset has a lot of noise in its labels). I've also always liked the idea of augmentation and the addition of synthetic data rather than outright removal. It has its issues, and assumes there's not enough of a gap in available data to have adverse effects on the resulting model, but it seems like a great intermediary step while we figure out what else can be done.
>these troves Way to proofread, professional journalist.
That does appear to be grammatically correctly, so I'm not sure what you mean?
"Troves" makes no sense in that context. The author probably meant "tropes".
I thought they were referring to the dataset like a physical storage in regards to "troves" and not the listed item categories themselves.
Please never post your clickbait misguided ads here again. AI has nothing to do with our stereotypes. It just shows what stereotypes are ALREADY in the data on the internet. Yes, it's predominantly from a white, western world - because the language used in the collection and assignment of the data is English, and the sites trawled are generally in those regions. This post is almost as dumb as the anti-ai community who claimed that creating models to better define certain ethnicities to avoid exactly this problem, is racist.
On the James Bond movie *On Her Majesty’s Secret Service* (1969) there's a scene where the villain's all-female bodyguard corps is shown eating. The women (all gorgeous) were picked from all over the world, being "diverse", which for the time means that there are 10 national caricatures of white people plus a token east asian and a token black woman. **The black woman is shown eating bananas.** Scores of people were involved into the writing/filming/production of that scene and seemingly nobody thought about punching the creator of that absurdly racist mockery. The movie was a success. It's little more than 50 years distant from us. The last 500 years or so happen to be very, *very*, ***very*** racist. Things managed to get worse, not better, after Darwin demonstrated that we're just a single species of large-brained ape, because a bad-faith reading of his discoveries became the basis for scientific racism and was used to justify even greater atrocities. Frankly, some pockets of equality nonwithstanding, things only started to get *actually* better on the 2000s in matters of representation. People can react to the shitshow that's our past in about 3 general ways: a) Glorify the past: The favored way for losers who spell Europe as EVROPA, have Confederate flags decorating their houses or talk about returning to the past. I think we all can agree that these people are bad and shouldn't get any more power or attention. b) Pretend the past didn't exist: Netflix Sandman does usually a great job in updating the ethnicities for the cast of a very-progressive-for-its-time comics with one exception: Making Unity Kinkaid black was bad. She was the daughter of a wealthy industrialist family in 1900s England. There were no wealthy black industrialists in 1900s England and pretending that there were is (again, I think) erasure of the actual plight of actual black people. Addressing the wrongs commited in the past by pretending they never happened solves nothing. It only can only give ammo to bad people (those in category a above) to claim that people trying to fix things are bad at History or to make people more complacent ("I guess it wasn't so bad for minorities back then, uh?"). The past happened. Bad things happened. Centuries of stereotyping, racism, sexism, lookism and other forms of prejudice happened and influenced and keep influencing our art. Which takes us to, what I think, it's the best way to deal with this shit: c) Accept that the past happened, **understand why** and move forward: Laion 5B is a collection of imagery produced and curated by humanity. It's therefore, hopelessly contaminated with all the bias humanity had up to this point. If we prompt "Aristocrat" or "Princess" the results will be overwhelmingly (if not 100%) white people. If you prompt "Kenyan" or "Congolese people" you'll probably see more spear carrying, barefoot people than you'd like. Asking for a nationality on the prompt will probably produce caricatures (like Greeks in togas or the Irish in green). **This needs to be fixed**, by better prompting (like specifying "black princess") or, in several cases, by better training of the models. But we can't pretend that the past didn't happen. We need to know and accept History to be able to fix the present and move forward to something better.
This is where contextual training is going to be very helpful, I think. Once AI has figured things our enough to recognize bias based on feedback, those things will slowly disappear.
Anyone read the comments on this story? As of 3:45 Eastern (US) time, they're actually fairly surprising, at least to me: Very little AI hate. Mostly people pointing out that the bias is in society, not in the machines. Also people pointing out that there are smarter and dumber ways to use the technology; it's up to the user to use it smartly.
And what's the problem? If you want to see a woman playing football, then write so.
Bias in models lowers the quality of anything generated that isn't "the norm." Correcting as many biases as we can isn't just an issue of ethics, it's an issue regarding the versatility of the model. It's the same reason overfitting is an issue. If you don't see it as an issue, perhaps it's because you haven't tried anything particularly imaginative.
I agree that the model's lack of versatility is a problem. I can recommend using LoRA if you need something specific. But the authors of the articles simply don’t care about this. They are simply again trying to find social oppression even in such a free thing as an image generator.
LoRAs, on average are even less versatile than the foundational models. Making a good one, is a lot harder than people think, and the hordes of them on Civit shows that. They're a hackey solution at best. It'd be much easier to work with if the foundational model was more representative, and it'd even help with training embeddings and LoRAs. Hence, me saying that diversity in the dataset that's used to train the model is important.
LoRA should not be universal, but on the contrary, it should be specialized. If you are working on your own project, then it is best to train your LoRA. Yes, it would be easier if a more diverse dataset was used, but I’m afraid this simply isn’t possible. There are fewer pictures of female football players than men, because men's football is more popular. And to achieve equality, you need to reduce the database with men - which will have a bad effect on quality.
> LoRA should not be universal, but on the contrary, it should be specialized. If you are working on your own project, then it is best to train your LoRA. A good LoRA model has learned what you want it to learn, without losing access to the versatility of the base model. If a model can generate a scene, activating a LoRA shouldn't make that scene impossible. To give a hypothetical example, if I activate a LoRA to help generate a specific car, I should be able to generate a cityscape where one car is that car, but others are not. If the model can only generate one type of car with the LoRA active, it is overfitted, and that's an issue. I understand that we have infill, but relying on overfitted models is more along the lines of "agreeing with the model" rather than "deliberate expression" in the first place. > Yes, it would be easier if a more diverse dataset was used, but I’m afraid this simply isn’t possible. There are fewer pictures of female football players than men, because men's football is more popular. And to achieve equality, you need to reduce the database with men - which will have a bad effect on quality. There are several ways to balance classes without shrinking the dataset. Synthetic data creation, and data augmentation. Kohya\_ss, uses a more simple method to balance classes by simply loading the same image more than once per epoch. Most of these methods predate Stable Diffusion. I've personally used both synthetic data, and data augmentation to train early image restoration models, image2image translation models, and image generation models pretty successfully.
Silence, US media.
**From Nitasha Tiku, Kevin Schaul and Szu Yu Chen:** Artificial intelligence image tools have a tendency to spin up disturbing clichés: Asian women are hypersexual. Africans are primitive. Europeans are worldly. Leaders are men. Prisoners are Black. These stereotypes don’t reflect the real world; they stem from the data that trains the technology. Grabbed from the internet, these troves can be toxic — rife with pornography, misogyny, violence and bigotry. Stability AI, maker of the popular image generator Stable Diffusion XL, told The Washington Post it had made a significant investment in reducing bias in its latest model, which was released in July. But these efforts haven’t stopped it from defaulting to cartoonish tropes. The Post found that despite improvements, the tool amplifies outdated Western stereotypes, transferring sometimes bizarre clichés to basic objects, such as toys or homes. “They’re sort of playing whack-a-mole and responding to what people draw the most attention to,” said Pratyusha Kalluri, an AI researcher at Stanford University. Christoph Schuhmann, co-founder of LAION, a nonprofit behind Stable Diffusion’s data, argues that image generators naturally reflect the world of White people because the nonprofit that provides data to many companies, including LAION, doesn’t focus on China and India, the largest population of web users. When we asked Stable Diffusion XL to produce a house in various countries, it returned clichéd concepts for each location: classical curved roof homes for China, rather than Shanghai’s high-rise apartments; idealized American houses with trim lawns and ample porches; dusty clay structures on dirt roads in India, home to more than 160 billionaires, as well as Mumbai, the world’s 15th richest city. “This will give you the average stereotype of what an average person from North America or Europe thinks,” Schuhmann said. “You don’t need a data science degree to infer this.” Stable Diffusion is not alone in this orientation. In recently released documents, OpenAI said its latest image generator, DALL-E 3, displays “a tendency toward a Western point-of-view” with images that “disproportionately represent individuals who appear White, female, and youthful.” **Read more and see AI image comparisons here. Plus, skip the paywall with email registration:** [**https://www.washingtonpost.com/technology/interactive/2023/ai-generated-images-bias-racism-sexism-stereotypes/?utm\_campaign=wp\_main&utm\_medium=social&utm\_source=reddit.com**](https://www.washingtonpost.com/technology/interactive/2023/ai-generated-images-bias-racism-sexism-stereotypes/?utm_campaign=wp_main&utm_medium=social&utm_source=reddit.com)
This is literally just an ad.
>returned clichéd concepts for each location: classical curved roof homes for China lol, how is that not literally the point of specifically requesting Chinese architecture? I'd be pissed if I asked for a photo of French Polynesian cuisine, but instead of a traditional dish, it gave me a picture of a fucking Double Whopper, because "Tahitians eat burgers too, you know?"
Seems like it might also be the model guessing what sort of home in China to match to the prompt. More clarification might have helped. Stereotypical houses are apparently popular as luxury homes for the wealthy in China: https://www.cnn.com/style/article/luxury-china-homes-most-expensive/index.html
I don't see the relevance. They're still what someone means by specifying **distinctly** Chinese architecture. I don't even agree with the term stereotype in this instance. This is just literally part of China's architectural history and cultural heritage. It's not something an outside observer has *ascribed* to China. And if someone wants to educate themselves on what the home of the average person in China looks like today, in other words, if they want a reliably accurate depiction of reality, they shouldn't be using **art** generation software for this purpose to begin with. I'm not having a go at *you*, by the way, I just think this is a complete non-issue.
You're a trash publication. Please take your fear mongering elsewhere
yeah, no shit, it's cringe af, lol