Isn't this just called user testing? Also this is in the context of a fucking dataset. If data needs to go through DI in case something blows up on Twitter, I guess it's sad state we're in.
Does it? Seems to me data is a prime place for exclusion to occur. Example: a dataset of tagged photos for training a neural net to analyze facial expressions. All the photos are of white faces.
Maye they should run a study on diversity approved data set and see how well they match the demographics where it is being used. Then they could compare it to data sets without diversity reviews and see which one has better representation of the actually demographics. A kind of performance test for the diversity review.
If, for example, the dataset only contains white faces and is intended to train facial recognition then yes, it needs to go through some kind of DI review.
Wouldn't this review be done on the data collection and planning side, rather than at point of publishing though? Surely you can publish datasets of just white faces or just black faces if during planning that's what you intended to do for some reason?
Wouldn't it be both? With a legal review I would make sure that we take into account any legal requirements in the planning stage, then at completion I'd still want legal to look at it and make sure those requirements were met. I don't see why this would be any different. Planning review: "Here's how we're going to make sure we get a suitably diverse set of faces". Pre-Publishing review: "Let's look at the data and make sure we have everything we planned on. Oops, looks like we missed New Zelanders somehow, better fix that before we publish."
I mean, maybe, but you still might need it to be reviewed. You don't have to wait until you're about to launch to start these kinds of reviews and if you know that some kind of DI review is necessary for your project you should start talking to the reviewers as early as possible, especially if you are making a potentially controversial design choice.
