September 2011 Archives
One of the dreams of solving the captioning backlog is to rely on speech recognition. I do have to say that speech recognition is far more effective at time than I would have dreamed, but still my intuition has told me it's not entirely working. A fascinating article from Robert Fortner on "The Unrecognized Death of Speech Recognition", essentially backs up the intuition with some hard numbers. He notes that accuracy has not improved much since the early 2000s and that in most cases, the rate is not within human tolerance (humans apparently have about a 2% error rate and even that can lead to some pretty ridiculous arguments).
When Speech Recognition Works
Speech recognition can be effective in two situations
- Specific context (airport kiosk, limited menu commands) - even here though it should be noted that it's pretty darn easy to frustrate the average health insurance voice recognition system so that they give up.
- Specific speaker - Speech recognition is effective when trainied on a single voice, and the training time is shorter than it used to be. For captioning purposes, this means that if a single speaker makes the original audio (e.g. faculty lecture) or someone else repeats what's on the audio (the captioner), speech recognition is pretty effective.
By the way, in the recent Second Accessibility Summit, Glenda Sims noted that correcting an inaccurate transcript is more difficult than starting from scratch.
What Speech Recognition Is
To understand why speech recognitin isn't improving, you should consider the task it's trying to perform. When human ears listens to language, it hears a stream of separate words and sounds and groups those into words and sentences. The reality is that speech is a continuous sound waves with very subtle acoustic transitions for different sounds (see images below, the bottom ones are the spectograms that phoneticians use). Your ears and brain are doing a lot of processing to help you understand that that person just said.
Your brain not only breaks up sound waves, it also accounts for the acoustics of different genders, different regional accents,filtering out different types of background noise and it probably includes some "smart guessing" on what a word is as well (which doesn't always work). It's no wonder that replicating the functionailty of the mechanism is taking time.
Ingoring the Linguists
There's one factor that Robert Fortner points to - speech specialists are not always involved. As one IBM researcher claimed "Every time I fire a linguist my system improves"...but apparently there is an upper limit to this without more information. Maybe it's time to start rethinking the problem and if the programming team might need some outside experts.
In the spirit of continuing to clean my desk, my next book to review is The Wisdom of Crowds by James Surowiecki. I think a lot of people at ETS are familiar with the book, but I think it's worth explaining exactly how the wisdom is generated.
The term "Wisdom of Crowds" seems to suggest a scenario where people make decisions as committees, but that's not what it really is. Rather the "wisdom" comes from being able to tap into the results of multiple individual decisions rather than relying on a single committee or expert.
A classic example is a contest to guess the weight of an ox. Individually, the guesses varied widely, but the average of the guesses was within one pound of the actual weight. It wasn't the case that the group decided the weight of the ox, but rather that the individual guesses added up to the correct answer.
I admit that I've always been a little skeptical of "collaboration" because I often equate with group think, but this kind of collective wisdom still values individual diversity. In fact, Surowiecki argues that you get the best results specifically when you can factor in individual input.
There are a lot of interesting applications to this concept in the book, but I think one of the most important is ensuring that you really ARE getting a diversity of opinion. One reason that anonymous voting is so important is that it does insure you are getting an accurate opinion from individuals and not votes partially based on social pressure.
Another situation this applies to is getting feedback from your students. I think a lot of us have experienced the eerie silence that follows the instructor's request for an answer, not to mention the awkward nods of agreement with slightly puzzled faces. Are the students agreeing with you or just trying to mirror your opinion?
One reason I like the concept of clickers is that it does enable the kind of high volume individual input needed to assess your students' actual thinking. We talk about how it can assess misconceptions (true), but sometimes it can access a wisdom you didn't know was there.
Earlier this week, I was talking about gender stereotypes in language and asking students if they could identify some stereotypes. In more than one case though, I saw some puzzled looks. I began to realize that some of my research may be getting out of date, at least in their circles.
I'm also reminded my personal guideline of multiple tabloid sources. If one tabloid claims a movie star is an alien spy, it's probably a lie. But if two more or tabloids independently have the same story...it is probably true.
Note: This is a bit off topic, but I did want to recommend a book from the ETS diversity library as a great bit of historic writing. Hopefully I won't stir up anything too controversial.
A book review that has been pending, but is timely once again is Juan Williams' Eyes on the Prize which recounts the Civil Rights struggle from 1954-1964.
I actually inherited this book when I took over some materials from semi-forgotten a diversity project at ETS and started reading it a few years ago. It's timely now though because both ITS (my mother ship at Penn State) is trying to increase diversity awareness and because the period is on our minds again due to the recent film version of the The Help.
The period of 1954-1964 is critical since it covers Brown vs. the Board of Education through the passing of the Civil Rights Bill which enshrined key civil rights in a political sense. This was really the period when the Civil Rights question really entered into mainstream (i.e. white) consciousness and the mainstream came to understand that the Jim Crow systemized segregation system was wrong in ways which we still cannot fully comprehend unless we experienced its injustice.
Actually though, Williams book begins a few years earlier when the African-American community understood full well that they could not achieve their dreams unless Jim Crow was dismantled. It not only chronicles early legal cases, but some of the everyday injustices including a particularly brutal killing of an African American boy, Emmitt Till, whose only crime was winking inappropriately at a white woman (apparently he was from Chicago and didn't understand his cousins' warning of the Mississippi code of behavior).
It was hard to imagine things were that bad once upon a time...but it was.
The joy of this book though is that it focuses on the triumph, not just the tragedy. Williams does not skimp on the struggles, from attacks on children and assassinations to sneaky legal maneuverings, but it makes the victory all the much sweeter.
Williams also shows us the behind-the-scenes discussions in the civil rights movement. There was a lot of ambivalence in who should be involved and what strategies to take. This truly was an effort requiring many generations of leadership and many groups of volunteers before the community as a whole was willing to take the risks to get involved.
The miracle of this book is that for me as a white Anglo it makes me proud to be an American, but it does remind me not to get too comfortable with the benefits of being a member of the politically dominant group. It's interesting to me that while none of my immediate family were Southerners in recent generations, none of us were actively involved with helping Jim Crow end. For too many people, it was just a part of Southern culture that was too much trouble to change (especially if we did not want to offend our white Southern neighbors too much).
For African Americans, it will probably be a different experience. I hope there is also a sense of triumph, but no doubt a sense a work left to be done in terms of social equality and healing from the centuries of injustice (some of which still manifests itself today).
This leads me to the recent movie The Help also set in the Civil Rights era in Mississippi which focuses on white woman who interviews the domestic help in her neighborhood. One of the controversies of both the book and the movie is that it was the work of a white woman, a well-meaning white woman, but still NOT African American. To give the author credit, she recognizes this to some extent which is why one some of the narrative focuses on the African American ladies being more than a little skeptical that a naïve white journalist isn't going to get them all fired...or worse.
For what it's worth, I do not think it's an attempt to "co-opt" history, but rather an attempt to help grasp the harsh reality that the African-American experience at that time was often so much worse than the white experience. It may also be a bit of a romantic fantasy of how the white community should have have understood more and helped earlier, but didn't.
At the same time, I think it does address the irony that on a personal level, many white people had great respect and affection for individual African Americans. It was all distorted by the Jim Crow system, but decades after the end, the New South is often much more friendly to African Americans than the North...precisely because whites and African Americans can begin to appreciate a joint heritage and understanding.
My recommendation - watch the movie and read the book (and ignore the "dialect") ...but also go beyond that to what some African American reactions to the book. Some of it is darned good reading.
Who Can Help Fix This?
A final criticism of The Help and other movies like Mississippi Burning is that the white person becomes the protagonist of the Civil Rights movement helping "helpless" African Americans. I do have to agree that this is a just comment.
I think a lot of right-thinking whites feel terrible for the injustices our ancestors our ancestors committed (and we ourselves may be committing). It would be nice if whites could fix what we have done to another culture as easily as we destroy it.
I'm finding that although an outside culture can easily break another culture through oppression, the most long-term fix may come from the injured culture itself. A book like Eyes on the Prize is a perfect example of how African American leaders adapted the white educational system and political philosophy for their own ends....but I think only African Americans could have done it. In the same way, liberation in the Arab Spring is much quicker and probably more effective than a well-intentioned regime change from outside.
Maybe part of our "penance" is having to stand aside, take some of the blame for things we didn't personally do, and hope that some the good part of our culture can be used by someone else blended with part of their own culture to solve a problem we created.