Automated versus Human Judgments

Automated versus Human Judgments: A couple of posts provoke an interesting discussion: William Cohen points to the issue of the popularity contest approach to ranking which may have undesirable consequences [...] As for the issue of automated ranking of web pages. The problem cited above exposes the frailty of addressing a content problem (finding a document whose text is appropriate) via an orthogonal structural solution. The structural solution (counting links and propagating results) may do well in some domains where it is regarded as a proxy for measurements of 'authority', however, the ambiguity in the structure cannot be determined, leading to the type of problem William cites. This is where solutions like Powerset come in. (Via Data Mining.)

Thanks for the link to William's blog which I didn't know about. Regarding this particular search ranking issue, where related lexical items have very different contexts of usage and associated sentiments — negative vs neutral or positive — I'm curious of what NLP methods, embodied Powerset's system or in any other system, or even in early research prototypes, would Matt recommend for a solution. The problem is not one of syntax, semantics, or even pragmatics and local discourse, as it can be easily seen from several controversies in this country where a word is considered derogatory when used by some people but friendly or even complimentary when used by others; and people can and will get into hot water when they breach those invisible but very real boundaries. There's a lot more in the context and charge of writing than any of our current automated methods can discern, whether they use global statistics or local structure. It's not a matter of ambiguity — the denotation of the terms is not in question — but one of association and rhetorical force — what ideas and feelings are triggered in the minds of different readers and writers by particular terms as a result of their social and cultural backgrounds and of their (lack of) sensitivity.

The original post by Lauren Weinstein that triggered this thread was about the visible global impact of search rankings, but William's discussion suggests a less global but possibly more powerful effect in search personalization, of whether a personalization algorithm could become a strong reinforcer of prejudice without the counter-pressure of critical discussion of globally visible search rankings.