atw: Re: Testing Wikipedia

Hi Graeme

Your suggestion that the only way to "test" Wikipedia reliability is to 
vandalise it and see how long it takes for long-suffering volunteers to waste 
their own valuable time (that could be spent on improving the quality of 
content) to fix up the mess. (This reminds me of the Japanese approach to whale 
research.)

I would suggest that Wikipedia's degree of reliability is reasonably well 
established, not "unknown" as you suggest, because there have been sufficient 
studies done to establish this. The links I provided are just two of many 
conducted over many years. I tried to help you out by suggesting you try 
"wikipedia reliability studies" in Google Scholar. 

Wikipedia can test its own anti-vandalism protocols without you "helping" by 
vandalising. According to a ProjectVandalism study from 2006, 4.61% of edits 
was identified as vandalism, and the average time before reversion was 2.69 
hours. 

The study I pointed you too did indeed use the unethical vandalism approach. 
However, it produced some statistics, which happily you have now read. You now 
know that a third to a half of introduced "fibs" were corrected in 48 hours. 
There you go, no need for the austechwriter list to repeat the same "study". (I 
presume you were planning to help Wikipedia by passing on the "reliability 
data" collected, or publishing it in an academic paper so that it could be used 
as a reliable source? Perhaps not?)

How was my direction to the Counter Vandalism Unit patronising? You were 
proposing vandalising Wikipedia. I was pointing out that you risk being caught 
conducting this abuse, and the Counter Vandalism Unit provides some information 
as to how you might get caught. There are over 2,500 people in this unit. They 
use dozens of different software tools to help them locate vandalism, and 
identify and block (ostracise) the culprit. As to your assertion that what you 
were proposing was not vandalism, because you might delete the page after a 
month, is nonsense. If I smash a few windows, wait for a month to see if 
someone fixes them, and then fix up any outstanding damage, is that vandalism? 
Wikipedia's definition of vandalism in the context of what you were proposing 
(from Wikipedia editorial policies) is as follows.

Vandalism is any addition, removal, or change of content in a deliberate 
attempt to compromise the integrity of Wikipedia. Examples of typical vandalism 
are adding irrelevant obscenities and crude humor to a page, illegitimately 
blanking pages, and inserting patent nonsense into a page. Vandalism is 
prohibited. While editors are encouraged to warn and educate vandals, warnings 
are by no means necessary for an administrator to block. 

The only loophole is "good faith", and maybe that's how the Magnus study got 
ethics approval. I can't say. 

I am very happy to assume that you were acting in good faith, but that doesn't 
preclude me from pointing out that what you were proposing fits the Wikipedia 
definition of vandalism. And continuing with that assumption of good faith, can 
I recommend another good read, namely "User Contribution and Trust in 
Wikipedia" at www.ics.uci.edu/~sjavanma/CollabCom.

Tony Self



>>> Graeme Worth 31/08/11 2:45 AM >>>
Hi Tony

Date: Mon, 29 Aug 2011 17:37:12 +1000 
From: "Anthony Self" 
Subject: Re: Wikipedia - can we test it?
|Hi Graeme 
|What you are suggesting (creating Wikipedia articles with deliberate errors) 
is unethical. You would possibly be breaching the terms of use, and you would 
certainly risk being ostracised. It is |known as vandalism or abuse.

So, taken in context with previous remarks, this roughly paraphrases as:
Wikipedia is a source of unknown reliability.
Wikipedia itself apparently has no interest in testing its own protocols for 
reliability.
Anyone who suggests running such tests is classified as a 'vandal' and is 
ostracised.
Brilliant - and this helps Wikipedia how?

|You may think you are anonymous, and therefore won't be caught. (And low risk 
of getting caught shouldn't be a justification for commiting anti-social acts.) 
It is important to realise that there is not |absolute anonymity on the 
Internet. The staff of the Prime Minister's Department discovered this when 
they were caught out making "bad faith" edits to the Wikipedia articles on 
Peter Costello, |immigration policy, etc. Wikipedia uses a number of software 
"agents" to identify vandalism, and third party sites, such as Wikipedia 
Scanner, also serve as a kind of "sheriff". There is also a group |of 
volunteers within Wikipedia called CVU - Counter Vandalism Unit. 
|http://en.wikipedia.org/wiki/Wikipedia:CVU
|For an idea about how abuses of Wikipedia are reported and handled: 
|http://en.wikipedia.org/wiki/Wikipedia:Abuse_reports

Thank you for your somewhat patronising direction to the Wikipedia Counter 
Vandalism Unit. I'm really not sure if you actually read my post for what it 
was, or for what you thought it might be - there again, this is the Austech 
Writers forum, so that is probably a stupid question. What I suggested should 
be done was so far removed from 'vandalism' that your response astonishes me. 
Did you seriously consider that I would suggest these article would be left 
there? Doesn't the phrase 'for a reasonable period of time' suggest that after 
that time the article would be removed?

|There have already been a number of studies into vandalism and the reliability 
of Wikipedia content. To locate these, try using the phrase "wikipedia 
reliabilty studies" in Google Scholar. As a first |port of call, I'd suggest 
you read "Early response to false claims in Wikipedia", 
http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2115.
 Another good one is "Epistemology |and the Wikipedia" 
http://dspace.sunyconnect.suny.edu/handle/1951/42589.

Have you actually read these articles you refer to? The first details a study 
where the author deliberately introduced "fibs" into existing articles to see 
how long it would take to have them corrected. So how, exactly, is that 
different from what I proposed? Also of interest are his conclusions - only 1/3 
to 1/2 of the errors were removed or corrected within 48 hours - hardly an 
example of the success of the system you so vigorously promoted earlier in this 
thread.
The second article, as you say, is a good one.
Having read both of these I cannot believe you made the comments you did about 
the reliability of Wikipedia and the systems put in place to eradicate errors. 
These systems are largely untested in any rigorous way, and what tests have 
been done suggest, if anything, that Wikipedia, overall, is far from reliable.
Having read this forum for some considerable time I really should have known 
better. I doubt that I'll bother contributing again.

|Please don't go smashing windows to see how long it takes for them to be 
repaired.

But it would be a very quick repair wouldn't it - that's what you told us 
originally!
Graeme

|Tony Self

Other related posts:

  • » atw: Re: Testing Wikipedia - Anthony Self