Testing the Redundancy of English

Find the fraction of the letters in English text that can be removed, while still allowing the text to be read

As a random sample of text, get the Wikipedia entry about "chicken":

In[1]:=

chtext=TextSentences[WikipediaData["chicken"]];

It contains 259 sentences:

In[2]:=

Length[chtext]

Out[2]=

253

Here is the 40th sentence:

In[3]:=

chtext〚40〛

Out[3]=

Domestic chickens are not capable of long-distance flight, although lighter chickens are generally capable of flying for short distances, such as over fences or into trees (where they would naturally roost).

Pick a random sentence, removing 10% of the characters:

In[4]:=

StringJoin[If[RandomReal[]<.1,"-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[4]=

In olde- s-urces, chicken as a species-were -yp-cally refe-red to as common fowl or domes--c -owl.

Remove 20% of the characters, never removing a space:

In[5]:=

StringJoin[If[RandomReal[]<.2&&#=!=" ","-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[5]=

An unusua- -ariety of c-i-k-n -hat has i-s or-gin- i- -out- A-erica -s t-e --a-cana, -red in southern -hile by th- Ma-uc-e p-opl-.

Remove 30% of the characters:

In[6]:=

StringJoin[If[RandomReal[]<.3&&#=!=" ","-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[6]=

Ke-pi-g c-ic-e-s as -ets became incr---i--ly pop-l-- in the 2-00s amo-g u-ban and suburban res-d--ts.

Remove 40%:

In[7]:=

StringJoin[If[RandomReal[]<.4&&#=!=" ","-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[7]=

The -roo-- --n -il- sto- layi-- --- i------ wil- --cu- -n th- i---ba-i-n o- th- eggs -a full c-ut-- i- usua-l- -b--t 1- eg-s).

Try it again:

In[8]:=

StringJoin[If[RandomReal[]<.4&&#=!=" ","-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[8]=

In 2003- more t-a- 75- of al- -l--k- -er- --u-t-d in t-e US-

Remove 50%:

In[9]:=

StringJoin[If[RandomReal[]<.5&&#=!=" ","-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[9]=

--i- t-rm- --n--in- - ---k --pt -o- ---e- sp--t- p---im- o- ent-r-ain-ent, ------- i- -6--- af--r "---- -- -he -am-- -sed b- G-o--e --l-o- -n -he -arl--st k--wn -o-- -n t-- -ecu-ar -po--- T-- ---m---atio- -- -o-ks a-d C-c- Fi-ht-ng of --07.

Remove 70%---and the text becomes unreadable:

In[10]:=

StringJoin[If[RandomReal[]<.7&&#=!=" ","-",#]&/@Characters[chtext〚RandomInteger[253]〛]]

Out[10]=

---- c----r---l ---ub---rs a-- -n-u---i------ed -i-h -h-lve- ---d-n- ---- -- th----n-s -- e--- -t - ----, -i-h r-----o- -- -he --g- - -ully auto---ed -----s--

Publisher Information

Contributed by: Stephen Wolfram