According to Matt Ridley,
There is a growing conviction within neuroscience that one of the human mind's chief preoccupations is prediction. Jeff Hawkins, the founder of Palm Computing who is now a full-time neuroscientist, argued in his 2004 book "On Intelligence" that the mind does this by detecting a familiar pattern in its input, then anticipating from past experience what usually follows. The more unexpected something is, the more conscious we are of it.
Although this research concerns immediate disruptions of pattern (sudden surprises or anomalies), and therefore I would not suggest an extrapolation to more deliberative situations is necessarily scientifically sound, the human brain's response to changes in "familiar pattern" as described here by Ridley does make me wonder if something like this phenomenon helps explain many readers' resistance to experimental or innovative fiction.
Through their own habitual reading experiences and/or the critical discourse and pedagogical practices associated with fiction, most readers believe "novel" or "short story" name identifiable forms with discernible features. Stories generally put more emphasis on plot, novels allow more development of character, both establish a setting within which the plot and characters are delineated, etc. Some stories or novels might be allowed to deviate to a limited extent from the underlying standard, as long as they can still be recognized as the sort of thing a novel or story is supposed to be.
An experimental fiction introduces the "unexpected." It makes the reader conscious the implicit paradigm is being violated. One might hope at this point that the reader would exploit this intensified consciousness of difference to give the new a chance, to let the work be what it will be. Some readers no doubt do this, perhaps assuming the work eventually can be accommodated to the paradigm after all. Most probably don't, either giving up when it continues to be "difficult" or finishing it but pronouncing it "boring" or "pointless" or "a slog."
That readers might be confused or uncertain when confronted with an aberrant work of fiction is understandable. What is disappointing is that these readers can't summon up more curiosity about the challenging or the unusual, using it to expand their appreciation of the possibilities of fiction rather than shut them down in favor of the already familiar. It's my belief that fiction as a literary art depends upon challenges to convention or it becomes just a somewhat more respectable alternative to watching tv. Perhaps many readers are comfortable with this role for fiction, preferring not to burden it with the expectation it be "literature." Perhaps it is just a fact of our brain's wiring that we favor the customary and find its transgression disturbing. In this way we are all inescapably conservative.
Many innovations in fictional form or style--although not all--eventually become more accepted, more established as among the devices readers of fiction might encounter in stories and novels. Many writers initially judged too difficult or too adventurous gradually seem less so, although often enough the greatest of such writers, Joyce or Beckett or Faulkner, still can give uninitiated readers fits. By that time, of course, their innovations have themselves become conventions, which can be as overworked as any other, consolidated into "familiar pattern." It may be that this is the best adventurous writers can hope for: long-lasting influence, but at best late recognition of their accomplishments.