Frank Wilson thinks that "artistic experiment" is defined by the amount of "trial and error" involved. He takes the scientific "experiment" to be the model for the use of "experimental" as a classificatory term in the discussion of literature. Scott Esposito more or less accepts Wilson's definition, although he has no problem with "art experiments being praised as ends in themselves," something about which Wilson seems skeptical. Scott also suggests that "unlike in science, we can continually come back to and learn new things from successful literary experiments, or simply admire their beauty."
Actually, we can probably do the same with certain especially compelling scientific experiments, but I think both Scott and Frank are mistaken to view "experiment" in literature as essentially analgous to the way the term is understood in science. Scott is correct in asserting that "a lot of trial and error is involved in the writing of most novels," and for that very reason "trial and error" is not really very helpful in capturing what literary critics/scholars have meant by using "experimental fiction" in describing selected works of fiction, especially fiction written since 1945, "experimental." For the most part, critical commentary on postwar experimental fiction (more broadly "postmodern" fiction) has focused on "experiment" as, in Jerome Klinkowitz's words, the "disruption" of a "conservative stability of form" in literary fiction (Literary Disruptions, 1980). Klinkowitz thought this stability had reigned since the 1920s, but it probably goes back farther than that, to the establishment of realism in the mid/late 19th century--experimentalists such as Joyce and Woolf could be said to have also "disrupted" this stability of form (often characterized as the "well-made story"), although their experiments did not disrupt the assumptions of realism itself, were in fact an extension of these assumptions into what is now called "psychological realism." From this perspective, "trial and error" is not so much the guiding principle of experiment (except insofar as it involves finding appropriate methods of disruption) as is the notion that "stability"--to which scientific experiments always return--is itself not a desirable state where the art of fiction is concerned.
It is true that the term "experimental fiction" is a catch-all term of convenience that doesn't necessarily signal anything very specific about what particular writers might be up to in their efforts to, if not "make it new," then "make it different." Thus Klinkowitz prefers "disruption," while other critics have written about "breaking the sequence" or "the art of excess" or "anti-story." In most cases, however, these critics are really interested in what Ellen Friedman and Miriam Fuchs in Breaking the Sequence simply accept as "innovations in form." Friedman and Fuchs also provide a handy description of the elements of "stability" against which most innovative writers are rebelling: "Plot linearity that implies a story's purposeful forward movement; a single, authoritative storyteller; well-motivated characters interacting in recognizable social patterns; the crucial conflict deterring the protagonist from the ultimate goal; the movement to closure. . . ." Perhaps the most succinct statement of the motivations underlying experimental fiction would be the remarks made by John Hawkes, which I've quoted on this blog before: "I began to write fiction on the assumption that the true enemies of the novel were plot, character, setting and theme, and having once abandoned these familiar ways of thinking about fiction, totality of vision or structure was really all that remained."
A critic who did use the term "experimental fiction" straightforwardly was Robert Scholes in his book Fabulation and Metafiction. In the chapter of that book called "The Nature of Experimental Fiction," he writes: "Forms atrophy and lose touch with the vital ideas of fiction. Originality in fiction, rightly understood, is the successful attempt to find new forms that are capable of tapping once again the sources of fictional vitality." Scholes's book popularized the term "metafiction" as a more specific term describing the tendencies in postwar American fiction that made readers think of them as "experimental": "Metafiction. . .attempts to assault or transcend the laws of fiction--an undertaking which can only be achieved from within fictional form." Writers like Gass, Barth, Coover, and Barthleme were "working in that rarefied air of metafiction, trying to climb beyond Beckett and Borges, toward things than no critic--not even a metacritic, if there were such a thing--can discern."
Eventually that air probably became too "rarefied." Many readers came to associate metafiction--and thus "experimental fiction"--as "game-playing," an obsession with "art" over "life." This perception probably informs Frank Wilson's disdain for experiments "as ends in themselves." (Also his disinclination to think of Joyce as an experimenter--Joyce "knew from the start what he was going to do and how he was going to do it" and would never have stooped to mere "experiment.") It may also explain why Christopher Sorrentino, in a comment on Scott Esposito's post, observes that "while I have met a great many novelists ranging in outlook and approach from Ben Marcus to Jonathan Franzen, not a single one of them to my knowledge has ever described his/her work as 'experimental.'" While the writers associated with the Journal of Experimental Fiction would probably be less skittish about the designation than Marcus or Franzen, it probably is now true that many "experimental" writers (putting aside whether Ben Marcus or Jonathan Franzen are actually very innovative in the first place), are uncomfortable with the word as applied to their work. I'd still accept the explanation given by Raymond Federman, who coined the term "surfiction" as an alternative to "metafiction" in identifying his own brand of experimental fiction:
The kind of fiction I am interested in is that fiction which the leaders of the literary establishment (publishers, editors, agents, and reviewers alike)* brush aside because it does not conform to their notions of what fiction should be; that fiction which supposedly has no value (commercially understood) for the common reader. And the easiest way for these people to brush aside that kind of fiction is to label it, quickly and bluntly, as experimental fiction. Everything that does not fall into the category of successful fiction (commericially that is), or what Jean-Paul Sartre once called "nutritious literature," everything that is found "unreadable for our readers". . .is immediately relegated to the domain of experimentation--a safe and useless place. (Surfiction, 1975)
* Including Frank Wilson?