As someone who would probably be associated with promoting the sort of novel being described here, I nevertheless have to say I find Alan Massie's evocation of the "self-enclosed novel" mostly incomprehensible:
One may make a distinction between two types of novel: the self-enclosed and the open. . .By the self-enclosed novel, I mean one which makes no reference — or almost no reference — to anything beyond itself. It belongs to its age of course, but it does not appear to be set in time. Time naturally passes, as it must in a narrative, but there is no suggestion that events in the world of fact beyond the novel might impinge on its characters, influence their behaviour, or affect the course of their lives. The doors of the novel are closed against the winds of the world.
I really have no idea what it would mean for a work of fiction to make "no reference. . .to anything beyond itself." It would at the least require that such a work be written in an invented language--and thus have no audience beyond the author him/herself--a language that would carry none of the "references" that English carries simply by being a historical language spoken by billions of people. And even if such a thing could be done, the invented language itself would have to make no reference to the "world of fact" its author would nonetheless still inhabit, presumably focusing entirely on an alternative "world of fact" that somehow only the author has ever experienced. This would indeed be quite a feat of self-isolation, and the resulting fiction would be cordoned off both from the actual "world of fact" and everyone inhabiting it, but the notion that some writers do this, or try to do it, is, of course, resolutely absurd.
In suggesting that certain fiction does "not appear to be set in time," Massie must mean that it does not directly refer to either current events as described by journalists or past events as related by historians. There's no other way to understand the bizarre claim that some novels want to deny "that events in the world of fact beyond the novel might impinge on its characters." Since all humans live in the world of fact and are subjected to the "winds of the world," and since writers are themselves human, the stories and novels they write bear all the marks of that wind, even if some writers are less concerned with charting it directly than other writers.
Presumably a story like Donald Barthelme's "The Balloon" is the sort of thing Massie has in mind (although he gives no examples at all of the sort of thing he does have in mind). Or a novel such as Robert Coover's The Universal Baseball Association. "The Balloon" is an obvious fantasy, in which an infinitely expandable hot-air balloon is inflated until it spreads out and covers all of New York City. The story records the way the city's people adapt to and come to understand this "phenomenon": "There was a certain amount of argumentation about the 'meaning' of the balloon; this subsided, because we have learned not to insist on meanings, and they are rarely even looked for now, except in cases involving the simplest, safest phenomena." Massie no doubt objects to the way in which a "self-enclosed" fiction like this casts doubt on "meaning," portrays meaning as something always up for grabs. But who could assert that this story rejects "the world of fact"? It is all about New York City, a "fact" that informs every line and paragraph. It's about New Yorkers, whose residence in the city most assuredly "influence[s] their behaviour" and "affect[s] the course of their lives."
The Universal Baseball Association is about as "self-enclosed" as a novel can get. It takes place completely inside the head of a man playing a game of fantasy baseball. He has created an entire league and invested it with a glorious past. He further invests it with a life-and-death significance that culminates in a horrible accident that tears his world (the baseball world) apart. Ultimately the novel is a kind of meditation on the interplay of fantasy and reality (the "world of fact" represented most obviously by baseball, a very real pastime in whose intricacies millions of people do become entangled), but it does subject its protagonist, however indirectly, to the "winds of the world." Those winds "impinge" on J. Henry Waugh in a particularly destructive way. He wants to be the God of his invented world, but the real world of chance and human imperfection intervenes nevertheless.
Massie essentially uses the distinction he draws between "self-enclosed" and "open" fiction to marginalize the former as "merely literary," while lauding the former for its willingness to "take on" history, "the world of harsh political fact which, working in conjunction with personal qualities, forms or deforms men’s lives." In Massie's view, the open novel "was invented more or less by Walter Scott," whose novels for Massie are exemplars of the kind of fiction that is "open" to the currents of reality. But I think he has it exactly backward. It's fiction like "The Balloon" and UBA that depicts the forces of "contingency" through exercises of the imagination, while writers of historical and "documentary" fiction are stuck with what was and what is. Self-enclosed fiction is actually "open" to any and all kinds of aesthetic innovation, while the "open" novel is closed to all but the most conventional approaches that allow the "world of fact" to predominate.
The important distinction to be made is not between "self-enclosed" and "open" works of fiction. It is between those works whose authors think of fiction as primarily an aesthetic form and those who think of it as a form of commentary on human behavior or the state of the the world, on "the world of harsh political fact" or some such thing. If you want to think of the latter kind of fiction as more "open," more "engaged" with facts and thus more relevant to your concerns as a reader, so be it. Some readers are impatient with art and want their novels to be like sociology only with stories, or like journalism with better stories. But this is no justification for defining a whole other kind of fiction almost out of existence and distorting it beyond recognition in the process.