This is a delight. Machen had existed on the fringes of the fringes of my literary knowledge: the kind of writer whose name one is unsure how to pronounce (it's "makken"); someone more referred to than read. "Few people read Arthur Machen nowadays," said one commentator many years ago, "he is the preserve, zealously guarded, of lonely men who step into the gutter when the bowler hatted jostle them in the street." I think he gets a mention in Antal Szerb's wonderfully crazy The Pendragon Legend, which would make sense, as that book is saturated with Welshness and spookiness, and those are things that Machen possessed to his fingertips.

But his legacy has been extraordinary. HP Lovecraft was influenced by him, and acknowledged the influence; Stephen King has praised his stories to the skies. His 1917 story "The Terror" must have inspired "The Birds"; "The Great Return" is the source, surely, of the belief that the holy grail finally came to rest in Wales; I bet that if you ask Neil Gaiman whether he's familiar with Machen's work, he'll respond enthusiastically; and "The Bowmen" (1914) directly and solely gave rise to the myth that an angelic consort of English archers from Agincourt rescued a battalion of British soldiers from an advancing German regiment at the battle of Mons. This gave rise to a belief that such a supernatural event had really happened, and Machen tried, in subsequent stories, to set the record straight by directly referring to it, but to little avail. "The Bowmen" isn't one of Machen's best by a long chalk; it's a rather sad bit of propaganda, but reprinted here, so you can make up your own mind.

But the typical Machen story is a terrific thing, although the terror that his first readers might have felt will have matured, a century after composition, into something approaching cosiness. A couple of late-Victorian gentlemen, one sceptical, another not so much, will debate whether there are more things in heaven and earth than are dreamt of in your philosophy, and then a weird story will be told, invoking ancient, obscure rites, the forgotten peoples of the earth, and a depraved and ancient sexuality. His first big success – his career oscillated wildly – was "The Great God Pan", a long short story that scandalised, among others, the Manchester Guardian. It is not reprinted here but there is a representation of Pan on the splendidly spooky front cover of this edition. "The White People", from 1899, is so full of suggestive scenery (a young girl clambering through thickets and bubbling streams to find a strange landscape of hollows and mounds, where she will – we infer – be impregnated by the Fair Folk) that you wonder whether he and Freud had been corresponding.

But Machen was not out to deprave or corrupt: he was as appalled by the events or implications in his stories as he hoped his audience would be. More than once an ancient artefact turns up, and it is either barely described, or destroyed. The "small piece of curious gold-work" at the end of "The Red Hand", called by its owner "the Pain of the Goat", only invites a nod to "the revolting obscenity of the thing". "Put it away, man; hide it, for heaven's sake, hide it!" says one to whom it is shown. That Machen was freaked out by his own imagination is one reason these stories are still rewarding.

We are, basically, somewhere between HP Lovecraft and MR James, but modern readers will particularly appreciate the way he appreciates the psychogeography of London, its hidden folds and histories. "The Red Hand" is very good at this, and, as a bonus, Machen invents a new method of detection based upon improbability, which makes me wonder whether he might have even, in some small way, given Douglas Adams an idea. (Probably not, but it's still a pleasing thought.)

So, whether in Wales or in London, we are faced with the antagonistic elemental evil of existence, and there is exquisite pleasure in reading these stories, tucked up in bed with a hot drink while the wind and rain beat against the windows.