In other words the detective story tells lies about human nature but is not allowed to lie freely or fantastically. It's neither realism nor magic realism but something in between.
The writer of detective fiction willingly dons a straitjacket.
It's no wonder, then, that there are so few good detective stories that are also great literature.
Bleak House is a detective story but it's not a good detective story. The problem is too elementary.
The Sherlock Holmes stories are great literature by accident. Doyle thought he was writing cheap magazine stories and stumbled into a whole new world. He is a great original. He created the genre and more or less exhausted it. Most later detectives and their sidekicks are shadows of Holmes and Watson.
G.K. Chesterton pushed the detective story as far in the direction of fantasy as it will go. A handful of the Father Brown stories are brilliant fables, brilliantly written. Most of them are just too wild
A number of the writers of the Golden Age attempted to write detective stories that were also serious literature. Margery Allingham, anyone?
The more literary Dorothy L. Sayers attempted to be, the more unreadable she became.
Agatha Christie is the best. The Murder of Roger Ackroyd is perfect of its kind. It's like the Rubick cube- wonderfully simple, wonderfully cunning. There's no way you could improve on it. A masterpiece, but not a literary masterpiece.
Ruth Rendell sensibly divides herself in two. Rendell for detection, Vine for literature.
I can think of only one great novel that's also a great detective story. The Name of the Rose. Not even its creator has been able to duplicate its success.