More or less, it means "referring to itself". Recursive science fiction might mean a book like Bimbos of the Death Sun, a very tongue-in-cheek murder mystery set at a science fiction convention. Hey, don't look at me that way - Linda liked it too!
And then there are books about books, writing, bookselling, book collecting . . . and librarians generally are unable to resist them. Right now I'm reading The Magician's Book by Laura Miller. Begun as a Salon.com essay about her on-again, off-again love for The Chronicles of Narnia, she has expanded it into a fascinating book about reading and C. S. Lewis and British legends and the Welsh countryside and friendship and other topics guaranteed to keep me glued to it for the next few days.
Miller says she wanted to go to Narnia when she was nine so badly she felt she would actually die. Then she stumbled on "what everybody knew", that the Chronicles were Christian stories or allegories (she has several pages on what allegory really is and why Lewis loved it). She felt betrayed by this alternate meaning, when she had little interest in religion, and an overpowering delight in fauns and dryads.
I read the Chronicles when I was a few years older, and the Christian elements were more obvious to me. Mostly, I enjoyed them, though I still liked the fauns and dryads more. But as I grew older, and read the books again (and again), I loved exploring the double layer of meaning, and I knew that there were more layers of poetry and mythology and the War and England that I was missing. Lewis was an Oxford don, after all.
And that's what Miller's wonderful book does for me: explores all of the layers in the books, without ever forgetting the power of the story, the ordinary cleverness and kindness of the characters, and the beauty of the wild garden that is Narnia.
There are many wonderful "recursive" books in the library, and here are a few staff favorites:
The Child that Books Built: A Life in Reading by Francis Spufford
Why We Read What We Read by Lisa Adams
Books: A Memoir by Larry McMurtry
And here's the big list of what the library owns on that subject:
Books and Reading
Now, books about bookstores, that's a whole 'nother post!