Excerpt from by "The Pinhole Camera Revisited or The Revenge of the Simple-Minded Engineer", by Kjell Carlsson:
"The image of a point object, the Point Spread Function (PSF),
should be as small as possible to produce a sharp image.
"Geometrical optics approximation: Hole should be as small as possible.
"Fraunhofer diffraction approximation: Hole should be as large as possible.
"No optimum size is found!
"Conclusion: The optimum size is found in a region where neither geometrical optics nor Fraunhofer diffraction is valid!"
For the more mathematically inclined, this paper discusses the preceeding conundrum and has simulated sample images for various pinhole sizes. I can’t follow the math, but I found it an interesting read nonetheless.