Short version: If I didn't see it, it didn't happen.
As a reader or a viewer, I don't expect to be told every detail and nuance of a story. Conservation of detail is important - otherwise we have to sit through ten pages of a person's morning routine every time the story goes to a new day, and as the works of Keiran Halycon have shown us, this is a terrible prospect.
But what you leave out is just as important as what you put in, which brings us to the key point of this rant: for a detail to count as part of the work, it must appear in the body of the work.
More definition will help. I cannot consider something part of a story if any of the following are true:
1. It takes place outside of the depicted action and is related to the main characters by another source (TV Tropes "Second Hand Storytelling").
2. It does not take place within the story, but is expounded upon by the creator at a later date (TV Tropes "Word of God").
3. It neither takes place nor is discussed, but is "filled in" by the fans.
The last, of course, is a category that most would agree with, but the first two might be points of contention. Examples will help.
I must again return to Harry Potter. One of the major reasons the last book and seventh film are so argued over is because of the massive amount of action that takes place in the background, unseen by either the characters or the viewers. Great deals of activity and character growth, including something in the region of 65% of Neville's character development, is entirely unshown, only related to Harry by other sources. There's a reason fans joke that the best book in the series is the one that takes place off-camera.
As a result, I cannot accept Neville as a strong character. And how could I? Everything that led to his development from a nebbish to a leader takes place unseen by the reader or viewer.
Then, of course, there are J.K. Rowling's interviews. I do not consider them canon. Author or not, she forfeits any right to develop her world further when she ceases to write in it. Spoken statements are not the same as written story, and do not hold the same weight.
Another example are the "appendixes", "compendiums", "encyclopedias" or other such business that some authors come up with. These hold a little more weight to me, but they feel like much of the same stuff as the interviews - airy attempts to continue holding onto the world you've created without actually adding to it.
Such bothers me not because I think an author has no say in the structure of their stories - far from it. But all of this seems a transparent attempt to insert an entirely different story into the margins of a story that's already on-going, entirely on your say-so. Without the immediacy of shown action to drive it, what's the point? It is invisible to us, as real as fairy's air, and about as substantial.
For an example from the third category, the fan-created, I turn to an anime series I'm fond of. In the final episode, two characters are apparently killed. I found the scene touching, and was not thrilled to hear that they were spared in an audioplay released in Japan. When another fan said "Their story's not over, they have a lot of atonement ahead of them" or something of that like, I exploded a little inside.
The story is over. The audioplay was released after the end of the series. No more episodes will be made; no further stories told. The characters were villains and murderers; the story is over, and thus they have gotten away with serious crimes with no true punishments. Their deaths were undone for no good reason, cheapening the end of the series, and no amount of fan-created "atonement ahead of them" will change that.
When writing your story, think carefully about what you put in the foreground and what you put in the background. Don't try to convince us there's anything truly amazing going on where we can't see it, and don't try to turn floodlights on the background later.