The brightest white is a simple choice. You can rest assured that you will have no color harmony issues. White, having no hue content, matches universally with all other hues. On the other hand, that bright white can be, on occasion, a tad blinding at times. It will also show grime more quickly than a white with some color in it.
An off-white is simply a white with a speck of pigment from another color or colors. You may not even notice that it is not pure white in some light situations, but the hint of color does two important things. It restrains the intensity of a pure white, and it pushes the white in either a warm or cool direction. A trace of blue with make the white cooler, while a tinge of red will make a warm, subtle pink. Generally, off-whites are blended with neutral browns or grays to keep the color content muted.
An extremely light neutral is just a shade more pigmented than an off white. Often you will find a nice array of them in the line of a paint company's so-called "historic colors." These colors add enough color to express more of a shade -- say, a soft breath of brown-mauve -- but stay light enough that they won't conflict with existing door, trim or landscape colors.
For light pastels, the key is the degree to which they are pigmented. Rather than mixing white with a neutral, a dash of pure color, like yellow or green, will be added. These colors will have a clear color expression, but will also be extremely light and white. Due to the color expression, it is advisable for these very light pastels that you exercise more care in the color selection, and seek to match the color tendency of the pastel with any colors they are to be paired with.