I purchased a small tea olive that same year and waited patiently. The first two years it bloomed sparsely and grew very little. I was concerned that I had bought the wrong species of osmanthus because the leaves were so small. This year, however, the plant had that third-year growth spurt. The leaf size increased and it rewarded me this fall with many fragrant blooms.
As you can see from the picture below, the blooms are tiny and nestled at the base of the leaves. From a distance, the blooms are hardly visible. I think this is part of the plant's charm: the nose discovers it first, and then the eyes must seek it out. Sometimes the scent eludes me if I pursue it too aggressively. I bend over to sniff the flowers and smell nothing, then later, detect the fragrance wafting across the garden from quite a distance.
If you are a fan of fragrant plants, I highly recommend the tea olive. In the three years or so it has been in our garden, I have given it no special care: no fertilizer, no spraying for bugs or disease, and very little watering. It is evergreen and blooms in the late fall/early winter and sporadically at other times of the years (it was early spring when I ran across it at the Dixon).
If you shop for one, be aware that there are a number of osmanthus species and a number of plants with "tea" in their common name (for example, Camellia sinensis, the tea plant). This article is informative about the various osmanthus species and cultivars. The one I've been discussing is Osmanthus fragrans, easy to remember if you think about it being named for its wonderful fragrance.