Cleaning flower pots and seed trays is not something that I imagine anyone relishes, but it is worth the effort. I find a combination of a pot brush and an old vegetable brush will brush up most plastic and terracota pots pretty well.
The reason is to avoid residual pests and soil borne infections which can destroy new seedlings. A small amount of soil left on the side of a pot can be enough to harbour desease. If you're feeling particularly energetic a scrub with a natural disinfectant such as white vinegar should see off any last bugs.