small mandarins

 My mandarin tree has only very small fruit.  I've only had it for a couple
of years, but there's quite a bit of fruit already, could this be why?
Should I have removed some of the fruit when they first began?  Will they be
better next year?

Thanks for any help

Jen

 

May be trimming back small branches should help too.