The structure of small (2-5 nm) Ge quantum dots prepared by the colloidal synthesis route is examined. Samples were synthesized using either GeO2 or GeCl4 as precursor. As-prepared samples were further annealed under Ar or H2/Ar atmosphere at different temperatures in order to understand the effect of annealing on their structure. It was found that as-prepared samples possess distinctly different structures depending on their synthesis route as indicated by their long-range ordering. An appreciable amount of oxygen was found to be bound to Ge in samples prepared with GeO2 as a precursor; however, not for GeCl4. Based on combined transmission electron microscope, Raman, X-ray diffraction and X-ray absorption measurements, it is suggested that as-prepared samples are best described by the core-shell model with a small nano-crystalline core and an amorphous outer layer terminated either with oxygen or hydrogen depending on the synthesis route. Annealing in an H2Ar atmosphere leads to sample crystallization and further nanoparticle growth, while at the same time reducing the Ge-O bonding. X-ray diffraction measurements for as-prepared and annealed samples indicate that diamond-type and metastable phases are present.