Observations of globular clusters (GCs) and field stars in the halos of the giant elliptical galaxy Cen A and the spiral galaxy M31 show a large range of cluster-to-star number ratios (or 'specific frequencies'). The cluster-to-star ratio decreases with increasing metallicity by over a factor of 100-1000, at all galactocentric radii and with a slope that does not seem to depend on radius. In dwarf galaxies, the GCs are also more metal-poor than the field stars on average. These observations indicate a strong dependence of either the cluster formation efficiency and/or the cluster destruction rate on metallicity and environment. We aim to explain the observed trends by considering the various effects that influence the cluster-to-star ratio as a function of metallicity, environment and cosmological history. We discuss the following effects that may influence the observed cluster-to-star ratio: (a) the formation efficiency of GCs; (b) the destruction of embedded GCs by gas expulsion; (c) the maximum masses of GCs; (d) the destruction of GCs by tidal stripping, dynamical friction, and tidal shocks as a function of environment; (e) the hierarchical assembly of GC systems during galaxy formation and the dependence on metallicity. We show that both the cluster formation efficiency and the maximum cluster mass increase with metallicity, so they cannot explain the observed trend. Destruction of GCs by tidal stripping and dynamical friction destroy clusters mostly within the inner few kpc, whereas the cluster-to-star ratio trend is observed over a much larger range of galactocentric radii. We show that cluster destruction by tidal shocks from giant molecular clouds in the high-density formation environments of GCs becomes increasingly efficient towards high galaxy masses and, hence, towards high metallicities. The predicted cluster-to-star ratio decreases by a factor 100-1000 towards high metallicities and should only weakly depend on galactocentric radius due to orbital mixing during hierarchical galaxy merging, consistent with the observations. The observed, strong dependence of the cluster-to-star ratio on metallicity and the independence of its slope on galactocentric radius can be explained by cluster destruction and hierarchical galaxy growth. During galaxy assembly, GC metallicities remain a good tracer of the host galaxy masses in which the GCs formed and experienced most of their destruction. As a result, we find that the metallicity-dependence of the cluster-to-star ratio does not reflect a GC formation efficiency, but a survival fraction.