The Equidistance Index of Population Structure

Yaron Granot, Omri Tal, Saharon Rosset, and Karl Skorecki

Contact the author: Please use for correspondence this email.
Submission date: 23. Mar. 2016
Pages: 21
published in: PLOS ONE, 11 (2016) 8, art-no. e0160413 
DOI number (of the published article): 10.1371/journal.pone.0160413
Bibtex
with the following different title: On the apportionment of population structure
Keywords and phrases: FST, population structure, panmixia, differentiation
Download full preprint: PDF (1284 kB)

Abstract:
Measures of population differentiation, such as F_ST, are traditionally derived from a partition of heterozygosity within and between populations. However, the emergence of population clusters from multilocus analysis is a function of genetic structure (departures from panmixia) rather than of diversity. If the populations are close to panmixia, slight differences between the mean pairwise distance within and between populations (low F_ST) can manifest as strong separation between the populations, thus population clusters are often evident even when the vast majority of diversity is partitioned within populations rather than between them. Moreover, because F_ST is also a function of internal diversity, it does not directly reflect the strength of separation between population clusters. For any given F_ST value, clusters can be tighter (more panmictic) or looser (more stratified), and in this respect higher F_ST does not always imply stronger differentiation. Finally, F_ST as a measure of structure or population distance is a supervised measure, in the sense that target populations have to be predefined (samples labeled). In this study we propose a measure for the partition of structure, denoted E_ST, which is more consistent with results from clustering schemes. Crucially, our measure is based on a statistic of the data that is a good measure of internal structure, mimicking the information extracted by unsupervised clustering or dimensionality reduction schemes. To assess the utility of our metric, we ranked various human (HGDP) population pairs based on F_ST and E_ST and found substantial differences in ranking order. In some cases examined, most notably among isolated Amazonian tribes, E_ST ranking seems more consistent with demographic, phylogeographic and linguistic measures of classification compared to F_ST. Thus, E_ST may at times outperform F_ST in identifying evolutionary significant differentiation.