PLoS Genetics
Estimating FST and kinship for arbitrary population structures
DOI 10.1371/journal.pgen.1009241 , Volume: 17 , Issue: 1
Article Type: research-article, Article History
•
•
• Altmetric

### Notes

Abstract

Kinship coefficients and FST, which measure relatedness and population structure, respectively, are important quantities needed to accurately perform various analyses on genetic data, including genome-wide association studies and heritability estimation. However, existing estimators require restrictive assumptions of independence that are not met by real human and other datasets. In this work we find that existing estimators can be severely biased under reasonable scenarios, first by theoretically determining their properties, and then using an admixture simulation to illustrate our findings. In particular, we find that existing FST estimators are downwardly biased, and that existing kinship matrix estimators have related biases that are on average downward and of similar magnitude but vary for every pair of individuals. These insights led us to a new estimation framework for kinship and FST that is practically unbiased for any population structure, as demonstrated by theory and simulations. Our new approaches—available as open-source R packages—are easy to use and are more widely applicable than existing approaches, and they are likely to improve downstream analyses that require accurate kinship and FST estimates.

Ochoa, Storey, and Feldman: Estimating FST and kinship for arbitrary population structures

## Introduction

In population genetics studies, one is often interested in characterizing structure, genetic differentiation, and relatedness among individuals. Two quantities often considered in this context are FST and kinship. FST is a parameter that measures structure in a subdivided population, satisfying FST = 0 for an unstructured population and FST = 1 if every locus has become fixed for some allele in each subpopulation. More generally, FST is the probability that alleles drawn randomly from a subpopulation are “identical by descent” (IBD) relative to an ancestral population [1, 2]. The kinship coefficient is a measure of relatedness between individuals defined in terms of IBD probabilities, and it is closely related to FST, since the mean kinship of the parents in a subpopulation is the FST of the following generation [1].

This work focuses on the estimation of FST and kinship from biallelic single-nucleotide polymorphism (SNP) data. Existing estimators can be classified into parametric estimators (methods that require a likelihood function) and non-parametric estimators (such as the method-of-moments estimators we focus on, which only require low-order moment equations). There are many likelihood approaches that estimate FST and kinship, but these are limited by assuming independent subpopulations or Normal approximations for FST [311] or totally outbred individuals for kinship [12, 13]. Additionally, more complete likelihood models such as that of Jacquard [14] are underdetermined for biallelic loci [15]. Non-parametric approaches such as those based on the method of moments are considerably more flexible and computationally tractable [16], so they are the natural choice to study arbitrary population structures.

The most frequently-used FST estimators are derived and justified under the “independent subpopulations model,” in which non-overlapping subpopulations evolved independently by splitting all at the same time from a common ancestral population. The Weir-Cockerham (WC) FST estimator assumes subpopulations of differing sample sizes and equal per-subpopulation FST relative to the common ancestral population [17]. The Weir-Hill FST estimator generalized WC for subpopulations with different FST values, and first considered arbitrary coancestry between subpopulations, resulting in estimates of a linearly-transformed FST, namely $\left({F}_{\text{ST}}-\stackrel{˜}{\theta }\right)/\left(1-\stackrel{˜}{\theta }\right)$ (where $\stackrel{˜}{\theta }$ is the unknown mean coancestry value between subpopulations) [4, 18, 19]. Weir-Hill has further evolved into the Weir-Goudet approach, incorporating relatedness for subpopulations and individuals based on allele matching, also estimating a linearly-transformed FST [2022]. Note that the Weir-Hill and Weir-Goudet approaches intended to estimate such linearly-transformed quantities, which may be negative, and they did not aim to estimate IBD probabilities [4, 1822]; in contrast, our goal is to estimate IBD probabilities, which must be non-negative and valid probabilities. The “Hudson” FST estimator [23] assumes two subpopulations with different FST values. All of the previous FST estimators are ratio estimators derived using the method of moments to have unbiased numerators and denominators, which gives approximately unbiased ratio estimates when their assumptions are met [4, 17, 23]. We also evaluate BayeScan [10], which estimates population-specific FST values using a Bayesian model and the Dirichlet-Multinomial likelihood function—thus representing non-method-of-moments approaches—but which like other existing FST estimators also assumes that subpopulations are non-overlapping and evolve independently. These FST estimators are important contributions, used widely in the field.

Kinship coefficients are now commonly calculated in population genetics studies to capture structure and relatedness. Kinship is utilized in principal components analyses and linear-mixed effects models to correct for structure in Genome-Wide Association Studies (GWAS) [16, 2430] and to estimate genome-wide heritability [31, 32]. Often absent in previous models is a clear identification and role of the ancestral population T that sets the scale of the kinship estimates used. Omission of T makes sense when kinship is estimated on an unstructured population (where only a few individual pairs are closely related; there T is the current population). Our more complete notation brings T to the fore and highlights its key role in kinship estimation and its applications. The most commonly-used kinship estimator [16, 27, 3036] is also a method-of-moments estimator whose operating characteristics are largely unknown in the presence of structure. We show here that this popular estimator is accurate only when the average kinship is zero, which implies that the population must be unstructured.

The goal of our work is to consistently estimate IBD probabilities, namely kinship coefficients and FST , for which there are currently no consistent estimators under general relatedness. Estimation of these as probabilities, as opposed to linearly-transformed quantities that may be negative, is important since the probabilistic definition of these parameters was required to derive their fundamental connections to many applications in genetics, including allele fixation [1, 2, 37], DNA forensics [3], and heritability [38, 39]. Although IBD probabilities are not absolute, but rather depend on the choice of ancestral population [40], their values become fixed upon agreeing to estimate them in terms of the Most Recent Common Ancestor (MRCA) population, which has long been the choice for models of FST [17, 23, 41] and kinship estimation from pedigrees [42, 43] or markers [12, 13].

Recent genome-wide studies have revealed that humans and other natural populations are structured in a complex manner that break the assumptions of the above estimators. Such complex population structures has been observed in several large human studies, such as the Human Genome Diversity Project [44, 45], the 1000 Genomes Project [46], Human Origins [4749], and other contemporary [5054] and archaic populations [55, 56]. We have also demonstrated that the global human population has a complex kinship matrix and no independent subpopulations [5759]. Therefore, there is a need for innovative approaches designed for complex population structures. To this end, we reveal the operating characteristics of these frequently-used FST and kinship estimators in the presence of arbitrary forms of structure, which leads to a new estimation strategy for FST and kinship.

Here, we study existing FST and kinship method-of-moments estimators in models that allow for arbitrary population structures (see Fig 1 for an overview of the results). First, in section The generalized FST for arbitrary population structures we present the generalized definition of FST for arbitrary population structures [57]. In section The kinship and coancestry models we review the kinship model for genotype covariance [1, 14] and the coancestry model for individual-specific allele frequencies [57, 60, 61]. In section Assessing the accuracy of genome-wide ratio estimators we obtain new strong convergence results for a family of ratio estimators that includes the most common FST and kinship estimators. Next, we calculate the convergence values of these FST (section FST estimation based on the independent subpopulations model) and kinship (section Characterizing a kinship estimator and its relationship to FST) estimators under arbitrary population structures, where we find biases that are not present under their original assumptions about structure (panels “Indep. Subpop. FST Estimator” and “Existing Kinship Estimator” in Fig 1). We characterize the limit of the standard kinship estimator, identifying complex biases or distortions, in agreement with recent work [21, 62].

Fig 1
Our analysis is based on the generalized FST definition (section The generalized FST for arbitrary population structures) and two parallel models: the “Coancestry Model” for individual-specific allele frequencies (πij), and the “Kinship Model” for genotypes (xij). The “Coancestry in Terms of Kinship” panel connects kinship (${\phi }_{jk}^{T}$, ${f}_{j}^{T}$) and coancestry (${\theta }_{jk}^{T}$) parameters (section The kinship and coancestry models). We use these models to study the accuracy of FST and kinship method-of-moment estimators under arbitrary population structures. The “Indep. Subpop. FST Estimator” panel shows the bias resulting from the misapplication of FST estimators for independent subpopulations (${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$) to arbitrary structures (section FST estimation based on the independent subpopulations model), as calculated under the coancestry model. The “Existing Kinship Estimator” panel shows the bias in the standard kinship model estimator (${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$) and its resulting plug-in FST estimator (${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$; section Characterizing a kinship estimator and its relationship to FST), as calculated under the kinship model. The “New Kinship Estimator” panel presents a new statistic Ajk that estimates kinship with a uniform bias, which together with a consistent estimator of its minimum value (${\stackrel{^}{A}}_{\text{min}}$) results in our new kinship (${\stackrel{^}{\phi }}_{jk}^{T,\text{new}}$) and FST (${\stackrel{^}{F}}_{\text{ST}}^{\text{new}}$) estimators, which are consistent under arbitrary population structure (section A new approach for kinship and FST estimation).Accuracy of FST and kinship estimators: Overview of models and results.

In section A new approach for kinship and FST estimation we introduce a new approach for kinship and FST estimation for arbitrary population structures, and demonstrate the improved performance using a simple implementation of these estimators (panel “New Kinship Estimator” in Fig 1). There are two key innovations. First, based on the method of moments, we derive a statistic that estimates kinship coefficients up to a shared unknown scaling factor. Second, we propose a new condition, the identification of unrelated individual pairs in the data, which yields the value of the unknown scaling factor and enables the consistent estimation of kinship matrices and FST. We present a simple implementation of this second estimator, based on taking the minimum average statistic value between subpopulations, which in section Simulations evaluating FST and kinship estimators is shown to perform well under some misspecification, namely in an admixture scenario that does not actually have subpopulations [6365]. Elsewhere, we analyze the Human Origins and 1000 Genomes Project datasets with our novel kinship and FST estimation approach, where we demonstrate its coherence with the African Origins model, and illustrate the shortcomings of previous approaches in these complex data [59]. In summary, we identify a new approach for unbiased estimation of FST and kinship, and we provide new estimators that are nearly unbiased.

## Results

### The generalized FST for arbitrary population structures

The existing FST definition requires individuals to belong to discrete, non-overlapping subpopulations, so it must be generalized in order to apply to arbitrary population structures (such as the admixture model with individual-specific ancestry proportions considered in our simulations). Our generalized FST can be understood as a two-step strategy: (1) we define FST on a per-individual basis, and (2) we define FST for a group of individuals as a weighted average of the per-individual FST values [57].

The inbreeding coefficient ${f}_{j}^{T}$ of an individual j relative to an ancestral population T is defined as the probability that the two alleles at a random locus are identical by descent (IBD) [37]. Note that the ancestral population T determines what is IBD: only relationships since T count toward IBD. This total inbreeding coefficient (${f}_{j}^{T}$) is the individual analog of Wright’s total inbreeding coefficient FIT, the latter of which is the mean ${f}_{j}^{T}$ over a group of individuals [2]. Wright partitioned total inbreeding (FIT) into local (FIS) and structural (FST) coefficients defined by a subpopulation S that contains all individuals in question and evolved from the ancestral population T, so that FIS is the inbreeding of individuals relative to S (as opposed to T) and FST is inbreeding of the subpopulation S relative to T, and these coefficients satisfy (1 − FIT) = (1 − FIS)(1 − FST ) [2]. In our generalized definitions for one individual j, we restrict the subpopulation of interest (S) to be Lj, called the local subpopulation of j, which is the most recent subpopulation from which j drew its alleles. In this case, ${f}_{j}^{{L}_{j}}$ is the local inbreeding coefficient of j (always relative to its local subpopulation Lj), and ${f}_{{L}_{j}}^{T}$ is the structural inbreeding coefficient of j (equal to the inbreeding of the subpopulation Lj relative to T ), and being a special case of Wright’s equation, they also satisfy [57]

$\begin{array}{c}\hfill \left(1-{f}_{j}^{T}\right)=\left(1-{f}_{j}^{{L}_{j}}\right)\left(1-{f}_{{L}_{j}}^{T}\right).\end{array}$

Now we discuss estimating the three quantities we just introduced. First, the total inbreeding coefficient (${f}_{j}^{T}$) should be estimated from the variance of genotypes, using the practically unbiased approach we introduce in this work. Second, note that the local inbreeding coefficient (${f}_{j}^{{L}_{j}}$) corresponds to (non-population) family relatedness, so it can be taken to be the inbreeding calculated from a pedigree if it is available [42]. Note that estimation of the various inbreeding coefficients from pedigrees was the only approach available to Wright when he studied cattle and defined inbreeding and FST [2, 37]. Alternatively, in the absence of pedigrees, local inbreeding can be estimated from inferred self-IBD blocks or unusually-large runs of homozygosity [6668]. Lastly, since the structural inbreeding coefficient (${f}_{{L}_{j}}^{T}$) is given by the previous two quantities (solving from Eq (1)) by

$\begin{array}{c}\hfill {f}_{{L}_{j}}^{T}=\frac{{f}_{j}^{T}-{f}_{j}^{{L}_{j}}}{1-{f}_{j}^{{L}_{j}}},\end{array}$
then we propose estimating ${f}_{{L}_{j}}^{T}$ using this equation, from the above estimates of ${f}_{j}^{T}$ and ${f}_{j}^{{L}_{j}}$.

As a toy example, suppose we estimate a total inbreeding coefficient of ${f}_{j}^{T}=0.15$ for a given individual whose parents are first cousins, then the pedigree expectation for its local inbreeding is ${f}_{j}^{{L}_{j}}=\frac{1}{16}=0.0625$, and the structural inbreeding (i.e. the FST of this individual) using Eq (2) is ${f}_{{L}_{j}}^{T}\approx 0.093$. However, if in the same example (${f}_{j}^{T}=0.15$) the individual instead had parents who were second cousins, then ${f}_{j}^{{L}_{j}}=\frac{1}{64}\approx 0.0156$, then the structural estimate becomes ${f}_{{L}_{j}}^{T}\approx 0.137$, which is much closer to the total inbreeding value. Thus, when total inbreeding estimates are much larger than local inbreeding estimates, correcting for the latter via Eq (2) may not change the numerical estimate of structural inbreeding by a meaningful amount. Conversely, as the local inbreeding coefficient is reduced exponentially with the degree of relatedness of the parents (${f}_{j}^{{L}_{j}}=\frac{1}{{4}^{n+1}}$ for n-th cousins), and as local inbreeding is required to be recent (to exclude population-level inbreeding), then sufficiently-accurate estimates of structural inbreeding can be obtained by estimating non-zero local inbreeding only for individuals with the most related parent pairs (above a certain degree of relatedness).

We define the generalized FST across n individuals as the weighted average of the per-individual structural inbreeding coefficients (i.e., individual FST values) [57],

$\begin{array}{c}\hfill {F}_{\text{ST}}=\sum _{j=1}^{n}{w}_{j}{f}_{{L}_{j}}^{T},\end{array}$
where wj is the weight of individual j and the weights are required to sum to one and be non-negative. The above is a straightforward generalization of Wright’s FST: if every individual j has Lj = S as its local subpopulation, then Eq (3) becomes ${F}_{\text{ST}}={\sum }_{j=1}^{n}{w}_{j}{f}_{S}^{T}={f}_{S}^{T}$, where ${f}_{S}^{T}$ is the inbreeding coefficient of subpopulation S relative to T, so it has the same meaning as Wright’s FST (the exact weights here do not matter as long as ${\sum }_{j=1}^{n}{w}_{j}=1$, as required). Moreover, if each individual j belongs to one of K subpopulations Su (u ∈ {1, …, K}) and if subpopulations are weighted equally (${\sum }_{j\in {S}_{u}}{w}_{j}=\frac{1}{K}$ for every Su ), then Eq (3) becomes ${F}_{\text{ST}}=\frac{1}{K}{\sum }_{u=1}^{K}{f}_{{S}_{u}}^{T}$, so it equals the (unweighted) average subpopulation-specific FST (i.e., ${f}_{{S}_{u}}^{T}$), which is the FST definition for multiple subpopulations prevalent in modern work [4, 21, 23]. The last case illustrates the need for weights, which above downweights individuals that belong to subpopulations with greater numbers of observations. In general, weights allow adjustment for skewed or unbalanced samples. However, in complicated scenarios without subpopulations and no obvious sampling biases, for simplicity we recommend using uniform weights (${w}_{j}=\frac{1}{n}$) for the target generalized FST.

In terms of total and local inbreeding coefficients (using Eq (2)), the generalized FST equals

$\begin{array}{c}\hfill {F}_{\text{ST}}=\sum _{j=1}^{n}{w}_{j}\frac{{f}_{j}^{T}-{f}_{j}^{{L}_{j}}}{1-{f}_{j}^{{L}_{j}}},\end{array}$
which immediately suggests the estimation strategy when estimates of the total and local inbreeding coefficients are available. For simplicity, in the remainder of this work we shall consider only locally-outbred individuals (${f}_{j}^{{L}_{j}}=0$ for all j), for which the generalized FST simply equals the weighted mean total inbreeding coefficient:
$\begin{array}{c}\hfill {F}_{\text{ST}}=\sum _{j=1}^{n}{w}_{j}{f}_{j}^{T}.\end{array}$
This greatly simplifies our discussion of bias for all of the FST estimators we analyzed; determining the statistical properties of local inbreeding estimators is beyond the scope of this work. Moreover, the assumption of locally-outbred individuals is satisfied in all of the simulations presented in this work.

### The kinship and coancestry models

The generalized FST above is given solely in terms of inbreeding coefficients. In order to establish our results and framework, it is necessary to consider kinship coefficients as well. The kinship coefficient is the extension of the inbreeding coefficient for a pair of individuals: the kinship coefficient ${\phi }_{jk}^{T}$ of two individuals j and k relative to an ancestral population T is the probability that two alleles, chosen at random from each individual at a random locus, are IBD [1]. Note that the self-kinship coefficient is related to the inbreeding coefficient by ${\phi }_{jj}^{T}=\frac{1}{2}\left(1+{f}_{j}^{T}\right)$ [16].

Kinship coefficients determine the covariance structure of genotypes, which is the key to estimating kinship and FST from genotype data. We shall concentrate on biallelic variants, which include single-nucleotide polymorphisms, and are the dominant data from genotyping microarrays and whole-genome sequencing studies. We shall also restrict our attention to diploid organisms in this present work. Genotypes are encoded into variables xij for each locus i and individual j that count the number of alleles (dosage) of a given reference type, so for diploid organisms xij = 2 is homozygous for the reference allele, xij = 0 is homozygous for the alternative allele, and xij = 1 is heterozygous. Based on the definition of the IBD probabilities, the kinship model determines the mean and covariance structure of the genotype random variables at neutral loci [1, 2, 14, 16, 37]:

$\begin{array}{cc}\hfill \text{E}\left[{x}_{ij}|T\right]& =2{p}_{i}^{T},\hfill \\ \hfill \text{Cov}\left({x}_{ij},{x}_{ik}|T\right)& =4{p}_{i}^{T}\left(1-{p}_{i}^{T}\right){\phi }_{jk}^{T},\hfill \end{array}$
where ${p}_{i}^{T}$ is the allele frequency at locus i in the ancestral population T and $0<{p}_{i}^{T}<1$.

The coancestry model resembles the kinship model, but it is formulated in terms of allele frequencies, which simplifies our analysis of FST estimators for subpopulations as well as yielding kinship coefficients under the admixture model we simulate from in this work. Let πij be the individual-specific allele frequency (IAF) at locus i for individual j , which is a real number between zero and one [60, 61]. Our coancestry model assumes that [57]

$\begin{array}{c}\hfill \begin{array}{cc}\hfill E\left[{\pi }_{ij}|T\right]& ={p}_{i}^{T},\hfill \\ \hfill Cov\left({\pi }_{ij},{\pi }_{ik}|T\right)& ={p}_{i}^{T}\left(1-{p}_{i}^{T}\right){\theta }_{jk}^{T},\hfill \end{array}\end{array}$
where ${\theta }_{jk}^{T}$ is the coancestry coefficient between individuals j and k relative to the ancestral population T. This model is inspired by coancestry models for subpopulations common in the FST literature [4, 5, 21, 23], and exactly equals those models when subpopulation sizes go to infinity, in which case j and k index subpopulations rather than individuals, and πij is interpreted as the true allele frequency at locus i for subpopulation j.

The coancestry model connects to the kinship model under the additional assumption that the alleles of an individual j are drawn independently from its IAF,

$\begin{array}{c}\hfill {x}_{ij}|{\pi }_{ij}\sim \text{Binomial}\left(2,{\pi }_{ij}\right).\end{array}$
In this case, marginalizing the intermediate IAF random variables (πij ) and matching the resulting genotype moments results in the following equivalence [57]:
$\begin{array}{c}\hfill {\theta }_{jk}^{T}=\left\{\begin{array}{cc}{f}_{j}^{T}\hfill & \text{if}\phantom{\rule{1em}{0ex}}j=k,\hfill \\ {\phi }_{jk}^{T}\hfill & \text{if}\phantom{\rule{1em}{0ex}}j\ne k.\hfill \end{array}\end{array}$
The coancestry coefficient equals the kinship coefficient between two different individuals, but the self-coancestry coefficient equals the inbreeding coefficient (rather than the self-kinship coefficient). However, since in the coancestry model alleles are drawn independently conditional on the IAF in Eq (7), then the only structure present is the population structure, so these coancestry models cannot generate family structures, unlike the more general kinship model that also encompasses pedigrees. Therefore, despite Eq (8), the kinship and coancestry are not equivalent models except under the more restrictive assumptions of the coancestry model. Thus, individuals drawn from this model are always locally-outbred, so ${\theta }_{jj}^{T}={f}_{{L}_{j}}^{T}$ also equals the structural inbreeding coefficient, and the generalized FST under the coancestry model is therefore
$\begin{array}{c}\hfill {F}_{\text{ST}}=\sum _{j=1}^{n}{w}_{j}{\theta }_{jj}^{T},\end{array}$
which also generalizes previous definitions of FST under coancestry for subpopulations [4, 5, 21, 23]. The kinship and coancestry models, and their connection, is included in the overview Fig 1.

### Assessing the accuracy of genome-wide ratio estimators

In this section we change gears to focus on theoretical convergence properties of two broad estimator families. The resulting theory will be applied repeatedly to various FST and kinship estimators of interest in later sections.

Many FST and kinship coefficient method-of-moments estimators are ratio estimators , a general class of estimators that tends to be biased and to have no closed-form expectation [69]. In the FST literature, the expectation of a ratio is frequently approximated with a ratio of expectations [4, 17, 23]. Specifically, ratio estimators are often called “unbiased” if the ratio of expectations is unbiased, even though the ratio estimator itself may be biased [69]. Here we characterize the behavior of two ratio estimator families calculated from genome-wide data, known as “ratio-of-means” and “mean-of-ratios” estimators [23], detailing conditions where the previous approximation is justified and providing additional criteria to assess the accuracy of such estimators.

#### Ratio estimators

The general problem of forming ratio estimators involves random variables ai and bi calculated from genotypes at each locus i, such that E[ai] = Aci and E[bi] = Bci and the goal is to estimate $\frac{A}{B}$. A and B are constants shared across loci (given by FST or ${\phi }_{jk}^{T}$), while ci depends on the ancestral allele frequency ${p}_{i}^{T}$ and varies per locus. The problem is that the single-locus estimator $\frac{{a}_{i}}{{b}_{i}}$ is biased, since $E\left[\frac{{a}_{i}}{{b}_{i}}\right]\ne \frac{E\left[{a}_{i}\right]}{E\left[{b}_{i}\right]}=\frac{A}{B}$, which applies to ratio estimators in general [69]. Below we study two estimator families that combine large numbers of loci to better estimate $\frac{A}{B}$.

#### Convergence

The solution we recommend is the “ratio-of-means” estimator $\frac{{\stackrel{^}{A}}_{m}}{{\stackrel{^}{B}}_{m}}$, where ${\stackrel{^}{A}}_{m}=\frac{1}{m}{\sum }_{i=1}^{m}{a}_{i}$, and ${\stackrel{^}{B}}_{m}=\frac{1}{m}{\sum }_{i=1}^{m}{b}_{i}$, which is common for FST estimators [4, 17, 19, 23, 70]. Note that $E\left[{\stackrel{^}{A}}_{m}\right]=A{\overline{c}}_{m}$ and $E\left[{\stackrel{^}{B}}_{m}\right]=B{\overline{c}}_{m}$, where ${\overline{c}}_{m}=\frac{1}{m}{\sum }_{i=1}^{m}{c}_{i}$. We will assume bounded terms (|ai|, |bi| ≤ C for some finite C), a convergent ${\overline{c}}_{m}\to c$, and Bc ≠ 0, which are satisfied by common estimators. Given independent loci, we prove almost sure convergence to the desired quantity (S1 Text),

$\begin{array}{c}\frac{{\stackrel{^}{A}}_{m}}{{\stackrel{^}{B}}_{m}}=\frac{\frac{1}{m}\sum _{i=1}^{m}{a}_{i}}{\frac{1}{m}\sum _{i=1}^{m}{b}_{i}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{A}{B},\end{array}$
a strong result that implies $E\left[\frac{{\stackrel{^}{A}}_{m}}{{\stackrel{^}{B}}_{m}}\right]\to \frac{A}{B}$, justifying previous work [4, 17, 23]. Moreover, the error between these expectations scales with $\frac{1}{m}$ (S1 Text), just as for standard ratio estimators [69]. Although real loci are not independent due to genetic linkage, their dependence is very localized, so this estimator will perform well if the effective number of independent loci is large.

In order to test if a given ratio-of-means estimator converges to its ratio of expectations as in Eq (10), the following three conditions can be tested. (i) The expected values of each term ai, bi must be calculated and shown to be of the form E[ai] = Aci and E[bi] = Bci for some A and B shared by all loci i and some ci that may vary per locus i but must be shared by both E[ai], E[bi]. In the estimators we study, A and B are functions of IBD probabilities such as ${\phi }_{jk}^{T}$ and FST, while ci is a function of ${p}_{i}^{T}$ only. (ii) The mean ci must converge to a non-zero value for infinite loci. (iii) Both |ai|, |bi| ≤ C must be bounded for all i by some finite C (the estimators we study usually have C = 1 or C = 4). If these conditions are satisfied, then Eq (10) holds for independent loci and the A and B found in the first step. See the next section for an example application of this procedure to an FST estimator.

Another approach is the “mean-of-ratios” estimator $\frac{1}{m}{\sum }_{i=1}^{m}\frac{{a}_{i}}{{b}_{i}}$, used often to estimate kinship coefficients [16, 27, 3035] and FST [46]. If each $\frac{{a}_{i}}{{b}_{i}}$ is biased, their average across loci will also be biased, even as m → ∞. However, if $E\left[\frac{{a}_{i}}{{b}_{i}}\right]\to \frac{A}{B}$ for all loci i = 1, …, m as the number of individuals n → ∞, and $Var\left(\frac{{a}_{i}}{{b}_{i}}\right)$ is bounded, then

$\begin{array}{c}\hfill \frac{1}{m}\sum _{i=1}^{m}\frac{{a}_{i}}{{b}_{i}}\underset{n,m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{A}{B}.\end{array}$
Therefore, mean-of-ratios estimators must satisfy more restrictive conditions than ratio-of-means estimators, as well as large n (in addition to the large m needed by both estimators), to estimate $\frac{A}{B}$ well. We do not provide a procedure to test whether a given mean-of-ratios estimator converges as shown above.

### FST estimation based on the independent subpopulations model

Now that we have detailed how ratio estimators may be evaluated for their accuracy, we turn to existing estimators and assess their accuracy under arbitrary population structures. We study the FST estimators Weir-Cockerham (WC) [17], Weir-Hill [4], “Hudson” [23], and Weir-Goudet (equals HudsonK below for biallelic loci; S1 Text) [21]. The panel “Indep. Subpop. FST Estimator” in Fig 1 provides an overview of our results, which we detail in this section.

#### The FST estimator for independent subpopulations and infinite subpopulation sample sizes

The WC, Weir-Hill, and Hudson method-of-moments estimators have small sample size corrections that remarkably make them consistent (as the number of independent loci m goes to infinity) for finite numbers of individuals. However, these small sample corrections also make the estimators unnecessarily cumbersome for our purposes (see Methods, section Previous FST estimators for the independent subpopulations model for complete formulas). In order to illustrate clearly how these estimators behave, both under the independent subpopulations model and for arbitrary structure, here we construct simplified versions that assume infinite sample sizes per subpopulation (Methods, section Previous FST estimators for the independent subpopulations model ). This simplification corresponds to eliminating statistical sampling, leaving only genetic sampling to analyze [71]. Note that our simplified estimator nevertheless illustrates the general behavior of the WC, Weir-Hill, and Hudson estimators under arbitrary structure, and the results are equivalent to those we would obtain under finite sample sizes of individuals. While the Hudson FST estimator compares two subpopulations [23], based on that work we derive a generalized “HudsonK” estimator for more than two subpopulations in Methods, section Generalized HudsonK FST estimator . Note that HudsonK, first derived in [58], also equals the Weir-Goudet FST estimator for subpopulations [21] when loci are biallelic, which was derived independently using allele matching (S1 Text).

Under infinite subpopulation sample sizes, the allele frequencies at each locus and every subpopulation are known. Let j ∈ {1, …, n} index subpopulations rather than individuals and πij be the true allele frequency in subpopulation j at locus i. Note that πij are not estimated allele frequencies, but rather true subpopulation allele frequencies; this abstraction does not result in a practical estimation approach, but it greatly simplifies understanding of bias for subpopulations in a setting where there there is no statistical sampling. Although in this analysis of FST estimators the πij values are applied to subpopulations, for coherence with our previous work we shall call them “individual-specific allele frequencies” (IAF) [60, 61]. Whether for individuals or subpopulations, the key assumption is that IAFs satisfy the coancestry model of Eq (6). In this special case of infinite subpopulation sample sizes, all of WC, Weir-Hill, and HudsonK simplify to the following FST estimator for independent subpopulations (“indep”; derived in Methods, section Previous FST estimators for the independent subpopulations model):

${\stackrel{^}{p}}_{i}^{T}=\frac{1}{n}\sum _{j=1}^{n}{\pi }_{ij},$
${\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{n-1}\sum _{j=1}^{n}{\left({\pi }_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},$
${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}=\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}}.$
The goal is to estimate ${F}_{\text{ST}}=\frac{1}{n}{\sum }_{j=1}^{n}{\theta }_{jj}^{T}$, which is the special case of Eq (9) that weighs every subpopulation j equally (${w}_{j}=\frac{1}{n}\phantom{\rule{0.277778em}{0ex}}\forall j$).

#### FST estimation under the independent subpopulations model

Under the independent subpopulations model ${\theta }_{jk}^{T}=0$ for jk, where T is the most recent common ancestor (MRCA) population of the set of subpopulations. Note that the estimator in Eq (13) can be derived directly from Eq (6) and these assumptions using the method of moments (ignoring the existence of previous FST estimators; S1 Text). The expectations of the two recurrent terms in Eq (13) are

$\begin{array}{cc}\hfill E\left[\frac{1}{m}\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}|T\right]& ={\overline{p\left(1-p\right)}}^{T}{F}_{\text{ST}},\hfill \\ \hfill E\left[\frac{1}{m}\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)|T\right]& ={\overline{p\left(1-p\right)}}^{T}\left(1-\frac{{F}_{\text{ST}}}{n}\right),\phantom{\rule{1em}{0ex}}\text{where}\hfill \\ \hfill {\overline{p\left(1-p\right)}}^{T}& =\frac{1}{m}\sum _{i=1}^{m}{p}_{i}^{T}\left(1-{p}_{i}^{T}\right).\hfill \end{array}$
Eliminating ${\overline{p\left(1-p\right)}}^{T}$ and solving for FST in this system of equations recovers the estimator in Eq (13).

Before applying the convergence result in Eq (10), we test that the three conditions listed in section Assessing the accuracy of genome-wide ratio estimators are met. Condition (i): The locus i terms are ${a}_{i}={\stackrel{^}{\sigma }}_{i}^{2}$ and ${b}_{i}={\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}$, which satisfy E[ai] = Aci and E[bi] = Bci with A = FST, B = 1, and ${c}_{i}={p}_{i}^{T}\left(1-{p}_{i}^{T}\right)$. Condition (ii): ${\overline{c}}_{m}\to c=E\left[{p}_{i}^{T}\left(1-{p}_{i}^{T}\right)|T\right]\ne 0$ over the ${p}_{i}^{T}$ distribution across loci. Condition (iii): Since $0\le {\pi }_{ij},{\stackrel{^}{p}}_{i}^{T}\le 1$, then $0\le {\stackrel{^}{\sigma }}_{i}^{2}\le 1$ and $0\le {\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)\le \frac{1}{4}$, and since n ≥ 2, C = 1 bounds both |ai| and |bi|. Therefore, for independent loci,

$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}.\end{array}$

#### FST estimation under arbitrary coancestry

Now we consider applying the independent subpopulations FST estimator to dependent subpopulations. The key difference is that now ${\theta }_{jk}^{T}\ne 0$ for every (j, k ) will be assumed in our coancestry model in Eq (6), and now T may be either the MRCA population of all subpopulations or a more ancestral population. In this general setting, (j, k) may index either subpopulations or individuals. The two terms of ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ now satisfy

$\begin{array}{cc}\hfill E\left[\frac{1}{m}\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}|T\right]& ={\overline{p\left(1-p\right)}}^{T}\left({F}_{\text{ST}}-{\overline{\theta }}^{T}\right)\frac{n}{n-1},\hfill \\ \hfill E\left[\frac{1}{m}\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)|T\right]& ={\overline{p\left(1-p\right)}}^{T}\left(1-{\overline{\theta }}^{T}\right),\hfill \end{array}$
where ${\overline{\theta }}^{T}=\frac{1}{{n}^{2}}{\sum }_{j=1}^{n}{\sum }_{k=1}^{n}{\theta }_{jk}^{T}$ is the mean coancestry with uniform weights. There are two equations but three unknowns: FST, ${\overline{\theta }}^{T}$, and ${\overline{p\left(1-p\right)}}^{T}$. The independent subpopulations model satisfies ${\overline{\theta }}^{T}=\frac{1}{n}{F}_{\text{ST}}$, which allows for the consistent estimation of FST. Therefore, the new unknown ${\overline{\theta }}^{T}$ precludes consistent FST estimation without additional assumptions. As shown later, our additional assumption is that we can identify unrelated individuals in the data, which determines all unknowns. We defer our complete solution to this problem until kinship and its estimation challenges have been presented.

The FST estimator for independent subpopulations converges more generally to

${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{F}_{\text{ST}}-{\stackrel{˜}{\theta }}^{T}}{1-{\stackrel{˜}{\theta }}^{T}},$
(the conclusion of panel “Indep. Subpop. FST Estimator” in Fig 1), where
$\begin{array}{c}\hfill {\stackrel{˜}{\theta }}^{T}=\frac{1}{n-1}\left(n{\overline{\theta }}^{T}-{F}_{\text{ST}}\right)=\frac{1}{n\left(n-1\right)}\sum _{j\ne k}{\theta }_{jk}^{T}\end{array}$
is the average of all between-subpopulation coancestry coefficients, in agreement with related calculations regarding the WC and Weir-Hill estimators [4, 21]. Therefore, under arbitrary structure the independent subpopulations estimator’s bias is due to the coancestry between subpopulations. While the limit in Eq (14) appears to vary depending on the choice of T, it is in fact a constant with respect to T (proof in S1 Text).

Since $\frac{1}{n}{F}_{\text{ST}}\le {\overline{\theta }}^{T}\le {F}_{\text{ST}}$ (S1 Text), this estimator has a downward bias in the general setting: it is asymptotically unbiased (${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}$) only when ${\overline{\theta }}^{T}=\frac{1}{n}{F}_{\text{ST}}$, while bias is maximal when ${\overline{\theta }}^{T}={F}_{\text{ST}}$, where ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}0$. For example, if $min\phantom{\rule{4pt}{0ex}}{\theta }_{jk}^{T}=0$ so the MRCA population T is fixed, but n is large and ${\theta }_{jk}^{T}\approx {F}_{\text{ST}}$ for most pairs of subpopulations, then ${\overline{\theta }}^{T}\approx {F}_{\text{ST}}$ as well, and ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}\approx 0$. Therefore, the magnitude of the bias of ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ is unknown if ${\overline{\theta }}^{T}$ is unknown, and small ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ estimates may arise even if FST is very large.

#### Coancestry estimation as a method of moments

Since the generalized FST is given by coancestry coefficients ${\theta }_{jj}^{T}$ in Eq (9), a new FST estimator could be derived from estimates of ${\theta }_{jj}^{T}$. Here we attempt to define a method-of-moments estimator for ${\theta }_{jk}^{T}$, and find an underdetermined estimation problem, just as for FST . This is consistent with IBD parameters in general requiring a reference population to be determined [40], whereas in this subsection this reference population is unspecified.

Given IAFs and the coancestry model of Eq (6), the first and second moments that average across loci are

$E\left[\frac{1}{m}\sum _{i=1}^{m}{\pi }_{ij}|T\right]={\overline{p}}^{T},$
$E\left[\frac{1}{m}\sum _{i=1}^{m}{\pi }_{ij}{\pi }_{ik}|T\right]={\overline{{p}^{2}}}^{T}+{\overline{p\left(1-p\right)}}^{T}{\theta }_{jk}^{T},$
where ${\overline{p}}^{T}=\frac{1}{m}{\sum }_{i=1}^{m}{p}_{i}^{T}$, ${\overline{{p}^{2}}}^{T}=\frac{1}{m}{\sum }_{i=1}^{m}{\left({p}_{i}^{T}\right)}^{2}$, and ${\overline{p\left(1-p\right)}}^{T}$ is as before.

Suppose first that only ${\theta }_{jj}^{T}$ are of interest. There are n estimators given by Eq (16) with j = k, each corresponding to an unknown ${\theta }_{jj}^{T}$. However, all these estimators share two nuisance parameters: ${\overline{p}}^{T}$ and ${\overline{{p}^{2}}}^{T}$. While ${\overline{p}}^{T}$ can be estimated from Eq (15), there are no more equations left to estimate ${\overline{{p}^{2}}}^{T}$, so this system is underdetermined. The estimation problem remains underdetermined if all $\frac{n\left(n+1\right)}{2}$ estimators in Eq (16) are considered rather than only the j = k cases. Therefore, we cannot estimate coancestry coefficients consistently using only the first two moments without additional assumptions.

### Characterizing a kinship estimator and its relationship to FST

Given the biases we see for ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ under arbitrary structures in the previous section, we now turn to the generalized definition of FST and pursue an estimate of it. Recall that our generalized FST in Eq (3) is defined in terms of inbreeding coefficients, which are a special case of the kinship coefficient. Kinship coefficients also determine the bias of ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eq (14) (since coancestry and kinship coefficients are closely related: see panel “Coancestry in Terms of Kinship” in Fig 1). Therefore, we will consider estimates of kinship and inbreeding in this section. Estimating kinship is also important for GWAS approaches that control for population structure [16, 2435, 72, 73].

In this section, we focus on a standard kinship method-of-moments estimator and calculate its limit for the first time (panel “Existing Kinship Estimator” in Fig 1). We study estimators that use genotypes or IAFs, and construct FST estimators from their kinship estimates. We find biases comparable to those of ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in the previous section, and define unbiased FST estimators that require knowing the mean kinship or coancestry, or its proportion relative to FST. The results of this section directly motivate and help construct our new kinship and FST estimation approach in the following section.

#### Characterization of the standard kinship estimator

Here we analyze a standard kinship estimator that is frequently used [16, 27, 3036]. We generalize this estimator to use weights in estimating the ancestral allele frequencies, and we write it as a ratio-of-means estimator due to the favorable theoretical properties of this format as detailed in the earlier section Assessing the accuracy of genome-wide ratio estimators:

${\stackrel{^}{p}}_{i}^{T}=\frac{1}{2}\sum _{j=1}^{n}{w}_{j}{x}_{ij},$
${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}=\frac{\sum _{i=1}^{m}\left({x}_{ij}-2{\stackrel{^}{p}}_{i}^{T}\right)\left({x}_{ik}-2{\stackrel{^}{p}}_{i}^{T}\right)}{4\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)}.$
The estimator in Eq (18) resembles the sample covariance estimator applied to genotypes, but centers by locus i rather than by individuals j and k, and normalizes using estimates of $4{p}_{i}^{T}\left(1-{p}_{i}^{T}\right)$. We derive Eq (18) directly using the method of moments in S1 Text. The weights in Eq (17) must satisfy wj > 0 and ${\sum }_{j=1}^{n}{w}_{j}=1$, so that $0\le {\stackrel{^}{p}}_{i}^{T}\le 1$ and $E\left[{\stackrel{^}{p}}_{i}^{T}|T\right]={p}_{i}^{T}$.

Utilizing the kinship model for genotypes from Eq (5), we find that Eq (18) converges to

$\begin{array}{c}\hfill {\stackrel{^}{\phi }}_{jk}^{T,\text{std}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{\phi }_{jk}^{T}-{\overline{\phi }}_{j}^{T}-{\overline{\phi }}_{k}^{T}+{\overline{\phi }}^{T}}{1-{\overline{\phi }}^{T}},\end{array}$
where ${\overline{\phi }}_{j}^{T}={\sum }_{{k}^{\prime }=1}^{n}{w}_{{k}^{\prime }}{\phi }_{j{k}^{\prime }}^{T}$ and ${\overline{\phi }}^{T}={\sum }_{{j}^{\prime }=1}^{n}{\sum }_{{k}^{\prime }=1}^{n}{w}_{{j}^{\prime }}{w}_{{k}^{\prime }}{\phi }_{{j}^{\prime }{k}^{\prime }}^{T}$, which agrees with related derivations [21, 62]. (This is the conclusion of panel “Existing Kinship Estimator” in Fig 1; see S1 Text for intermediate calculations that lead to Eq (19).) Therefore, the bias of ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ varies per pair of individuals j and k . Analogous distortions have been observed for sample covariances of genotypes [74]. The limit of ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ in Eq (19) is constant with respect to T (proof in S1 Text). Similarly, inbreeding coefficient estimates derived from Eq (18) converge to
$\begin{array}{c}\hfill {\stackrel{^}{f}}_{j}^{T,\text{std}}=2{\stackrel{^}{\phi }}_{jj}^{T}-1\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{f}_{j}^{T}-4{\overline{\phi }}_{j}^{T}+3{\overline{\phi }}^{T}}{1-{\overline{\phi }}^{T}}.\end{array}$
The difference between the bias of ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ for jk in Eq (19) and ${\stackrel{^}{f}}_{j}^{T,\text{std}}$ in Eq (20) is visible in the kinship estimates shown toward the end of the results section. The limits of the ratio-of-means versions of two more ${f}_{j}^{T}$ estimators [32] are, if ${\stackrel{^}{p}}_{i}^{T}$ uses Eq (17),
$\begin{array}{c}\begin{array}{cc}{\stackrel{^}{f}}_{j}^{T,\text{stdII}}& =1-\frac{\sum _{i=1}^{m}{x}_{ij}\left(2-{x}_{ij}\right)}{2\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{f}_{j}^{T}-{\overline{\phi }}^{T}}{1-{\overline{\phi }}^{T}},\hfill \\ {\stackrel{^}{f}}_{j}^{T,\text{stdIII}}& =\frac{\sum _{i=1}^{m}{x}_{ij}^{2}-\left(1+2{\stackrel{^}{p}}_{i}^{T}\right){x}_{ij}+2{\left({\stackrel{^}{p}}_{i}^{T}\right)}^{2}}{2\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{f}_{j}^{T}+{\overline{\phi }}^{T}-2{\overline{\phi }}_{j}^{T}}{1-{\overline{\phi }}^{T}}.\hfill \end{array}\end{array}$

The estimators in Eqs (18) and (21) are unbiased when ${\stackrel{^}{p}}_{i}^{T}$ is replaced by ${p}_{i}^{T}$ [16, 32, 36], and are consistent when ${\stackrel{^}{p}}_{i}^{T}$ is consistent [60]. Surprisingly, ${\stackrel{^}{p}}_{i}^{T}$ in Eq (17) is not consistent (it does not converge almost surely to ${p}_{i}^{T}$) for arbitrary population structures, which is at the root of the bias in Eqs (19) to (21). In particular, although ${\stackrel{^}{p}}_{i}^{T}$ is unbiased, its variance (S1 Text, and some special cases shown elsewhere, e.g. , [19]),

$\begin{array}{c}\hfill Var\left({\stackrel{^}{p}}_{i}^{T}|T\right)={p}_{i}^{T}\left(1-{p}_{i}^{T}\right){\overline{\phi }}^{T},\end{array}$
may be asymptotically non-zero as n → ∞, since ${p}_{i}^{T}\in \left(0,1\right)$ is fixed and ${lim}_{n\to \infty }{\overline{\phi }}^{T}$ may take on any value between zero and one for arbitrary population structures. Further, ${\overline{\phi }}^{T}\to 0$ as n → ∞ if and only if ${\phi }_{jk}^{T}=0$ for almost all pairs of individuals (j, k). These observations hold for any weights such that ${w}_{j}>0,{\sum }_{j=1}^{n}{w}_{j}=1$. An important consequence is that the plug-in estimate of ${p}_{i}^{T}\left(1-{p}_{i}^{T}\right)$ is biased (S1 Text),
$\begin{array}{c}\hfill E\left[{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)|T\right]={p}_{i}^{T}\left(1-{p}_{i}^{T}\right)\left(1-{\overline{\phi }}^{T}\right),\end{array}$
which is present in all estimators we have studied.

#### Estimation of coancestry coefficients from IAFs

Here we form a coancestry coefficient estimator analogous to Eq (18) but using IAFs. Assuming the moments in Eq (6), this estimator and its limit are

${\stackrel{^}{p}}_{i}^{T}=\sum _{j=1}^{n}{w}_{j}{\pi }_{ij},$
${\stackrel{^}{\theta }}_{jk}^{T,\text{std}}=\frac{\sum _{i=1}^{m}\left({\pi }_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)\left({\pi }_{ik}-{\stackrel{^}{p}}_{i}^{T}\right)}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{\theta }_{jk}^{T}-{\overline{\theta }}_{j}^{T}-{\overline{\theta }}_{k}^{T}+{\overline{\theta }}^{T}}{1-{\overline{\theta }}^{T}},$
where ${\overline{\theta }}_{j}^{T}={\sum }_{k=1}^{n}{w}_{k}{\theta }_{jk}^{T}$ and ${\overline{\theta }}^{T}={\sum }_{j=1}^{n}{\sum }_{k=1}^{n}{w}_{j}{w}_{k}{\theta }_{jk}^{T}$ are analogous to ${\overline{\phi }}_{j}^{T}$ and ${\overline{\phi }}^{T}$. Eq (23) generalizes Eq (11) for arbitrary weights. Thus, use of IAFs does not ameliorate the estimation problems we have identified for genotypes. Like Eq (22), ${\stackrel{^}{p}}_{i}^{T}$ in Eq (23) is not consistent because $\text{Var}\left({\stackrel{^}{p}}_{i}^{T}|T\right)={p}_{i}^{T}\left(1-{p}_{i}^{T}\right){\overline{\theta }}^{T}$ may not converge to zero for arbitrary population structures, which causes the bias observed in Eq (24).

#### FST estimator based on the standard kinship estimator

Since the generalized FST is defined as a mean inbreeding coefficient in Eq (3), here we study the FST estimator constructed as ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}={\sum }_{j=1}^{n}{w}_{j}{\stackrel{^}{f}}_{j}^{T,\text{std}}$ where ${\stackrel{^}{f}}_{j}^{T,\text{std}}$ is the inbreeding estimator derived from the standard kinship estimator. Although ${\stackrel{^}{f}}_{j}^{T,\text{std}}$ is biased, we nevertheless plug it into our definition of FST so that we may study how bias manifests. Note that we do not recommend utilizing this FST estimator in practice, but we find these results informative for identifying how to proceed in deriving new estimators in the following section.

Remarkably, the three ${f}_{j}^{T}$ estimators in Eqs (20) and (21) give exactly the same plug-in ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ if the weights in FST and ${\stackrel{^}{p}}_{i}^{T}$ in Eq (17) match, namely

$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{std}}=\sum _{j=1}^{n}{w}_{j}{\stackrel{^}{f}}_{j}^{T,\text{std}}=\frac{\sum _{i=1}^{m}\sum _{j=1}^{n}{w}_{j}{\left({x}_{ij}-2{\stackrel{^}{p}}_{i}^{T}\right)}^{2}}{2\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)}-1\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{F}_{\text{ST}}-{\overline{\phi }}^{T}}{1-{\overline{\phi }}^{T}},\end{array}$
where the limit assumes locally-outbred individuals so Eq (4) holds. The analogous FST estimator for IAFs and its limit are
$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{std}}=\sum _{j=1}^{n}{w}_{j}{\stackrel{^}{\theta }}_{jj}^{T,\text{std}}=\frac{\sum _{i=1}^{m}\sum _{j=1}^{n}{w}_{j}{\left({\pi }_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2}}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{F}_{\text{ST}}-{\overline{\theta }}^{T}}{1-{\overline{\theta }}^{T}}.\end{array}$
The estimators in Eqs (25) and (26) for individuals and their limits resemble those of classical FST estimators for populations of the form $\frac{{\sigma }_{p}^{2}}{\overline{p}\left(1-\overline{p}\right)}$ [4, 5]. ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (26) for subpopulations j with uniform weight and one locus is also GST for two alleles [75]. Compared to ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eq (13), ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (26) admits arbitrary weights and, by forgoing bias correction under the independent subpopulations model, is a simpler target of study.

Like ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eq (13), ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eqs (25) and (26) are downwardly biased since $0\le {\overline{\phi }}^{T},{\overline{\theta }}^{T}$. ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (26) may converge arbitrarily close to zero since ${\overline{\theta }}^{T}$ can be arbitrarily close to FST (S1 Text). Moreover, although ${\overline{\phi }}^{T}\approx {\overline{\theta }}^{T}$ for large n (see Eq (8) and panel “Coancestry in Terms of Kinship” in Fig 1), in extreme cases ${\overline{\phi }}^{T}$ can exceed FST under the coancestry model (where ${\overline{\theta }}^{T}\le {\overline{\phi }}^{T}$) and also under extreme local kinship, where ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (25) converges to a negative value.

#### Adjusted consistent oracle FST estimators and the “bias coefficient”

Here we explore two adjustments to ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ from IAFs in Eq (26) that rely on having minimal additional information needed to correct its bias. These “oracle” approaches require information that is not known in practice, but this exercise helps us understand the problem more deeply and finds further connections between the various FST estimators.

If ${\overline{\theta }}^{T}$ is known, the bias in Eq (26) can be reversed, yielding the consistent estimator

$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\prime }={\stackrel{^}{F}}_{\text{ST}}^{\text{std}}\left(1-{\overline{\theta }}^{T}\right)+{\overline{\theta }}^{T}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}.\end{array}$
Consistent estimates are also possible if a scaled version of ${\overline{\theta }}^{T}$ is known, namely
$\begin{array}{c}\hfill {s}^{T}=\frac{{\overline{\theta }}^{T}}{{F}_{\text{ST}}}=\frac{\sum _{j=1}^{n}\sum _{k=1}^{n}{w}_{j}{w}_{k}{\theta }_{jk}^{T}}{\sum _{j=1}^{n}{w}_{j}{\theta }_{jj}^{T}},\end{array}$
which we call the “bias coefficient” and which has interesting properties. The bias coefficient quantifies the departure from the independent subpopulations model by comparing the mean coancestry (${\theta }_{jk}^{T}$) to the mean inbreeding coefficient (${\theta }_{jj}^{T}$), and given FST > 0 satisfies 0 < sT ≤ 1 (S1 Text). The limit in Eq (26) in terms of sT is
$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{std}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}\frac{1-{s}^{T}}{1-{s}^{T}{F}_{\text{ST}}}.\end{array}$
Treating the limit as equality and solving for FST yields the following consistent estimator:
${\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{1-{s}^{T}}\sum _{j=1}^{n}{w}_{j}{\left({\pi }_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},$
${\stackrel{^}{F}}_{\text{ST}}^{\prime \prime }=\frac{{\stackrel{^}{F}}_{\text{ST}}^{\text{std}}}{1-{s}^{T}\left(1-{\stackrel{^}{F}}_{\text{ST}}^{\text{std}}\right)}=\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)+{s}^{T}{\stackrel{^}{\sigma }}_{i}^{2}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}.$
Note that ${\stackrel{^}{\sigma }}_{i}^{2}$ and ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ from Eqs (12) and (13) are the special case of Eqs (30) and (31) for uniform weights and ${s}^{T}=\frac{1}{n}$; hence, ${\stackrel{^}{F}}_{\text{ST}}^{\prime \prime }$ generalizes ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$.

Lastly, using either Eqs (26) or (29), the relative error of ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ converges to

$\begin{array}{c}\hfill 1-\frac{{\stackrel{^}{F}}_{\text{ST}}^{\text{std}}}{{F}_{\text{ST}}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{\overline{\theta }}^{T}\left(1-{F}_{\text{ST}}\right)}{{F}_{\text{ST}}\left(1-{\overline{\theta }}^{T}\right)}={s}^{T}\frac{1-{F}_{\text{ST}}}{1-{s}^{T}{F}_{\text{ST}}},\end{array}$
which is approximated by sT if FST ≪ 1, hence the name “bias coefficient”. Note sT varies depending on the choice of T, which is necessary since FST (and hence the relative bias of ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ from FST) depends on the choice of T.

### A new approach for kinship and FST estimation

Here, we propose a new estimation approach for kinship coefficients that has properties favorable for obtaining nearly unbiased estimates (panel “New Kinship Estimator” in Fig 1). These new kinship estimates yield an improved FST estimator. We present the general approach and implement a simple version of one key estimator that results in the complete proof-of-principle estimator that is evaluated in the next section and applied to human data in [59]. We also compare our approach to a related estimator of non-IBD linearly-transformed kinship values [2022] that was proposed concurrently to ours [58].

#### General approach

In this subsection we develop our new estimator in two steps. First, we compute a new statistic Ajk that is proportional in the limit of infinite loci to ${\phi }_{jk}^{T}-1$ times a nuisance factor vT. Second, we estimate and remove vT to yield the proposed estimator ${\stackrel{^}{\phi }}_{jk}^{T,\text{new}}$. ${\stackrel{^}{A}}_{\text{min}}$—an estimator of the limit of the minimum Ajk—yields vT if the least related pair of individuals in the data has ${\phi }_{jk}^{T}=0$, which sets T to the MRCA population of all the individuals in the data. The new kinship estimator immediately results in new inbreeding (${\stackrel{^}{f}}_{j}^{T,\text{new}}$) and FST (${\stackrel{^}{F}}_{\text{ST}}^{\text{new}}$) estimators. This general approach leaves the implementation of ${\stackrel{^}{A}}_{\text{min}}$ open; the simple implementation applied in this work is described in subsection Proof-of-principle kinship estimator using subpopulation labels, but our method can be readily improved by substituting in a better ${\stackrel{^}{A}}_{\text{min}}$ in the future.

Applying the method of moments to Eq (5), we derive the following statistic (S1 Text), whose expectation is proportional to ${\phi }_{jk}^{T}-1$:

$\begin{array}{c}\hfill \begin{array}{cc}\hfill {A}_{jk}& =\frac{1}{m}\sum _{i=1}^{m}\left({x}_{ij}-1\right)\left({x}_{ik}-1\right)-1,\hfill \\ \hfill E\left[{A}_{jk}|T\right]& =\left({\phi }_{jk}^{T}-1\right){v}_{m}^{T},\phantom{\rule{1em}{0ex}}\text{where}\hfill \\ \hfill {v}_{m}^{T}& =\frac{4}{m}\sum _{i=1}^{m}{p}_{i}^{T}\left(1-{p}_{i}^{T}\right).\hfill \end{array}\end{array}$
Compared to the standard kinship estimator in Eq (19), which has a complex asymptotic bias determined by n parameters (${\overline{\phi }}_{j}^{T}$ for each j ∈ {1, …, n}), the Ajk statistics estimate kinship with a bias controlled by the sole unknown parameter ${v}_{m}^{T}$ shared by all pairs of individuals. The key to estimating ${v}_{m}^{T}$ is to notice that if ${\phi }_{jk}^{T}=0$ then $E\left[{A}_{jk}|T\right]=-{v}_{m}^{T}$. Thus, assuming ${min}_{j,k}{\phi }_{jk}^{T}=0$, which sets T to the MRCA population, then the minimum Ajk yields the nuisance parameter. However, we recommend using a more stable estimate than the minimum Ajk to unbias all Ajk, such as the estimator presented in the next subsection.

In general, suppose ${\stackrel{^}{A}}_{\text{min}}$ is a consistent estimator of the limit of the minimum E[Ajk|T], or equivalently,

$\begin{array}{c}\hfill {\stackrel{^}{A}}_{\text{min}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}-{v}^{T},\end{array}$
along with the assumption that ${v}_{m}^{T}\underset{m\to \infty }{\to }{v}^{T}$ for some vT ≠ 0. Our new kinship estimator follows directly from replacing ${v}_{m}^{T}$ with $-{\stackrel{^}{A}}_{\text{min}}$ and solving for ${\phi }_{jk}^{T}$ in Eq (33), which results in a consistent kinship estimator (given the convergence proof of section Assessing the accuracy of genome-wide ratio estimators):
$\begin{array}{c}\hfill {\stackrel{^}{\phi }}_{jk}^{T,\text{new}}=1-\frac{{A}_{jk}}{{\stackrel{^}{A}}_{\text{min}}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{\phi }_{jk}^{T}.\end{array}$
The resulting new inbreeding coefficient estimator is
$\begin{array}{c}\hfill {\stackrel{^}{f}}_{j}^{T,\text{new}}=2{\stackrel{^}{\phi }}_{jj}^{T,\text{new}}-1\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{f}_{j}^{T},\end{array}$
and the new FST estimator is consistent for locally-outbred individuals (estimates Eq (4)):
$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{new}}=\sum _{j=1}^{n}{w}_{j}{\stackrel{^}{f}}_{j}^{T,\text{new}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}.\end{array}$
Thus, only the implementation of ${\stackrel{^}{A}}_{\text{min}}$ is left unspecified from this general estimation approach of kinship and FST. The implementation of ${\stackrel{^}{A}}_{\text{min}}$ used in the analyses in this work is given in the next subsection.

#### Proof-of-principle kinship estimator using subpopulation labels

To showcase the potential of the new estimators, we implement a simple proof-of-principle version of ${\stackrel{^}{A}}_{\text{min}}$ needed for our new kinship estimator (${\stackrel{^}{\phi }}_{jk}^{T,\text{new}}$ in Eq (34)). This ${\stackrel{^}{A}}_{\text{min}}$ relies on an appropriate partition of the n individuals into K subpopulations (denoted Su for u ∈ {1, …, K}), where the only requirement is that the kinship coefficients between pairs of individuals across the two most unrelated subpopulations is zero, as detailed below. Note that, unlike the the independent subpopulations model of section FST estimation based on the independent subpopulations model, these K subpopulations need not be independent nor unstructured. The desired estimator ${\stackrel{^}{A}}_{\text{min}}$ is the minimum average Ajk over all subpopulation pairs:

$\begin{array}{c}\hfill {\stackrel{^}{A}}_{\text{min}}=\underset{u\ne v}{min}\frac{1}{|{S}_{u}||{S}_{v}|}\sum _{j\in {S}_{u}}\sum _{k\in {S}_{v}}{A}_{jk}.\end{array}$
This ${\stackrel{^}{A}}_{\text{min}}$ consistently estimates the limit of the minimum Ajk if ${\phi }_{jk}^{T}=0\phantom{\rule{3.33333pt}{0ex}}\forall j\in {S}_{u},\forall k\in {S}_{v}$ for the least related pair of subpopulations Su, Sv.

This estimator should work well for individuals truly divided into subpopulations, but may be biased for a poor choice of subpopulations, in particular if the minimum mean ${\phi }_{jk}^{T}$ between subpopulations is far greater than zero. For this reason, inspection of the kinship estimates is required and careful construction of appropriate subpopulations may be needed. See our analysis of human data for detailed examples [59]. Future work could focus on a more general ${\stackrel{^}{A}}_{\text{min}}$ that circumvents the need for subpopulations of our proof-of-principle estimator.

#### Comparison to the Weir-Goudet kinship estimator for individuals

Here we analyze the Weir-Goudet (WG) kinship estimator for individuals [2022]. This has connections to our new estimator but differs in having the goal of estimating linearly-transformed kinship values. In our framework, the WG estimator is given by

$\begin{array}{c}\hfill {\stackrel{^}{\phi }}_{jk}^{T,\text{WG}}=1-\frac{{A}_{jk}}{{\stackrel{^}{A}}_{\text{avg}}},\phantom{\rule{1em}{0ex}}\text{where}\phantom{\rule{1em}{0ex}}{\stackrel{^}{A}}_{\text{avg}}=\frac{2}{n\left(n-1\right)}\sum _{j=2}^{n}\sum _{k=1}^{j-1}{A}_{jk}.\end{array}$
Therefore, this estimator differs from our proposal [58] by replacing our ${\stackrel{^}{A}}_{\text{min}}$ with ${\stackrel{^}{A}}_{\text{avg}}$. Under the kinship model, the expectation of ${\stackrel{^}{A}}_{\text{avg}}$ is
$\begin{array}{c}\hfill E\left[{\stackrel{^}{A}}_{\text{avg}}|T\right]=\left({\stackrel{˜}{\phi }}^{T}-1\right){v}_{m}^{T},\phantom{\rule{1em}{0ex}}\text{where}\phantom{\rule{1em}{0ex}}{\stackrel{˜}{\phi }}^{T}=\frac{2}{n\left(n-1\right)}\sum _{j=2}^{n}\sum _{k=1}^{j-1}{\phi }_{jk}^{T}.\end{array}$
Therefore, the limit of this estimator is
${\stackrel{^}{\phi }}_{jk}^{T,\text{WG}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{\phi }_{jk}^{T}-{\stackrel{˜}{\phi }}^{T}}{1-{\stackrel{˜}{\phi }}^{T}},$
which agrees with calculations in the original WG work [2022]. Note that, assuming that kinship coefficients must be non-negative, the above estimator recovers the kinship IBD probabilities if and only if ${\stackrel{˜}{\phi }}^{T}=0$ which holds if and only if ${\phi }_{jk}^{T}=0$ for every pair of individuals jk. The resulting WG inbreeding coefficient estimator is
${\stackrel{^}{f}}_{jk}^{T,\text{WG}}=2{\stackrel{^}{\phi }}_{jk}^{T,\text{WG}}-1\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{f}_{j}^{T}-{\stackrel{˜}{\phi }}^{T}}{1-{\stackrel{˜}{\phi }}^{T}},$
which estimates linearly-transformed inbreeding values [21]. Therefore, the resulting WG FST estimator (for individuals) also targets a linearly-transformed FST value (under locally-outbred individuals, where FST is given by Eq (4)), namely
${\stackrel{^}{F}}_{\text{ST}}^{\text{WG}}=\frac{1}{n}\sum _{j=1}^{n}{\stackrel{^}{f}}_{j}^{T,\text{WG}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{F}_{\text{ST}}-{\stackrel{˜}{\phi }}^{T}}{1-{\stackrel{˜}{\phi }}^{T}}.$
The WG authors also briefly consider a variant of their kinship estimator that is normalized using the minimum kinship value as we did, developed concurrently with our approach [58], but was largely dismissed as an unnecessary correction [21, 76]. See S1 Text for a detailed proof that the general estimator framework we propose here (Eqs (33) and (34)) is algebraically equivalent to our original formulation in [58].

Note that the original WG does not estimate FST from individuals as considered above; instead, FST is estimated from coancestry estimates for subpopulations (which equals our HudsonK for biallelic loci, S1 Text) [2022]. For completeness, we consider both kinds of FST estimates in the evaluations that follow.

### Simulations evaluating FST and kinship estimators

#### Overview of simulations

We simulate genotypes from two models to illustrate our results when the true population structure parameters are known. Both simulations have clearly-defined IBD probability parameters in terms of the MRCA population. The first simulation satisfies the independent subpopulations model that existing FST estimators assume. The second simulation is from an admixture model with no independent subpopulations and pervasive kinship designed to induce large downward biases in existing kinship and FST estimators (Fig 2). This admixture scenario resembles the population structure we estimated for Hispanics in the 1000 Genomes Project [59]: compare the simulated kinship matrix (Fig 2B) and admixture proportions (Fig 3C) to our estimates on the real data [59]. Both simulations have n = 1000 individuals, m = 300, 000 loci, and K = 10 subpopulations or intermediate subpopulations. These simulations have FST = 0.1, comparable to previous estimates between human populations (in 1000 Genomes, the estimated FST between CEU (European-Americans) and CHB (Chinese) is 0.106, between CEU and YRI (Yoruba from Nigeria) it is 0.139, and between CHB and YRI it is 0.161 [23]).

Fig 2
Both panels have n = 1000 individuals along both axes, K = 10 subpopulations (final or intermediate), and FST = 0.1. Color corresponds to ${\theta }_{jk}^{T}$ between individuals j and k (equal to ${\phi }_{jk}^{T}$ off-diagonal, ${f}_{j}^{T}$ along the diagonal). (A) The independent subpopulations model has ${\theta }_{jk}^{T}=0$ between subpopulations, and varying ${\theta }_{jj}^{T}$ per subpopulation, resulting in a block-diagonal coancestry matrix. (B) Our admixture scenario models a 1D geography with extensive admixture and intermediate subpopulation differentiation that increases with distance, resulting in a smooth coancestry matrix with no independent subpopulations (no ${\theta }_{jk}^{T}=0$ between blocks). Individuals are ordered along each axis by geographical position.Coancestry matrices of simulations.
Fig 3
We model a 1D geography population that departs strongly from the independent subpopulations model. (A) K = 10 intermediate subpopulations, evenly spaced on a line, evolved independently in the past with FST increasing with distance, which models a sequence of increasing founder effects (from left to right) to mimic the global human population. (B) Once differentiated, individuals in these intermediate subpopulations spread by random walk modeled by Normal densities. (C) n = 1000 individuals, sampled evenly in the same geographical range, are admixed proportionally to the previous Normal densities. Thus, each individual draws most of its alleles from the closest intermediate subpopulation, and draws the fewest alleles from the most distant populations. Long-distance random walks of intermediate subpopulation individuals results in kinship for admixed individuals that decays smoothly with distance in Fig 2B. (D) For FST estimators that require a partition of individuals into subpopulations, individuals are clustered by geographical position (K = 10).1D admixture scenario.

The independent subpopulations simulation satisfies the HudsonK and BayeScan estimator assumptions: each independent subpopulation Su has a different FST value of ${f}_{{S}_{u}}^{T}$ relative to the MRCA population T (Fig 2A). Ancestral allele frequencies ${p}_{i}^{T}$ are drawn uniformly between 0.01 and 0.5. Allele frequencies ${p}_{i}^{{S}_{u}}$ for Su and locus i are drawn independently from the Balding-Nichols (BN) distribution [3] with parameters ${p}_{i}^{T}$ and ${f}_{{S}_{u}}^{T}$. Every individual j in subpopulation Su draws alleles randomly with probability ${p}_{i}^{{S}_{u}}$. Subpopulation sample sizes are drawn randomly (Methods, section Simulations).

The admixture simulation corresponds to a “BN-PSD” model [6, 27, 34, 60, 77]: the intermediate subpopulations are independent subpopulations that draw ${p}_{i}^{{S}_{u}}$ from the BN model, then each individual j constructs its allele frequencies as ${\pi }_{ij}={\sum }_{u=1}^{K}{p}_{i}^{{S}_{u}}{q}_{ju}$, which is a weighted average of the subpopulation allele frequencies ${p}_{i}^{{S}_{u}}$ with the admixture proportions qju of individual j and subpopulation u as weights (which satisfy ${\sum }_{u=1}^{K}{q}_{ju}=1$), as in the Pritchard-Stephens-Donnelly (PSD) admixture model [6365]. We constructed qju that model admixture resulting from spread by random walk of the intermediate subpopulations along a one-dimensional geography, as follows. Intermediate subpopulations Su are placed on a line with differentiation ${f}_{{S}_{u}}^{T}$ that grows with distance, which corresponds to a serial founder effect (Fig 3A). Upon differentiation, individuals in each Su spread by random walk, a process modeled by Normal densities (Fig 3B). Admixed individuals derive their ancestry proportional from these Normal densities, resulting in a genetic structure governed by geography (Figs 3C and 2B) and departing strongly from the independent subpopulations model (Fig 3D). The amount of spread—which sets the mean kinship across all individuals—was chosen to give a bias coefficient of ${s}^{T}=\frac{{\overline{\theta }}^{T}}{{F}_{\text{ST}}}=0.5$, which by Eq (32) results in a large downward bias for ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ (in contrast, the independent subpopulations simulation has sT = 0.1). The true coancestry and FST parameters of this simulation are given by the ${f}_{{S}_{u}}^{T}$ values of the intermediate subpopulations and the admixture coefficients qju of the individuals via the following equations [57]:

$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\theta }_{jk}^{T}& =\sum _{u=1}^{K}{q}_{ju}{q}_{ku}{f}_{{S}_{u}}^{T},\hfill \\ \hfill {F}_{\text{ST}}& =\sum _{j=1}^{n}\sum _{u=1}^{K}{w}_{j}{q}_{ju}^{2}{f}_{{S}_{u}}^{T}.\hfill \end{array}\end{array}$
The first equation above connecting coancestry to admixture proportions was derived independently in other work [62], but the FST for the admixed individuals was absent and instead follows from our generalized FST definition given in Eq (9). See Methods, section Simulations for additional details regarding these simulations.

#### Evaluation of FST estimators

Our admixture simulation illustrates the large biases that can arise if existing FST estimators that require independent subpopulations or FST estimates derived from existing kinship estimators are misapplied to arbitrary population structures to estimate the generalized FST, and demonstrate the higher accuracy of our new FST estimator (${\stackrel{^}{F}}_{\text{ST}}^{\text{new}}$ given by the combination of Eqs (36) and (37)). The WC FIT (total inbreeding) estimator was also evaluated.

First, we test these estimators in our independent subpopulations simulation. The HudsonK (Methods, section Generalized HudsonK FST estimator) and BayeScan FST estimators are consistent in this simulation, since their assumptions are satisfied (Fig 4A). The WC FST estimator assumes that ${f}_{{S}_{u}}^{T}={F}_{\text{ST}}$ for all subpopulations Su , which does not hold; nevertheless, WC has only a small bias (Fig 4A). The WC FIT estimator arrives at similar estimates, as it should since there is no local inbreeding, so the true FIT also equals FST. The Weir-Hill estimator permits different ${f}_{{S}_{u}}^{T}$ values per subpopulation, but assigns equal weight to individuals rather than subpopulations (Methods, section The Weir-Hill FST estimator), resulting in a slightly different target FST (we verified that these estimates are unbiased for this FST). For comparison, we show the standard kinship-based ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (25) (weights from Methods, section Simulations) and ${\stackrel{^}{F}}_{\text{ST}}^{\text{WG}}$ based on the Weir-Goudet kinship estimates for individuals, both of which do not have corrections that would make them consistent under the independent subpopulations model. Since the number of subpopulations K is large, ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ has a small relative bias of about ${s}^{T}=\frac{1}{K}=10%$ (Fig 4A); greater bias is expected for smaller K. Our new FST estimator has a very small bias in this simulation resulting from estimating the minimum kinship from the smallest kinship between subpopulations (see Eq (37)) rather than their average as HudsonK does implicitly (Fig 4A).

Fig 4
The Weir-Cockerham, Weir-Hill, Weir-Goudet (for individuals), HudsonK (equal to Weir-Goudet for subpopulations, S1 Text), BayeScan, ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (25) derived from the standard kinship estimator, and our new FST estimator in Eqs (34) and (37), are evaluated on simulated genotypes from our two models (Fig 2). The Weir-Cockerham FIT estimator was also included to show that estimation of total inbreeding behaves similarly to FST estimators. (A) The independent subpopulations model required by the Weir-Hill, HudsonK, and BayeScan FST estimators. All but standard kinship (${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$) and Weir-Goudet (for individuals) recover the target FST IBD probability in Eq (9) (red line) with small errors. (B) Our admixture scenario, which has no independent subpopulations, was constructed so ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}\approx \frac{1}{2}{F}_{\text{ST}}$. Only our new estimates are accurate. The rest of these estimators give values smaller than the target FST IBD probability, which result from treating kinship as zero between every subpopulations imposed by geographic clustering (or between individuals for Standard Kinship and Weir-Goudet). The ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ estimator limit in Eq (14) (green dotted line) overlaps the true FST (red line) in (A) but not (B). Estimates (blue) include 95% prediction intervals (often too narrow to see) from 39 independently-simulated genotype matrices for each model (Methods, section Prediction intervals).Evaluation of FST estimators.

Next we test these estimators in our admixture simulation. To apply the FST estimators that require subpopulations to the admixture model, individuals are clustered into subpopulations by their geographical position (Fig 3D). We find that estimates of all existing methods are smaller than the true FST by nearly half, as predicted by the limit of ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eq (14) (Fig 4B). The WC FIT estimator obtains slightly larger estimates than the WC FST estimator, but overall remains as biased as the other FST estimators, showing that the use of a total inbreeding estimator for independent subpopulations displays the same bias as the corresponding FST estimator. By construction, the kinship-based ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ also has a large relative bias of about sT = 50%; remarkably, all existing FST estimators for subpopulations suffer from comparable biases. Thus, the corrections for independent subpopulations present in the WC, Weir-Hill and HudsonK estimators, or the Bayesian likelihood modeling of BayeScan, are insufficient for accurate estimation of the target generalized FST (Eq (9)) in this admixture scenario. Only our new FST estimator achieves accurate estimates of the generalized FST in the admixture simulation (Fig 4B).

#### Evaluation of kinship estimators

Our admixture simulation illustrates the distortions of the standard kinship estimator ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ in Eq (18), the linearly-transformed kinship values given by the Weir-Goudet estimator, and demonstrates the improved accuracy of our new kinship estimator ${\stackrel{^}{\phi }}_{jk}^{T,\text{new}}$ given by the combination of Eqs (34) and (37). Kinship matrix estimates and their limits are visualized as heatmaps in Fig 5, whereas estimator accuracy is shown directly in Fig 6. The limit of the standard estimator ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ in Eq (18) would have had a uniform bias if ${\overline{\phi }}_{j}^{T}={\overline{\phi }}^{T}$ held for all individuals j. For that reason, our admixture simulation has varying differentiation ${f}_{{S}_{u}}^{T}$ per intermediate subpopulation Su (Fig 3A), which causes large differences in ${\overline{\phi }}_{j}^{T}$ per individual j and therefore large distortions in ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$. The Weir-Goudet approach estimates the linearly-transformed kinship values calculated in Eq (38).

Fig 5
Observed accuracy for two existing kinship coefficient estimators is illustrated in our admixture simulation and contrasted to the nearly unbiased estimates of our new estimator. Plots show n = 1000 individuals along both axes, and color corresponds to ${\phi }_{jk}^{T}$ between individuals jk and to ${f}_{j}^{T}$ along the diagonal (${f}_{j}^{T}$ is in the same scale as ${\phi }_{jk}^{T}$ for jk; plotting ${\phi }_{jj}^{T}$, which have a minimum value of $\frac{1}{2}$, would result in a discontinuity in this figure). (A) True kinship matrix. (B) Estimated kinship using our new estimator in Eqs (34) and (37) from simulated genotypes recovers the true kinship matrix with high accuracy. (C) Theoretical limit of ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ in Eq (19) as the number of independent loci goes to infinity demonstrates the accuracy of our bias predictions under the kinship model. (D) Standard kinship estimates ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ given by Eq (18) from simulated genotypes are downwardly biased on average and distorted by pair-specific amounts. (E) Theoretical limit of the Weir-Goudet kinship estimator given by Eq (38). (F) Weir-Goudet kinship estimates from the same simulated genotypes agree with our calculated limit.Evaluation of kinship estimators.
Fig 6
Here the estimated kinship values are directly compared to their true values, in the same admixture simulation data (n = 1000 individuals) shown in the previous figure. (A) Kinship between different individuals (excluding inbreeding). The new estimator has practically no bias in this evaluation (falls on the 1-1 dashed gray line). The standard estimator has a complex, non-linear bias that covers a large area of errors. (B) Inbreeding comparison, shows the bias of the standard estimate follows a different pattern for inbreeding compared to kinship between individuals. To better visualize and compare data across panels, a random subset of n points (out of the original n(n − 1)/2 unique individual pairs) were plotted in (A), matching the number of individuals (number of points in (B)).Accuracy of kinship estimators.

Our new kinship estimator (Fig 5B) recovers the true kinship matrix of this complex population structure (Fig 5A), with an RMSE of 2.83% relative to the mean ${\phi }_{jk}^{T}$ (Fig 6). In contrast, estimates using the standard estimator have a large overall downward bias (Fig 5C), resulting in an RMSE of 115.72% from the true ${\phi }_{jk}^{T}$ relative to the mean ${\phi }_{jk}^{T}$ (Fig 6). Additionally, estimates from ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ are very distorted, with an abundance of ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}<{\phi }_{jk}^{T}$ cases—some of which are negative estimates (blue in Fig 5C)—but remarkably also cases with ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}>{\phi }_{jk}^{T}$ (top left corner of Figs 5C and 6).

Now we compare the convergence of the ratio-of-means and mean-of-ratios versions of the standard kinship estimator to their biased limit we calculated in Eq (19) (Fig 5D). The ratio-of-means estimate ${\stackrel{^}{\phi }}_{jk}^{T,\text{std}}$ (Fig 5C) has an RMSE of 2.14% from its limit relative to the mean ${\phi }_{jk}^{T}$. In contrast, the mean-of-ratios estimates that are prevalent in the literature have a greater RMSE of 10.77% from the same limit in Eq (19). Thus, as expected from our theoretical results in section Assessing the accuracy of genome-wide ratio estimators , the ratio-of-means estimate is much closer to the desired limit than the mean-of-ratio estimate. The distortions are similar for the estimator that uses IAFs in Eq (24), with reduced RMSEs from its limit of 0.32% and 8.82% for the ratio-of-means and mean-of-ratios estimates, respectively.

#### Evaluation of oracle-adjusted FST estimators

Here we verify additional calculations for the bias of the standard kinship-based estimator ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ and the unbiased adjusted “oracle” FST estimators that require the true mean kinship ${\overline{\phi }}^{T}$ or the bias coefficient sT to be known. Note that ${\stackrel{^}{F}}_{\text{ST}}^{\text{new}}$ in Eq (36) is related but not identical to these oracle estimators. We tested both IAF (Fig 7A) and genotype (Fig 7B) versions of these estimators. The unadjusted ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ in Eq (26) is severely biased (blue in Fig 7) by construction, and matches the calculated limit for IAFs and genotypes (green lines in Fig 7, which are close because ${\overline{\phi }}^{T}\approx {\overline{\theta }}^{T}$). In contrast, the two consistent adjusted estimators ${\stackrel{^}{F}}_{\text{ST}}^{\prime }$ and ${\stackrel{^}{F}}_{\text{ST}}^{\prime \prime }$ in Eqs (27) and (31) estimate FST quite well (blue predictions overlap the true FST red line in Fig 7). However, ${\stackrel{^}{F}}_{\text{ST}}^{\prime }$ and ${\stackrel{^}{F}}_{\text{ST}}^{\prime \prime }$ are oracle methods, since they require parameters (${\overline{\phi }}^{T}$, ${\overline{\theta }}^{T}$, sT) that are not known in practice.

Fig 7
The convergence values we calculated for the standard kinship plug-in and adjusted FST estimators are validated using our admixture simulation. All adjusted estimators are unbiased but are “oracle” methods, since the mean kinship (${\overline{\phi }}^{T}$), mean coancestry (${\overline{\theta }}^{T}$), or bias coefficient (${s}^{T}=\frac{{\overline{\theta }}^{T}}{{F}_{\text{ST}}}$ for IAFs, replaced by $\frac{{\overline{\phi }}^{T}}{{F}_{\text{ST}}}$ for genotypes) are usually unknown. (A) Estimation from individual-specific allele frequencies (IAFs): ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ is the standard coancestry plug-in estimator in Eq (26); ${\stackrel{^}{F}}_{\text{ST}}^{\prime }$ “Adj. ${\overline{\theta }}^{T}$” is in Eq (27); ${\stackrel{^}{F}}_{\text{ST}}^{\prime \prime }$ “Adj. s ” is in Eq (31). (B) For genotypes, ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ is given in Eq (25), and the adjusted estimators use ${\overline{\phi }}^{T}$ rather than ${\overline{\theta }}^{T}$. Lines: true FST (red line), limits of biased estimators ${\stackrel{^}{F}}_{\text{ST}}^{\text{std}}$ (green lines, which differ slightly per panel). Estimates (blue) include 95% prediction intervals (too narrow to see) from 39 independently-simulated genotype matrices for our admixture model (Methods, section Prediction intervals).Evaluation of standard and adjusted FST estimators.

Prediction intervals were computed from estimates over 39 independently-simulated IAF and genotype matrices (Methods, section Prediction intervals). Estimator limits are always contained in these intervals because the number of independent loci (m = 300, 000) is sufficiently large. Estimates that use genotypes have wider intervals than estimates from IAFs; however, IAFs are not known in practice, and use of estimated IAFs might increase noise. Genetic linkage, not present in our simulation, will also increase noise in real data.

## Discussion

We studied analytically the most commonly-used estimators of FST and kinship, which can be derived using the method of moments. We determined the estimation limits of convergence of these approaches under two models of arbitrary population structure (Fig 1). We found that no existing approaches estimate the generalized FST (an IBD probability) accurately (but note that some of these approaches intended to estimate a linearly-transformed FST quantity and not the IBD probability). We also showed that the standard kinship estimator is biased on structured populations (particularly when the average kinship is comparable to the kinship coefficients of interest), and this bias varies for each pair of individuals. These results led us to a new kinship estimator, which is consistent if the minimum kinship is estimated consistently (Fig 1). We presented an implementation of this approach, which is practically unbiased in our simulations. Our kinship and FST estimates in human data are consistent with the African Origins model while suggesting that human differentiation is considerably greater than previously estimated [59].

Estimation of FST in the correct scale is crucial for its interpretation as an IBD probability, for obtaining comparable estimates in different datasets and across species, as well as for DNA forensics [3, 7, 19, 20, 7880]. Our framework results in a new unbiased genome-wide FST estimator. However, our findings may not have direct implications for single-locus FST estimate approaches where only the relative ranking matters, such as for the identification of loci under selection [8, 10, 8186], assuming that the bias of the genome-wide estimator carries over uniformly to all single-locus estimates. Our convergence calculations in section Assessing the accuracy of genome-wide ratio estimators require large numbers of loci, so they do not apply to single-locus estimates. Moreover, various methods for single-locus FST estimation for multiple alleles suffer from a strong dependence to the maximum allele frequency and heterozygosity [8385, 8790] that suggests that a more complicated bias is present in these single-locus FST estimators.

We have shown that the misapplication of existing FST estimators for independent subpopulations may lead to downwardly-biased estimates that can approach zero even when the true generalized FST is large. Weir-Cockerham [17], Weir-Hill [4], HudsonK (which generalizes the Hudson pairwise FST estimator [23] to K independent populations; also equals the Weir-Goudet approach for subpopulations [21]; S1 Text), and BayeScan [10]FST estimates in our admixture simulation are all smaller than the FST target by nearly a factor of two (Fig 4B), and differ from our new FST estimates in humans by nearly a factor of three [59]. To be accurate, existing FST estimators require independent subpopulations, so the observed biases arise from their misapplication to subpopulations that are neither independent not homogeneous. Nevertheless, natural populations—particularly humans—often do not adhere to the independent subpopulations model [59, 9195].

The standard kinship coefficient estimator we investigated is often used to control for population structure in GWAS and to estimate genome-wide heritability [16, 27, 3035]. While this estimator was known to be biased [16, 35], no closed-form limit had been calculated until very recently [21, 62]. These kinship estimates are biased downwards on average, but bias also varies for each pair of individuals (Figs 1 and 5). Thus, the use of these distorted kinship estimates may be problematic in GWAS or for estimating heritability, but the extent of the problem remains to be determined.

We developed a theoretical framework for assessing genome-wide ratio estimators of FST and kinship. We proved that common ratio-of-means estimators converge almost surely to the ratio of expectations for infinite independent loci (S1 Text). Our result justifies approximating the expectation of a ratio-of-means estimator with the ratio of expectations [4, 17, 23]. However, mean-of-ratios estimators may not converge to the ratio of expectations for infinite loci. Mean-of-ratios estimators are potentially asymptotically unbiased for infinite individuals, but it is unclear which estimators have this behavior. We found that the ratio-of-means kinship estimator had much smaller errors from the ratio of expectations than the more common mean-of-ratios estimator, whose convergence value is unknown. Therefore, we recommend ratio-of-means estimators, whose asymptotic behavior is well understood.

Our new framework enables accurate FST estimation in more complex datasets than before, but challenges remain. One challenge is the estimation of local inbreeding coefficients, which are required for estimating the generalized FST when not all individuals are locally outbred. To this end, we suggest employing existing approaches that infer inbreeding from large runs of homozygosity or related strategies [6668], particularly when such self-IBD blocks are much larger than observed between individuals in the same subpopulation. A streamlined approach for jointly estimating total and local inbreeding is desirable, but will require an appropriate evaluation featuring realistic simulation of local inbreeding in a complex population structure. Another challenge is the estimation of the minimum kinship value without the use of subpopulation labels, so that accurate FST estimates can be obtained with even less user supervision. A more general unsupervised method could better ensure accuracy under extreme cases, such as when there are few unrelated individual pairs. These challenges can be overcome with the estimators we have presented, although supervision is needed to ensure that local inbreeding and the minimum kinship are estimated correctly.

We have demonstrated the need for new models and methods to study complex population structures, and have proposed a new approach for kinship and FST estimation that provides nearly unbiased estimates in this setting. Extending our implementation to deliver consistent accuracy in arbitrary population structures will require further innovation, and the results provided here may be useful in leading to more robust estimators in the future.

## Methods

### Previous FST estimators for the independent subpopulations model

Here we summarize the previous Weir-Cockerham, Weir-Hill, and Hudson FST estimators for independent subpopulations and derive the generalized HudsonK estimator for more than two subpopulations (which also equals the recent Weir-Goudet FST estimator for subpopulations under biallelic loci; S1 Text). We show that each of these estimators reduces, under infinite subpopulation sizes, to ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eqs (11) to (13) that was studied in the results. In this section, let i index the m loci, j index the n subpopulations, nj be the number of individuals sampled from subpopulation j, and ${\stackrel{^}{p}}_{ij}$ be the sample reference allele frequency at locus i in subpopulation j.

#### The Weir-Cockerham FST estimator

The Weir-Cockerham (WC) FST estimator [17] estimates the coancestry parameter θT shared by each of the n independent subpopulation in consideration. Let ${\stackrel{^}{h}}_{ij}$ denote the fraction of heterozygotes in subpopulation j for locus i. The ratio-of-means WC FST estimator and its limit for independent subpopulations (${\theta }_{jk}^{T}=0$ for jk) with equal differentiation (${\theta }_{jj}^{T}={\theta }^{T}$) is

$\begin{array}{cc}\hfill \overline{n}& =\frac{1}{n}\sum _{j=1}^{n}{n}_{j},\phantom{\rule{1em}{0ex}}{C}^{2}=\frac{1}{{\overline{n}}^{2}\left(n-1\right)}\sum _{j=1}^{n}{\left({n}_{j}-\overline{n}\right)}^{2},\hfill \\ \hfill {\stackrel{^}{p}}_{i}^{T}& =\frac{1}{n}\sum _{j=1}^{n}\frac{{n}_{j}}{\overline{n}}{\stackrel{^}{p}}_{ij},\phantom{\rule{1em}{0ex}}{\overline{h}}_{i}=\frac{1}{n}\sum _{j=1}^{n}\frac{{n}_{j}}{\overline{n}}{\stackrel{^}{h}}_{ij},\phantom{\rule{1em}{0ex}}{\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{n-1}\sum _{j=1}^{n}\frac{{n}_{j}}{\overline{n}}{\left({\stackrel{^}{p}}_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},\hfill \\ \hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{WC}}& =\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}-\frac{1}{\overline{n}-1}\left({\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)-\frac{n-1}{n}{\stackrel{^}{\sigma }}_{i}^{2}-\frac{1}{4}{\overline{h}}_{i}\right)}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)\left(1-\frac{\overline{n}{C}^{2}}{n\left(\overline{n}-1\right)}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}\left(1+\frac{\left(n-1\right)\overline{n}{C}^{2}}{n\left(\overline{n}-1\right)}\right)+\frac{{\overline{h}}_{i}{C}^{2}}{4n\left(\overline{n}-1\right)}}\hfill \\ & \underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}={\theta }^{T}.\end{array}$
Note that ${\stackrel{^}{p}}_{i}^{T}$ above weighs every individual equally by weighing subpopulation j proportional to its sample size nj , so it equals the estimator in Eq (17) with uniform weights.

Now we simplify this estimator as the sample size of every subpopulation becomes infinite. First set the sample size of every subpopulation nj equal to their mean $\overline{n}$, which implies C2 = 0 and

$\begin{array}{cc}\hfill {\stackrel{^}{p}}_{i}^{T}& =\frac{1}{n}\sum _{j=1}^{n}{\stackrel{^}{p}}_{ij},\phantom{\rule{1em}{0ex}}{\overline{h}}_{i}=\frac{1}{n}\sum _{j=1}^{n}{\stackrel{^}{h}}_{ij},\phantom{\rule{1em}{0ex}}{\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{n-1}\sum _{j=1}^{n}{\left({\stackrel{^}{p}}_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},\\ \hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{WC}}& =\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}-\frac{1}{\overline{n}-1}\left({\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)-\frac{n-1}{n}{\stackrel{^}{\sigma }}_{i}^{2}-\frac{1}{4}{\overline{h}}_{i}\right)}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}}.\hfill \end{array}$
Now we take the limit as the sample size $\overline{n}\to \infty$, which results in sample allele frequencies converging to the true subpopulation allele frequencies ${\stackrel{^}{p}}_{ij}\to {\pi }_{ij}$ for every subpopulation j and locus i, and
${\stackrel{^}{p}}_{i}^{T}=\frac{1}{n}\sum _{j=1}^{n}{\pi }_{ij},\phantom{\rule{1em}{0ex}}{\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{n-1}\sum _{j=1}^{n}{\left({\pi }_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},\phantom{\rule{1em}{0ex}}{\stackrel{^}{F}}_{\text{ST}}^{\text{WC}}=\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}},$
which matches the ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eqs (11) to (13) as desired. Note the number of subpopulations n remains finite, and the sample heterozygosity ${\overline{h}}_{i}$ is not needed in the limit.

#### The Weir-Hill FST estimator

Weir and Hill developed new estimators for subpopulation-specific FST values and considered the effects of non-independent subpopulations [4]. However, these estimators target linearly-transformed FST values, and recover the FST defined in Eq (9) only when subpopulations are independent [4], so we group them here with other estimators that strictly assume independent subpopulations. For simplicity, here we only consider the global FST estimator; the estimators of the coancestry matrix of the subpopulations was found to have the same overall linear transformation [4]. In the limit of infinite subpopulation sizes, this estimator also converges to the asymptotic FST estimator for independent subpopulations (${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$) discussed in the main text.

The Weir-Hill (WH) FST estimator, simplified here for biallelic loci but extended to average over loci, and its limit, are given by

$\begin{array}{cc}\hfill {\stackrel{^}{p}}_{i}^{T}& =\sum _{j=1}^{n}{w}_{j}{\stackrel{^}{p}}_{ij},\phantom{\rule{1em}{0ex}}\phantom{\rule{1em}{0ex}}{w}_{j}=\frac{{n}_{j}}{\sum _{j=1}^{n}{n}_{j}},\hfill \\ \hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{WH}}& =1-\frac{\left(\sum _{j=1}^{n}{n}_{j}\left(1-{w}_{j}\right)\right)\left(\sum _{i=1}^{m}\sum _{j=1}^{n}{w}_{j}\frac{2{n}_{j}}{2{n}_{j}-1}{\stackrel{^}{p}}_{ij}\left(1-{\stackrel{^}{p}}_{ij}\right)\right)}{\sum _{i=1}^{m}\sum _{j=1}^{n}{n}_{j}{\left({\stackrel{^}{p}}_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2}+{n}_{j}\left(1-{w}_{j}\right){\stackrel{^}{p}}_{ij}\left(1-{\stackrel{^}{p}}_{ij}\right)}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}\frac{{F}_{\text{ST}}-{\stackrel{˜}{\theta }}^{T}}{1-{\stackrel{˜}{\theta }}^{T}},\end{array}$
where the target FST and ${\stackrel{˜}{\theta }}^{T}$ both weigh individuals (rather than subpopulations) equally [4]:
$\begin{array}{c}\hfill {F}_{\text{ST}}=\sum _{j=1}^{n}{w}_{j}{\theta }_{jj}^{T},\phantom{\rule{1em}{0ex}}\phantom{\rule{1em}{0ex}}{\stackrel{˜}{\theta }}^{T}=\frac{2}{1-{\sum }_{j=1}^{n}{w}_{j}^{2}}\sum _{j=2}^{n}\sum _{k=1}^{j-1}{w}_{j}{w}_{k}{\theta }_{jk}^{T}.\end{array}$
For equal sample sizes nj = nSj, we have ${w}_{j}=\frac{1}{n}$, ${n}_{jc}={n}_{S}\left(1-\frac{1}{n}\right)$, and the estimator becomes
$\begin{array}{cc}\hfill {\stackrel{^}{p}}_{i}^{T}& =\frac{1}{n}\sum _{j=1}^{n}{\stackrel{^}{p}}_{ij},\phantom{\rule{1em}{0ex}}{\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{n-1}\sum _{j=1}^{n}{\left({\stackrel{^}{p}}_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},\\ \hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{WH}}& =\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}\left(\frac{2{n}_{S}-\frac{1}{n}}{2{n}_{s}-1}\right)-{p}_{i}^{T}\left(1-{p}_{i}^{T}\right)\left(\frac{1}{2{n}_{s}-1}\right)}{\sum _{i=1}^{m}{p}_{i}^{T}\left(1-{p}_{i}^{T}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}}.\end{array}$
Therefore, as sample sizes per subpopulation go to infinity (nS → ∞, which results in ${\stackrel{^}{p}}_{ij}\to {\pi }_{ij}$ for every (i, j)), we again recover the desired limiting FST estimator for independent subpopulations (${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eqs (11) to (13)).

#### The Hudson FST estimator

The Hudson pairwise FST estimator [23] measures the differentiation of two subpopulations (j, k). The estimator and its limit for two independent subpopulations (${\theta }_{jk}^{T}=0$) is

$\begin{array}{c}\hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{Hudson}}=\frac{\sum _{i=1}^{m}{\left({\stackrel{^}{p}}_{ij}-{\stackrel{^}{p}}_{ik}\right)}^{2}-\frac{{\stackrel{^}{p}}_{ij}\left(1-{\stackrel{^}{p}}_{ij}\right)}{2{n}_{j}-1}-\frac{{\stackrel{^}{p}}_{ik}\left(1-{\stackrel{^}{p}}_{ik}\right)}{2{n}_{k}-1}}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{ij}\left(1-{\stackrel{^}{p}}_{ik}\right)+{\stackrel{^}{p}}_{ik}\left(1-{\stackrel{^}{p}}_{ij}\right)}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}=\frac{{\theta }_{jj}^{T}+{\theta }_{kk}^{T}}{2}.\end{array}$

#### Generalized HudsonK FST estimator

Here we derive the “HudsonK” estimator (first made available in [58]), which generalizes the Hudson pairwise FST estimator in Eq (40) to n independent subpopulations. This estimator also equals the recent Weir-Goudet FST estimator for subpopulations [21] (for biallelic loci; S1 Text). Note that for independent subpopulations, the FST of all the subpopulations equals the mean pairwise FST of every pair of subpopulations:

$\begin{array}{c}\hfill \frac{1}{{n}^{2}}\sum _{j=1}^{n}\sum _{k=1}^{n}\left(\frac{{\theta }_{jj}^{T}+{\theta }_{kk}^{T}}{2}\right)=\frac{1}{n}\sum _{j=1}^{n}{\theta }_{jj}^{T}={F}_{\text{ST}}.\end{array}$
For that reason, averaging numerators and denominators of the pairwise estimator in Eq (40) before computing the ratio, we obtain the generalized estimator and a limit under independent subpopulations of
$\begin{array}{cc}\hfill {\stackrel{^}{p}}_{i}^{T}& =\frac{1}{n}\sum _{j=1}^{n}{\stackrel{^}{p}}_{ij},\phantom{\rule{1em}{0ex}}{\stackrel{^}{\sigma }}_{i}^{2}=\frac{1}{n-1}\sum _{j=1}^{n}{\left({\stackrel{^}{p}}_{ij}-{\stackrel{^}{p}}_{i}^{T}\right)}^{2},\hfill \\ \hfill {\stackrel{^}{F}}_{\text{ST}}^{\text{HudsonK}}& =\frac{\sum _{i=1}^{m}{\stackrel{^}{\sigma }}_{i}^{2}-\frac{1}{n}\sum _{j=1}^{n}\frac{{\stackrel{^}{p}}_{ij}\left(1-{\stackrel{^}{p}}_{ij}\right)}{2{n}_{j}-1}}{\sum _{i=1}^{m}{\stackrel{^}{p}}_{i}^{T}\left(1-{\stackrel{^}{p}}_{i}^{T}\right)+\frac{1}{n}{\stackrel{^}{\sigma }}_{i}^{2}}\underset{m\to \infty }{\overset{\text{a.s.}}{\to }}{F}_{\text{ST}}=\frac{1}{n}\sum _{j=1}^{n}{\theta }_{jj}^{T}.\end{array}$
Note that unlike the WC and Weir-Hill estimators, ${\stackrel{^}{p}}_{i}^{T}$ above weighs every subpopulation equally, so every individual is weighed inversely proportional to the sample sizes nj of their subpopulation j.

Like WC and Weir-Hill, ${\stackrel{^}{F}}_{\text{ST}}^{\text{HudsonK}}$ simplifies to ${\stackrel{^}{F}}_{\text{ST}}^{\text{indep}}$ in Eqs (11) to (13) in the limit of infinite sample sizes nj → ∞, where ${\stackrel{^}{p}}_{ij}\to {\pi }_{ij}$ for every (i, j).

### Simulations

#### Construction of subpopulation allele frequencies

We simulate K = 10 subpopulations Su and m = 300, 000 independent loci. Every locus i draws ${p}_{i}^{T}~\text{Uniform}\left(0.01,0.5\right).$ We set ${f}_{{S}_{u}}^{T}=\frac{u}{K}\tau ,$ where τ ≤ 1 tunes FST. For the independent subpopulations model, ${F}_{\text{ST}}=\frac{1}{K}{\sum }_{u=1}^{K}{f}_{{S}_{u}}^{T}=\frac{\tau \left(K+1\right)}{2K},$ so $\tau =\frac{2K{F}_{\text{ST}}}{K+1}$ gives the desired FST (τ ≈ 0.18 for FST = 0.1). For the admixture model, τ is found numerically (τ ≈ 0.90 for FST = 0.1; see last subsection). Lastly, ${p}_{i}^{{S}_{u}}$ values are drawn from the Balding-Nichols distribution,

${p}_{i}^{{S}_{u}}|T\sim \text{Beta}\left({p}_{i}^{T}\left(\frac{1}{{f}_{{S}_{u}}^{T}}-1\right),\left(1-{p}_{i}^{T}\right)\left(\frac{1}{{f}_{{S}_{u}}^{T}}-1\right)\right),$
which results in subpopulation allele frequencies that obey the coancestry model of Eq (6), with $E\left[{p}_{i}^{{S}_{u}}|T\right]={p}_{i}^{T}$ and $Var\left({p}_{i}^{{S}_{u}}|T\right)={f}_{{S}_{u}}^{T}{p}_{i}^{T}\left(1-{p}_{i}^{T}\right)$ [3], as desired.

#### Random subpopulation sizes

We randomly generate sample sizes r = (ru) for K subpopulations and ${\sum }_{u=1}^{K}{r}_{u}=n=1000$ individuals, as follows. First, draw x ∼ Dirichlet (1, …, 1) of length K and r = round(n x). While ${min}_{u}{r}_{u}<\frac{n}{3K}$, draw a new r, to prevent small subpopulations (they do not occur in real data). Due to rounding, ${\sum }_{u=1}^{K}{r}_{u}$ may not equal n as desired. Thus, while $\delta =n-{\sum }_{u=1}^{K}{r}_{u}\ne 0$, a random u is updated to ruru + sgn(δ), which brings δ closer to zero at every iteration. Weights for individuals j in Su are ${w}_{j}=\frac{1}{K{r}_{u}}$ so the generalized FST matches ${F}_{\text{ST}}=\frac{1}{K}{\sum }_{u=1}^{K}{f}_{{S}_{u}}^{T}$ from the independent subpopulations model (see section The generalized FST for arbitrary population structures), which HudsonK estimates.

#### Admixture proportions from 1D geography

We construct qju from random-walk migrations along a one-dimensional geography. Let xu be the coordinate of intermediate subpopulation u and yj the coordinate of a modern individual j. We assume qju is proportional to f(|xuyj|), or

$\begin{array}{c}\hfill {q}_{ju}=\frac{f\left(|{x}_{u}-{y}_{j}|\right)}{\sum _{v=1}^{K}f\left(|{x}_{v}-{y}_{j}|\right)}.\end{array}$
where f is the Normal density function with μ = 0 and tunable σ. The Normal density models random walks, where σ sets the spread of the populations (Fig 5). Our simulation uses xu = u and ${y}_{j}=\frac{1}{2}+\frac{j-1}{n-1}K$, so the intermediate subpopulations span between 1 and K and individuals span between $\frac{1}{2}$ and $K+\frac{1}{2}$. For the FST estimators that require subpopulations, individual j is assigned to the nearest subpopulation Su (the u that minimizes |xuyj |; Fig 3D); these subpopulations have equal sample size, so ${w}_{j}=\frac{1}{n}$ is appropriate.

#### Choosing σ and τ

Here we find values for σ (controls qjk) and τ (scales ${f}_{{S}_{u}}^{T}$) that give ${s}^{T}=\frac{1}{2}$ and FST = 0.1 in the admixture model. In our simulation, ${w}_{j}=\frac{1}{n}$ and ${f}_{{S}_{u}}^{T}=\frac{u}{K}\tau$, so applying those parameters to Eq (39) gives ${\theta }_{jk}^{T}=\frac{\tau }{K}{\sum }_{u=1}^{K}u{q}_{ju}{q}_{ku}$ and ${F}_{\text{ST}}=\frac{\tau }{nK}{\sum }_{j=1}^{n}{\sum }_{u=1}^{K}u{q}_{ju}^{2}$. Therefore,

$\begin{array}{c}\hfill {s}^{T}=\frac{{\overline{\theta }}^{T}}{{F}_{\text{ST}}}=\frac{1}{n}\frac{\sum _{u=1}^{K}u{\left(\sum _{j=1}^{n}{q}_{ju}\left(\sigma \right)\right)}^{2}}{\sum _{u=1}^{K}u\left(\sum _{j=1}^{n}{q}_{ju}^{2}\left(\sigma \right)\right)}\end{array}$
depends only on σ. A numerical root finder finds that σ ≈ 1.78 gives ${s}^{T}=\frac{1}{2}$. For fixed qju,
$\begin{array}{c}\hfill \tau =\frac{{F}_{\text{ST}}}{\frac{1}{K}\sum _{u=1}^{K}u\left(\frac{1}{n}\sum _{j=1}^{n}{q}_{ju}^{2}\right)}.\end{array}$
FST = 0.1 is achieved with τ ≈ 0.901.

### Prediction intervals

Prediction intervals with α = 95% correspond to the range of n = 39 independent FST estimates. In the general case, n independent statistics are given in order X(1) < … < X(n). Then I = [X(j), X(n+1−j)] is a prediction interval with confidence $\alpha =\frac{n+1-2j}{n+1}$ [96]. In our case, j = 1 and n = 39 gives α = 0.95, as desired. Each estimate was constructed from simulated data with the same dimensions and structure as before (fixed ${f}_{{S}_{u}}^{T}$ and qju; fixed sample sizes for the independent subpopulations model), but with ${p}_{i}^{T},{p}_{i}^{{S}_{u}},{\pi }_{ij},{x}_{ij}$ drawn separately for each estimate.

### BayeScan and Weir-Goudet implementations

Weir-Goudet (WG) kinship estimates [2022] were calculated using the function snpgdsIndivBeta in the R package SNPRelate 1.20.1 available on Bioconductor and GitHub. We found identical estimates using the function beta.dosage in the R package hierfstat 0.4.30 available on GitHub. WG (individuals) FST estimates were computed from the kinship estimates as described in section Comparison to the Weir-Goudet kinship estimator for individuals.

BayeScan 2.1 was downloaded from http://cmpg.unibe.ch/software/BayeScan/. To estimate FST, first the per-subpopulation FST values were estimated across loci assuming no selection, then the global FST was given by the mean FST across subpopulations.

### Software

An R package called popkin, which implements the kinship and FST estimation methods proposed here, is available on the Comprehensive R Archive Network (CRAN) at https://cran.r-project.org/package=popkin and on GitHub at https://github.com/StoreyLab/popkin.

An R package called bnpsd, which implements the BN-PSD admixture simulation, is available on CRAN at https://cran.r-project.org/package=bnpsd and on GitHub at https://github.com/StoreyLab/bnpsd.

An R package called popkinsuppl, which implements memory-efficient algorithms for the Weir-Cockerham, Weir-Hill, and HudsonK FST estimators, and the standard kinship estimator, is available on GitHub at https://github.com/OchoaLab/popkinsuppl.

Public code reproducing these analyses are available at https://github.com/StoreyLab/human-differentiation-manuscript.

## References

MalécotG. Mathématiques de l’hérédité. Masson et Cie; 1948.

WrightS. The genetical structure of populations. Ann Eugen. 1951;15(4):323354.

BaldingDJ, NicholsRA. A method for quantifying differentiation between populations at multi-allelic loci and its implications for investigating identity and paternity. Genetica. 1995;96(1-2):312.

WeirBS, HillWG. Estimating F-Statistics. Annual Review of Genetics. 2002;36(1):721750.

NicholsonG, SmithAV, JónssonF, GústafssonO, StefánssonK, DonnellyP. Assessing population differentiation and isolation from single-nucleotide polymorphism data. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2002;64(4):695715.

FalushD, StephensM, PritchardJK. Inference of population structure using multilocus genotype data: linked loci and correlated allele frequencies. Genetics. 2003;164(4):15671587.

BaldingDJ. Likelihood-based inference for genetic correlation coefficients. Theoretical Population Biology. 2003;63(3):221230.

BeaumontMA, BaldingDJ. Identifying adaptive genetic divergence among populations from genome scans. Molecular Ecology. 2004;13(4):969980.

FollM, GaggiottiO. Identifying the Environmental Factors That Determine the Genetic Structure of Populations. Genetics. 2006;174(2):875891.

10

FollM, GaggiottiO. A genome-scan method to identify selected loci appropriate for both dominant and codominant markers: a Bayesian perspective. Genetics. 2008;180(2):977993.

11

CoopG, WitonskyD, RienzoAD, PritchardJK. Using Environmental Correlations to Identify Loci Underlying Local Adaptation. Genetics. 2010;185(4):14111423.

12

ThompsonEA. The estimation of pairwise relationships. Ann Hum Genet. 1975;39(2):173188.

13

MilliganBG. Maximum-likelihood estimation of relatedness. Genetics. 2003;163(3):11531167.

14

JacquardA. Structures génétiques des populations. Paris: Masson et Cie; 1970.

15

CsűrösM. Non-identifiability of identity coefficients at biallelic loci. Theor Popul Biol. 2014;92:2229.

16

AstleW, BaldingDJ. Population Structure and Cryptic Relatedness in Genetic Association Studies. Statist Sci. 2009;24(4):451471.

17

WeirBS, CockerhamCC. Estimating F-Statistics for the Analysis of Population Structure. Evolution. 1984;38(6):13581370.

18

WeirBS, CardonLR, AndersonAD, NielsenDM, HillWG. Measures of human population structure show heterogeneity among genomic regions. Genome Res. 2005;15(11):14681476.

19

BuckletonJ, CurranJ, GoudetJ, TaylorD, ThieryA, WeirBS. Population-specific FST values for forensic STR markers: A worldwide survey. Forensic Science International: Genetics. 2016;23:91100.

20

WeirB, ZhengX. SNPs and SNVs in forensic science. Forensic Science International: Genetics Supplement Series. 2015;5(Dec):e267e268.

21

WeirBS, GoudetJ. A Unified Characterization of Population Structure and Relatedness. Genetics. 2017;206(4):20852103.

22

GoudetJ, KayT, WeirBS. How to estimate kinship. Mol Ecol. 2018;27(20):41214135.

23

BhatiaG, PattersonN, SankararamanS, PriceAL. Estimating and interpreting FST: the impact of rare variants. Genome Res. 2013;23(9):15141521.

24

XieC, GesslerDD, XuS. Combining different line crosses for mapping quantitative trait loci using the identical by descent-based variance component method. Genetics. 1998;149(2):11391146.

25

YuJ, PressoirG, BriggsWH, Vroh BiI, YamasakiM, DoebleyJF, et al A unified mixed-model method for association mapping that accounts for multiple levels of relatedness. Nat Genet. 2006;38(2):203208. doi: 10.1038/ng1702

26

AulchenkoYS, de KoningDJ, HaleyC. Genomewide rapid association using mixed model and regression: a fast and simple method for genomewide pedigree-based quantitative trait loci association analysis. Genetics. 2007;177(1):577585.

27

PriceAL, PattersonNJ, PlengeRM, WeinblattME, ShadickNA, ReichD. Principal components analysis corrects for stratification in genome-wide association studies. Nat Genet. 2006;38(8):904909. doi: 10.1038/ng1847

28

KangHM, ZaitlenNA, WadeCM, KirbyA, HeckermanD, DalyMJ, et al Efficient control of population structure in model organism association mapping. Genetics. 2008;178(3):17091723.

29

KangHM, SulJH, ServiceSK, ZaitlenNA, KongSY, FreimerNB, et al Variance component model to account for sample structure in genome-wide association studies. Nat Genet. 2010;42(4):348354. doi: 10.1038/ng.548

30

ZhouX, StephensM. Genome-wide efficient mixed-model analysis for association studies. Nat Genet. 2012;44(7):821824. doi: 10.1038/ng.2310

31

YangJ, BenyaminB, McEvoyBP, GordonS, HendersAK, NyholtDR, et al Common SNPs explain a large proportion of the heritability for human height. Nat Genet. 2010;42(7):565569. doi: 10.1038/ng.608

32

YangJ, LeeSH, GoddardME, VisscherPM. GCTA: a tool for genome-wide complex trait analysis. Am J Hum Genet. 2011;88(1):7682.

33

RakovskiCS, StramDO. A kinship-based modification of the armitage trend test to address hidden population structure and small differential genotyping errors. PLoS ONE. 2009;4(6):e5825

34

ThorntonT, McPeekMS. ROADTRIPS: case-control association testing with partially or completely unknown population and pedigree structure. Am J Hum Genet. 2010;86(2):172184.

35

SpeedD, BaldingDJ. Relatedness in the post-genomic era: is it still useful? Nat Rev Genet. 2015;16(1):3344. doi: 10.1038/nrg3821

36

WangB, SverdlovS, ThompsonE. Efficient Estimation of Realized Kinship from SNP Genotypes. Genetics. 2017; p. genetics.116.197004.

37

WrightS. Systems of Mating. V. General Considerations. Genetics. 1921;6(2):167178.

38

LushJL. Heritability of Quantitative Characters in Farm Animals. Hereditas. 1949;35(S1):356375.

39

FalconerDS, MackayTFC. Introduction to Quantitative Genetics. 4th ed Harlow: Pearson; 1996.

40

ThompsonEA. Identity by descent: variation in meiosis, across genomes, and in populations. Genetics. 2013;194(2):301326.

41

SlatkinM. Inbreeding coefficients and coalescence times. Genetics Research. 1991;58(2):167175.

42

EmikLO, TerrillCE. Systematic procedures for calculating inbreeding coefficients. J Hered. 1949;40(2):5155.

43

García-CortésLA. A novel recursive algorithm for the calculation of the detailed identity coefficients. Genetics Selection Evolution. 2015;47(1):33

44

RosenbergNA, PritchardJK, WeberJL, CannHM, KiddKK, ZhivotovskyLA, et al Genetic Structure of Human Populations. Science. 2002;298(5602):23812385.

45

RamachandranS, DeshpandeO, RosemanCC, RosenbergNA, FeldmanMW, Cavalli-SforzaLL. Support from the relationship of genetic and geographic distance in human populations for a serial founder effect originating in Africa. Proc Natl Acad Sci U S A. 2005;102(44):1594215947.

46

Consortium TGP. A map of human genome variation from population-scale sequencing. Nature. 2010;467(7319):10611073.

47

LazaridisI, PattersonN, MittnikA, RenaudG, MallickS, KirsanowK, et al Ancient human genomes suggest three ancestral populations for present-day Europeans. Nature. 2014;513(7518):409413.

48

LazaridisI, NadelD, RollefsonG, MerrettDC, RohlandN, MallickS, et al Genomic insights into the origin of farming in the ancient Near East. Nature. 2016;536(7617):419424.

49

SkoglundP, PosthC, SirakK, SpriggsM, ValentinF, BedfordS, et al Genomic insights into the peopling of the Southwest Pacific. Nature. 2016;538(7626):510513.

50

TishkoffSA, ReedFA, FriedlaenderFR, EhretC, RanciaroA, FromentA, et al The Genetic Structure and History of Africans and African Americans. Science. 2009;324(5930):10351044.

51

Moreno-EstradaA, GravelS, ZakhariaF, McCauleyJL, ByrnesJK, GignouxCR, et al Reconstructing the Population Genetic History of the Caribbean. PLOS Genetics. 2013;9(11):e1003925

52

Moreno-EstradaA, GignouxCR, Fernández-LópezJC, ZakhariaF, SikoraM, ContrerasAV, et al The genetics of Mexico recapitulates Native American substructure and affects biomedical traits. Science. 2014;344(6189):12801285.

53

LeslieS, WinneyB, HellenthalG, DavisonD, BoumertitA, DayT, et al The fine-scale genetic structure of the British population. Nature. 2015;519(7543):309314.

54

BaharianS, BarakattM, GignouxCR, ShringarpureS, ErringtonJ, BlotWJ, et al The Great Migration and African-American Genomic Diversity. PLoS Genet. 2016;12(5):e1006059

55

HaakW, LazaridisI, PattersonN, RohlandN, MallickS, LlamasB, et al Massive migration from the steppe was a source for Indo-European languages in Europe. Nature. 2015;522(7555):207211.

56

AllentoftME, SikoraM, SjögrenKG, RasmussenS, RasmussenM, StenderupJ, et al Population genomics of Bronze Age Eurasia. Nature. 2015;522(7555):167172.

57

Ochoa A, Storey JD. FST and kinship for arbitrary population structures I: Generalized definitions. bioRxiv. 2016; doi: 10.1101/083915

58

Ochoa A, Storey JD. FST and kinship for arbitrary population structures II: Method of moments estimators. bioRxiv. 2016; doi: 10.1101/083923

59

Ochoa A, Storey JD. New kinship and FST estimates reveal higher levels of differentiation in the global human population. bioRxiv. 2019; doi: 10.1101/653279

60

ThorntonT, TangH, HoffmannTJ, Ochs-BalcomHM, CaanBJ, RischN. Estimating kinship in admixed populations. Am J Hum Genet. 2012;91(1):122138.

61

HaoW, SongM, StoreyJD. Probabilistic models of genetic variation in structured populations applied to global human studies. Bioinformatics. 2016;32(5):713721.

62

ZhengX, WeirBS. Eigenanalysis of SNP data with an identity by descent interpretation. Theoretical Population Biology. 2016;107:6576.

63

PritchardJK, StephensM, DonnellyP. Inference of population structure using multilocus genotype data. Genetics. 2000;155(2):945959.

64

TangH, PengJ, WangP, RischNJ. Estimation of individual admixture: analytical and study design considerations. Genet Epidemiol. 2005;28(4):289301.

65

AlexanderDH, NovembreJ, LangeK. Fast model-based estimation of ancestry in unrelated individuals. Genome Res. 2009;19(9):16551664.

66

BrowningBL, BrowningSR. A Fast, Powerful Method for Detecting Identity by Descent. The American Journal of Human Genetics. 2011;88(2):173182.

67

GazalS, SahbatouM, PerdryH, LetortS, GéninE, LeuteneggerAL. Inbreeding Coefficient Estimation with Dense SNP Data: Comparison of Strategies and Application to HapMap III. HHE. 2014;77(1-4):4962.

68

JoshiPK, EskoT, MattssonH, EklundN, GandinI, NutileT, et al Directional dominance on stature and cognition in diverse human populations. Nature. 2015;523(7561):459462.

69

CochranWG. Sampling techniques. 3rd ed Wiley; 1977.

70

ReynoldsJ, WeirBS, CockerhamCC. Estimation of the Coancestry Coefficient: Basis for a Short-Term Genetic Distance. Genetics. 1983;105(3):767779.

71

WeirBS. Genetic data analysis II Methods for discrete population genetic data. Sunderland, USA: Sinauer Associates; 1996.

72

BourgainC, HoffjanS, NicolaeR, NewmanD, SteinerL, WalkerK, et al Novel case-control test in a founder population identifies P-selectin as an atopy-susceptibility locus. Am J Hum Genet. 2003;73(3):612626. doi: 10.1086/378208

73

ChoiY, WijsmanEM, WeirBS. Case-Control Association Testing in the Presence of Unknown Relationships. Genet Epidemiol. 2009;33(8):668678.

74

PickrellJK, PritchardJK. Inference of population splits and mixtures from genome-wide allele frequency data. PLoS Genet. 2012;8(11):e1002967

75

NeiM. Analysis of Gene Diversity in Subdivided Populations. PNAS. 1973;70(12):33213323.

76

Weir BS, Goudet J. A unified characterization of population structure and relatedness. bioRxiv. 2016; p. 088260.

77

RajA, StephensM, PritchardJK. fastSTRUCTURE: variational inference of population structure in large SNP data sets. Genetics. 2014;197(2):573589.

78

NelisM, EskoT, MägiR, ZimprichF, ZimprichA, TonchevaD, et al Genetic Structure of Europeans: A View from the North–East. PLOS ONE. 2009;4(5):e5472

78

SilvaNM, PereiraL, PoloniES, CurratM. Human Neutral Genetic Variation and Forensic STR Data. PLOS ONE. 2012;7(11):e49666

80

SteeleCD, CourtDS, BaldingDJ. Worldwide FST Estimates Relative to Five Continental-Scale Populations. Annals of Human Genetics. 2014;78(6):468477.

81

Cavalli-SforzaLL. Population Structure and Human Evolution. Proceedings of the Royal Society of London Series B, Biological Sciences. 1966;164(995):362379.

82

LewontinRC, KrakauerJ. Distribution of Gene Frequency as a Test of the Theory of the Selective Neutrality of Polymorphisms. Genetics. 1973;74(1):175195.

83

BeaumontMA, NicholsRA. Evaluating Loci for Use in the Genetic Analysis of Population Structure. Proceedings of the Royal Society of London B: Biological Sciences. 1996;263(1377):16191626.

84

VitalisR, DawsonK, BoursotP. Interpretation of Variation Across Marker Loci as Evidence of Selection. Genetics. 2001;158(4):18111823.

85

AkeyJM, ZhangG, ZhangK, JinL, ShriverMD. Interrogating a high-density SNP map for signatures of natural selection. Genome Res. 2002;12(12):18051814.

86

PorterAH. A test for deviation from island-model population structure. Molecular Ecology. 2003;12(4):903915.

87

BowcockAM, KiddJR, MountainJL, HebertJM, CarotenutoL, KiddKK, et al Drift, admixture, and selection in human evolution: a study with DNA polymorphisms. PNAS. 1991;88(3):839843.

88

HedrickPW. A Standardized Genetic Differentiation Measure. Evolution. 2005;59(8):16331638.

89

JakobssonM, EdgeMD, RosenbergNA. The Relationship Between FST and the Frequency of the Most Frequent Allele. Genetics. 2013;193(2):515528.

90

EdgeMD, RosenbergNA. Upper bounds on FST in terms of the frequency of the most frequent allele and total homozygosity: the case of a specified number of alleles. Theor Popul Biol. 2014;97:2034.

91

LewontinRC. The Apportionment of Human Diversity. Evolutionary Biology. 1972;6:381398.

92

BarbujaniG, MagagniA, MinchE, Cavalli-SforzaLL. An apportionment of human DNA diversity. PNAS. 1997;94(9):45164519.

93

NovembreJ, JohnsonT, BrycK, KutalikZ, BoykoAR, AutonA, et al Genes mirror geography within Europe. Nature. 2008;456(7218):98101.

94

CoopG, PickrellJK, NovembreJ, KudaravalliS, LiJ, AbsherD, et al The Role of Geography in Human Adaptation. PLoS Genet. 2009;5(6):e1000500

95

PattersonN, MoorjaniP, LuoY, MallickS, RohlandN, ZhanY, et al Ancient admixture in human history. Genetics. 2012;192(3):10651093.

96

BeranR, HallP. Interpolated Nonparametric Prediction Intervals and Confidence Intervals. Journal of the Royal Statistical Society Series B (Methodological). 1993;55(3):643652.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Citing articles via
https://www.researchpad.co/tools/openurl?pubtype=article&doi=10.1371/journal.pgen.1009241&title=Estimating <i>F</i><sub>ST</sub> and kinship for arbitrary population structures&author=&keyword=&subject=Research Article,Biology and Life Sciences,Genetics,Heredity,Inbreeding,Physical Sciences,Mathematics,Statistics,Statistical Models,Biology and Life Sciences,Computational Biology,Genome Analysis,Genome-Wide Association Studies,Biology and Life Sciences,Genetics,Genomics,Genome Analysis,Genome-Wide Association Studies,Biology and Life Sciences,Genetics,Human Genetics,Genome-Wide Association Studies,Research and Analysis Methods,Simulation and Modeling,Biology and Life Sciences,Genetics,Heredity,Genetic Mapping,Variant Genotypes,Biology and Life Sciences,Genetics,Heredity,Biology and Life Sciences,Evolutionary Biology,Population Genetics,Biology and Life Sciences,Genetics,Population Genetics,Biology and Life Sciences,Population Biology,Population Genetics,Biology and Life Sciences,Genetics,Genetic Loci,