quantile
Quantiles of a data set
Syntax
Description
Y = quantile(X,p)X for the
cumulative probability or probabilities p in the interval [0,1].
If
Xis a vector, thenYis a scalar or a vector having the same length asp.If
Xis a matrix, thenYis a row vector or a matrix where the number of rows ofYis equal to the length ofp.For multidimensional arrays,
quantileoperates along the first nonsingleton dimension ofX.
Y = quantile(X,N)N evenly spaced cumulative probabilities
(1/(N + 1), 2/(N + 1), ...,
N/(N + 1)) for integer N>1.
If
Xis a vector, thenYis a scalar or a vector with lengthN.If
Xis a matrix, thenYis a matrix where the number of rows ofYis equal toN.For multidimensional arrays,
quantileoperates along the first nonsingleton dimension ofX.
Y = quantile(___,vecdim)vecdim
for either of the first two syntaxes. For example, if X is a matrix,
then quantile(X,0.5,[1 2]) returns the 0.5 quantile of all the elements
of X because every element of a matrix is contained in the array
slice defined by dimensions 1 and 2.
Examples
Input Arguments
Output Arguments
More About
T-Digest
T-digest[2] is a probabilistic data structure that is a sparse representation of the empirical cumulative distribution function (CDF) of a data set. T-digest is useful for computing approximations of rank-based statistics (such as percentiles and quantiles) from online or distributed data in a way that allows for controllable accuracy, particularly near the tails of the data distribution.
For data that is distributed in different partitions, t-digest computes quantile estimates (and percentile estimates) for each data partition separately, and then combines the estimates while maintaining a constant-memory bound and constant relative accuracy of computation ( for the qth quantile). For these reasons, t-digest is practical for working with tall arrays.
To estimate quantiles of an array that is distributed in different partitions, first build a
t-digest in each partition of the data. A t-digest clusters the data in the partition and
summarizes each cluster by a centroid value and an accumulated weight that represents the
number of samples contributing to the cluster. T-digest uses large clusters (widely spaced
centroids) to represent areas of the CDF that are near q =
0.5 and uses small clusters (tightly spaced centroids) to represent areas of
the CDF that are near q = 0 or
q = 1.
T-digest controls the cluster size by using a scaling function that maps a quantile q to an index k with a compression parameter . That is,
where the mapping k is monotonic with minimum value k(0,δ) = 0 and maximum value k(1,δ) = δ. The following figure shows the scaling function for δ = 10.
The scaling function translates the quantile q to the scaling factor
k in order to give variable size steps in q. As a
result, cluster sizes are unequal (larger around the center quantiles and smaller near
q = 0 or q =
1). The smaller clusters allow for better accuracy near the edges of the
data.
To update a t-digest with a new observation that has a weight and location, find the cluster closest to the new observation. Then, add the weight and update the centroid of the cluster based on the weighted average, provided that the updated weight of the cluster does not exceed the size limitation.
You can combine independent t-digests from each partition of the data by taking a union of the t-digests and merging their centroids. To combine t-digests, first sort the clusters from all the independent t-digests in decreasing order of cluster weights. Then, merge neighboring clusters, when they meet the size limitation, to form a new t-digest.
Once you form a t-digest that represents the complete data set, you can estimate the end-points (or boundaries) of each cluster in the t-digest and then use interpolation between the end-points of each cluster to find accurate quantile estimates.
Algorithms
For an n-element vector X, quantile
computes quantiles by using a sorting-based algorithm as follows:
The sorted elements in
Xare taken as the (0.5/n), (1.5/n), ..., ([n – 0.5]/n) quantiles. For example:For a data vector of five elements such as {6, 3, 2, 10, 1}, the sorted elements {1, 2, 3, 6, 10} respectively correspond to the 0.1, 0.3, 0.5, 0.7, 0.9 quantiles.
For a data vector of six elements such as {6, 3, 2, 10, 8, 1}, the sorted elements {1, 2, 3, 6, 8, 10} respectively correspond to the (0.5/6), (1.5/6), (2.5/6), (3.5/6), (4.5/6), (5.5/6) quantiles.
quantileuses Linear Interpolation to compute quantiles for probabilities between (0.5/n) and ([n – 0.5]/n).For the quantiles corresponding to the probabilities outside that range,
quantileassigns the minimum or maximum values of the elements inX.
quantile treats NaNs
as missing values and removes them.
References
[1] Langford, E. “Quartiles in Elementary Statistics”, Journal of Statistics Education. Vol. 14, No. 3, 2006.