({{, (RefLink[Dataset, paclet:ref/Dataset])[data] 
represents a structured dataset based on a hierarchy of lists and associations.}})

OpenersNotesSection(Spacer1)Details

(RefLink[Dataset, paclet:ref/Dataset]) can represent not only full rectangular multidimensional arrays of data, but also arbitrary tree structures, corresponding to data with arbitrary hierarchical structure.

Depending on the data it contains, a (RefLink[Dataset, paclet:ref/Dataset]) object typically displays as a table or grid of elements.

Functions like (RefLink[Map, paclet:ref/Map]), (RefLink[Select, paclet:ref/Select]), etc. can be applied directly to a (RefLink[Dataset, paclet:ref/Dataset]) by writing (RefLink[Map, paclet:ref/Map])[f,dataset], (RefLink[Select, paclet:ref/Select])[dataset,crit], etc.

Subsets of the data in a (RefLink[Dataset, paclet:ref/Dataset]) object can be obtained by writing dataset[[parts]].

(RefLink[Dataset, paclet:ref/Dataset]) objects can also be queried using a specialized query syntax by writing dataset[query].

While arbitrary nesting of lists and associations is possible, two-dimensional (tabular) forms are most commonly used.

The following table shows the correspondence between the common display forms of a (RefLink[Dataset, paclet:ref/Dataset]), the form of Wolfram Language expression it contains, and logical interpretation of its structure as a table:

(RefLink[Dataset, paclet:ref/Dataset]) interprets nested lists and associations in a row-wise fashion, so that level 1 (the outermost level) of the data is interpreted as the rows of a table, and level 2 is interpreted as the columns.

Named rows and columns correspond to associations at level 1 and 2, respectively, whose keys are strings that contain the names. Unnamed rows and columns correspond to lists at those levels.

Rows and columns of a dataset can be exchanged by writing (RefLink[Transpose, paclet:ref/Transpose])[dataset].

The syntax dataset[[parts]] or (RefLink[Part, paclet:ref/Part])[dataset,parts] can be used to extract parts of a (RefLink[Dataset, paclet:ref/Dataset]).

The parts that can be extracted from a (RefLink[Dataset, paclet:ref/Dataset]) include all ordinary specifications for (RefLink[Part, paclet:ref/Part]).

Unlike the ordinary behavior of (RefLink[Part, paclet:ref/Part]), if a specified subpart of a (RefLink[Dataset, paclet:ref/Dataset]) is not present, (RefLink[Missing, paclet:ref/Missing])["PartAbsent",…] will be produced in that place in the result.

The following part operations are commonly used to extract rows from tabular datasets:

The following part operations are commonly used to extract columns from tabular datasets:

Like (RefLink[Part, paclet:ref/Part]), row and column operations can be combined. Some examples include:

The following operations can be used to remove the labels from rows and columns, effectively turning associations into lists:

The query syntax dataset[(Subscript[op, 1]),(Subscript[op, 2]),…] can be thought of as an extension of (RefLink[Part, paclet:ref/Part]) syntax to allow aggregations and transformations to be applied, as well as taking subsets of data.

Some common forms of query include:

Some more specialized forms of query include:

In dataset[(Subscript[op, 1]),(Subscript[op, 2]),…], the query operators (Subscript[op, i]) are effectively applied at successively deeper levels of the data, but any given one may be applied either while "descending" into the data or while "ascending" out of it.

The operators that make up a (RefLink[Dataset, paclet:ref/Dataset]) query fall into one of the following broad categories with distinct ascending and descending behavior:

A descending operator is applied to corresponding parts of the original dataset, before subsequent operators are applied at deeper levels.

Descending operators have the feature that they do not change the structure of deeper levels of the data when applied at a certain level. This ensures that subsequent operators will encounter subexpressions whose structure is identical to the corresponding levels of the original dataset.

The simplest descending operator is (RefLink[All, paclet:ref/All]), which selects all parts at a given level and therefore leaves the structure of the data at that level unchanged. (RefLink[All, paclet:ref/All]) can safely be replaced with any other descending operator to yield another valid query.

An ascending operator is applied after all subsequent ascending and descending operators have been applied to deeper levels. Whereas descending operators correspond to the levels of the original data, ascending operators correspond to the levels of the result.

Unlike descending operators, ascending operators do not necessarily preserve the structure of the data they operate on. Unless an operator is specifically recognized to be descending, it is assumed to be ascending.

The descending part operators specify which elements to take at a level before applying any subsequent operators to deeper levels:

The descending filtering operators specify how to rearrange or filter elements at a level before applying subsequent operators to deeper levels:

The syntax (Subscript[op, 1])/*(Subscript[op, 2]) can be used to combine two or more filtering operators into one operator that still operates at a single level.

The ascending aggregation operators combine or summarize the results of applying subsequent operators to deeper levels:

The syntax (Subscript[op, 1])/*(Subscript[op, 2]) can be used to combine two or more aggregation operators into one operator that still operates at a single level.

The ascending subquery operators perform a subquery after applying subsequent operators to deeper levels:

When one or more descending operators are composed with one or more ascending operators (e.g. desc/*asc), the descending part will be applied, then subsequent operators will be applied to deeper levels, and lastly, the ascending part will be applied to the result at that level.

The special descending operator (RefLink[GroupBy, paclet:ref/GroupBy])[spec] will introduce a new association at the level at which it appears and can be inserted or removed from an existing query without affecting subsequent operators.

Functions such as (RefLink[CountsBy, paclet:ref/CountsBy]), (RefLink[GroupBy, paclet:ref/GroupBy]), and (RefLink[TakeLargestBy, paclet:ref/TakeLargestBy]) normally take another function as one of their arguments. When working with associations in a (RefLink[Dataset, paclet:ref/Dataset]), it is common to use this "by" function to look up the value of a column in a table.

To facilitate this, (RefLink[Dataset, paclet:ref/Dataset]) queries allow the syntax "string" to mean (RefLink[Key, paclet:ref/Key])["string"] in such contexts. For example, the query operator (RefLink[GroupBy, paclet:ref/GroupBy])["string"] is automatically rewritten to (RefLink[GroupBy, paclet:ref/GroupBy])[(RefLink[Key, paclet:ref/Key])["string"]] before being executed.

Similarly, the expression (RefLink[GroupBy, paclet:ref/GroupBy])[dataset,"string"] is rewritten as (RefLink[GroupBy, paclet:ref/GroupBy])[dataset,(RefLink[Key, paclet:ref/Key])["string"]].

Where possible, type inference is used to determine whether a query will succeed. Operations that are inferred to fail will result in a (RefLink[Failure, paclet:ref/Failure]) object being returned without the query being performed.

By default, if any messages are generated during a query, the query will be aborted and a (RefLink[Failure, paclet:ref/Failure]) object containing the message will be returned.

When a query returns structured data (e.g. a list or association, or nested combinations of these), the result will be given in the form of another (RefLink[Dataset, paclet:ref/Dataset]) object. Otherwise, the result will be given as an ordinary Wolfram Language expression.

For more information about special behavior of (RefLink[Dataset, paclet:ref/Dataset]) queries, see the function page for (RefLink[Query, paclet:ref/Query]).

(RefLink[Normal, paclet:ref/Normal]) can be used to convert any (RefLink[Dataset, paclet:ref/Dataset]) object to its underlying data, which is typically a combination of lists and associations.

(RefLink[Dataset, paclet:ref/Dataset]) objects can be exported by writing (RefLink[Export, paclet:ref/Export])["file.ext",dataset] or (RefLink[ExportString, paclet:ref/ExportString])[dataset,"fmt"]. The following formats are supported:

(RefLink[SemanticImport, paclet:ref/SemanticImport]) can be used to import files as (RefLink[Dataset, paclet:ref/Dataset]) objects.

OpenersPrimaryExamplesSection(Spacer1)Examples  (14)

OpenersExampleSection0(Spacer1)Basic Examples  (1)

Create a (RefLink[Dataset, paclet:ref/Dataset]) object from tabular data:

dataset=Dataset[{
<|"a"->1,"b"->"x","c"->{1}|>,
<|"a"->2,"b"->"y","c"->{2,3}|>,
<|"a"->3,"b"->"z","c"->{3}|>,
<|"a"->4,"b"->"x","c"->{4,5}|>,
<|"a"->5,"b"->"y","c"->{5,6,7}|>,
<|"a"->6,"b"->"z","c"->{}|>}]

Take a set of rows:

dataset[1;;3]

Take a specific row:

dataset[2]

A row is merely an association:

dataset[2] // Normal

Take a specific element from a specific row:

dataset[3,"a"]

dataset[4,"b"]

dataset[5,"c"]

Take the contents of a specific column:

dataset[All,"a"]

dataset[All,"b"]

dataset[All,"c"]

Take a specific part within a column:

dataset[All,"c",1]

Take a subset of the rows and columns:

dataset[1;;4,{"a","b"}]

Apply a function to the contents of a specific column:

dataset[Total,"a"]

dataset[StringJoin,"b"]

dataset[Catenate,"c"]

Partition the dataset based on a column, applying further operators to each group:

dataset[GroupBy["b"],Catenate,"c"]

Apply a function to each row:

dataset[All,Reverse]

Apply a function both to each row and to the entire result:

dataset[Reverse, Reverse]

Apply a function f to every element in every row:

dataset[All,All,f]

Apply functions to each column independently:

dataset[All,{"a"->f,"b"->g,"c"->h}]

Construct a new table by specifying operators that will compute each column:

dataset[All,Key["c"]/*<|"ctotal"->Total,"clength"->Length|>]

Use the same technique to rename columns:

dataset[All,<|"A"->"a","B"->"b","C"->"c"|>]

Select specific rows based on a criterion:

dataset[Select[#a<5&]]

Take the contents of a column after selecting the rows:

dataset[Select[#a<5&],"b"]

Take a subset of the available columns after selecting the rows:

dataset[Select[#a<5&],{"b","c"}]

Take the first row satisfying a criterion:

dataset[SelectFirst[#a>5&]]

Take a value from this row:

dataset[SelectFirst[#a>5&], "b"]

Sort the rows by a criterion:

dataset[SortBy[Length[#c]&]]

Take the rows that give the maximal value of a scoring function:

dataset[MaximalBy[Length[#c]&]]

Give the top 3 rows according to a scoring function:

dataset[TakeLargestBy[Length[#c]&,3]]

Delete rows that duplicate a criterion:

dataset[DeleteDuplicatesBy["b"]]

dataset[DeleteDuplicatesBy[Length[#c]&]]

Compose an ascending and a descending operator to aggregate values of a column after filtering the rows:

dataset[Select[#b=="z"&]/*Total,"a"]

Do the same thing by applying (RefLink[Total, paclet:ref/Total]) after the query:

dataset[Select[#b=="z"&],"a"][Total]

OpenersExampleSection2(Spacer1)Applications  (3)

OpenersExampleSubsection0(Spacer1)Tables (Lists of Associations)  (1)

Load a dataset of passengers of the Titanic:

titanic=ExampleData[{"Dataset","Titanic"}]

Get the number of rows:

Length[titanic]

Get a random sample of passengers:

RandomSample[titanic,5]

The underlying data is a list of associations:

Normal[%]

Count the number of passengers with a missing age:

titanic[Count[_Missing],"age"]

Count the number of passengers in 1st, 2nd, and 3rd class:

titanic[Counts,"class"]

Get a histogram of passenger ages:

titanic[Histogram,"age"]

Get a histogram of passenger ages, grouped by passenger class:

titanic[GroupBy["class"],Histogram[#, {0,80,4}]&,"age"]

Find the age of the oldest passenger:

titanic[Max,"age"]

Calculate the overall survival ratio:

ratio[list_] := list // Boole//Mean//N;

titanic[ratio,"survived"]

Show the survival ratio against sex and passenger class:

titanic[GroupBy["sex"],GroupBy["class"],ratio,"survived"]

Show the survival ratio as a function of age:

titanic[GroupBy[Round[#age,5]&],ratio,"survived"]//ListPlot

OpenersExampleSubsection1(Spacer1)Indexed Tables (Associations of Associations)  (1)

Load a dataset of planets and their properties:

planets=ExampleData[{"Dataset","Planets"}]

Look up the mass of the Earth:

planets["Earth","Mass"]

Get the subtable corresponding to moons of a specific planet:

planets["Neptune","Moons"]

Produce a dataset of the radii of the planets:

planets[All,"Radius"]

Visualize the radii of the planets:

planets[BarChart,"Radius"]

Produce a dataset of the number of moons of each planet:

planets[All,"Moons",Length]

PieChart[%,ChartLegends->Automatic]

Obtain a list of the planets and their masses, sorted by increasing mass:

planets[SortBy["Mass"],"Mass"]

Find the total mass of each planet's moons:

planets[All,"Moons",Total,"Mass"]

Obtain a list of only those moons that have a mass larger than half that of Earth's moon:

moonMass = planets["Earth","Moons","Moon","Mass"];
planets[All,"Moons",Select[#Mass > moonMass/2&]/*Keys]

Find the heaviest moon of each planet:

planets[All,"Moons",MaximalBy["Mass"]/*Keys]

Obtain a list of all planetary moons:

moons=planets[Catenate,"Moons"]

Make a scatter plot of the mass against radius:

moons[ListLogLogPlot,{#Mass,#Radius}&]

Calculate and make a histogram of the densities:

sphereVolume=LinguisticAssistant;
moons[Histogram,#Mass /sphereVolume[#Radius]&]

Compute the mean density for the moons of each planet:

planets[All,"Moons",Mean,#Mass/sphereVolume[#Radius]&]

Create a table comparing the density of each planet with the mean density of its moons:

density=#Mass/sphereVolume[#Radius]&;

planets[All,<|"Planet Density" -> density,"Moon Density"->Query["Moons",Mean,density]|>]

OpenersExampleSubsection2(Spacer1)Hierarchical Data (Associations of Associations of Other Data)  (1)

Load a dataset associating countries to their administrative divisions to their populations:

populations=ExampleData[{"Dataset","StatePopulations"}]

The underlying data is an association whose keys are countries and whose values are further associations between administrative divisions and their populations:

Normal @ populations[1;;2,1;;2]

Look up the populations for a specific country:

populations[LinguisticAssistant]

Give the total population (not all countries in the world are included in this dataset):

populations[Total,Total]

Count the number of divisions within each country:

populations[All,Length]

Total the number of divisions:

populations[Total,Length]

Build a histogram of the number of divisions:

populations[Histogram,Length]

Calculate the total population of each country by adding the populations of each division:

populations[All,Total]

Obtain the five most populous countries:

populations[All,Total][Sort][-5;;-1]

Obtain the most populous divisions for each country:

populations[All,MaximalBy[Total]/*Keys/*First]

Correlate the number of divisions a country has with its population:

populations[ListLogPlot,{Length,Total}]

The underlying data being passed to (RefLink[ListLogPlot, paclet:ref/ListLogPlot]) is an association of lists, each of length 2:

populations[1;;5,{Length,Total}]

Normal[%]

OpenersExampleSection3(Spacer1)Properties & Relations  (4)

(RefLink[Query, paclet:ref/Query]) is the operator form of the query language supported by (RefLink[Dataset, paclet:ref/Dataset]):

Query["b",Total] @ <|"a"->{1,2},"b"->{3,4}|>

Dataset[<|"a"->{1,2},"b"->{3,4}|>]["b", Total]

Use (RefLink[EntityValue, paclet:ref/EntityValue]) to obtain a (RefLink[Dataset, paclet:ref/Dataset]) of the properties of (RefLink[Entity, paclet:ref/Entity]) objects from the Wolfram Knowledgebase:

gases=EntityValue[LinguisticAssistant,{"BoilingPoint","Density"},"Dataset"]

Plot the boiling point versus density on a log plot:

ListLogPlot[gases]

Use (RefLink[SemanticImport, paclet:ref/SemanticImport]) to import a file as a (RefLink[Dataset, paclet:ref/Dataset]):

sales=SemanticImport["ExampleData/RetailSales.tsv"]

Calculate the total quantity of sales:

sales[Total,"Sales"]

Obtain a small sample of the Titanic passenger dataset:

sample=RandomSample[ExampleData[{"Dataset","Titanic"}],5]

Export the sample as (StringTypeLink["JSON", paclet:ref/format/JSON]) format:

ExportString[sample,"JSON"]

Data with named columns can be more compactly represented if it is first transposed:

ExportString[Transpose @ sample,"JSON"]

Export the sample as (StringTypeLink["CSV", paclet:ref/format/CSV]):

ExportString[sample,"CSV"]

OpenersExampleSection4(Spacer1)Possible Issues  (4)

Data without a consistent structure will not usually format in the same way as structured data:

Dataset[{<|"A"->1,"B"->2|>,<|"A"->3,"B"->4|>,<|"A"->5,"B"->6|>}]

Dataset[{<|"A"->1,"B"->2|>,<|"A"->3,"B"->4|>,{5,6}}]

If a sub-operation of a query is inferred to fail, the entire query will not be performed, and a (RefLink[Failure, paclet:ref/Failure]) object will be returned:

Dataset[<|"A"->1,"B"->2|>]["C"]

By default, if any messages are generated during an operation on a (RefLink[Dataset, paclet:ref/Dataset]), a (RefLink[Failure, paclet:ref/Failure]) object will be returned:

Dataset[{"abc","abdefg","ab"}][All,StringTake[#,3]&]

To specify different behavior, use an explicit (RefLink[Query, paclet:ref/Query]) expression in conjunction with the option (RefLink[FailureAction, paclet:ref/FailureAction]):

Query[All,StringTake[#,3]&,FailureAction->"Replace"] @ Dataset[{"abc","abdefg","ab"}]

By default, a limited number of elements will be shown in a (RefLink[Dataset, paclet:ref/Dataset]):

Dataset[Range[300]]

To see more elements, set the value of Dataset`$DatasetTargetRowCount to a value higher than the default of 20:

Dataset`$DatasetTargetRowCount=40;

Dataset[Range[300]]

Note that this functionality may change in the future.

OpenersExampleSection5(Spacer1)Neat Examples  (2)

Calculate the survival likelihood of the characters Jack Dawson and Rose DeWitt Bukater from the movie Titanic by matching them with real passengers:

titanic= ExampleData[{"Dataset","Titanic"}];

characters=<|
"Rose"-><|"class"->"1st","age"->17,"sex"->"female"|>,
"Jack"-><|"class"->"3rd","age"->20,"sex"->"male"|>
|>;

matches[target_][passenger_]:=target==KeyDrop[passenger,"survived"];

titanic[Select[matches[#]],"survived"]&/@ characters

Map[Normal/*Boole/*Mean,%]

Make an interactive control that plots a heat map of a country's divisions when a country is selected:

populations=ExampleData[{"Dataset","StatePopulations"}];

brics=populations[KeyTake[EntityList[LinguisticAssistant]]];

brics[TabView,GeoRegionValuePlot[#,ImageSize->Medium,GeoBackground->None]&]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[Association, paclet:ref/Association])  ▪  (RefLink[List, paclet:ref/List])  ▪  (RefLink[Part, paclet:ref/Part])  ▪  (RefLink[SemanticImport, paclet:ref/SemanticImport])  ▪  (RefLink[Query, paclet:ref/Query])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Applying Functions to Lists, paclet:guide/ApplyingFunctionsToLists])}, {▪(RefLinkPlain[Computation with Structured Datasets, paclet:guide/ComputationWithStructuredDatasets])}, {▪(RefLinkPlain[Database Connectivity, paclet:guide/DatabaseConnectivity])}, {▪(RefLinkPlain[Handling Arrays of Data, paclet:guide/HandlingArraysOfData])}, {▪(RefLinkPlain[Automated Reports, paclet:guide/AutomatedReports])}, {▪(RefLinkPlain[Database-Like Operations on Datasets, paclet:guide/DatabaseLikeOperationsOnDatasets])}, {▪(RefLinkPlain[WDF (Wolfram Data Framework), paclet:guide/WDFWolframDataFramework])}, {▪(RefLinkPlain[Associations, paclet:guide/Associations])}, {▪(RefLinkPlain[Scientific Data Analysis, paclet:guide/ScientificDataAnalysis])}, {▪(RefLinkPlain[Expressions, paclet:guide/Expressions])}, {▪(RefLinkPlain[Wolfram Data Repository, paclet:guide/WolframDataRepository])}})}})

({{(Spacer1), ({{(Spacer1)Related Links}, {(WebLink[DocumentationFooterIconsRelatedLinks-EIWL.png(Spacer1)An Elementary Introduction to the Wolfram Language, https://www.wolfram.com/language/elementary-introduction/45-datasets.html])(WebLink[: Datasets, https://www.wolfram.com/language/elementary-introduction/45-datasets.html])}})}})