parquetread

Read columnar data from a Parquet file

collapse all in page

Syntax

T = parquetread(filename)
T = parquetread(filename,Name,Value)

Description

example

T = parquetread(filename) reads the Parquet file specified in filename into a table or timetable T.

example

T = parquetread(filename,Name,Value) reads the Parquet file into a table or timetable with additional options specified by one or more name-value pair arguments.

Examples

collapse all

Open Live Script

Get information about a Parquet file, read the data from the file into a table, and then read a subset of the variables into a table.

Create a ParquetInfo object for the file outages.parquet.

info = parquetinfo('outages.parquet')
info = 
  ParquetInfo with properties:

               Filename: "/mathworks/devel/bat/BR2020bd/build/matlab/toolbox/matlab/demos/outages.parquet"
               FileSize: 44202
           NumRowGroups: 1
        RowGroupHeights: 1468
          VariableNames: [1x6 string]
          VariableTypes: [1x6 string]
    VariableCompression: [1x6 string]
       VariableEncoding: [1x6 string]
                Version: "2.0"

Read data from the file into a table and display the first 10 rows.

T = parquetread('outages.parquet');
T(1:10,:) 
ans=10×6 table
      Region            OutageTime          Loss     Customers       RestorationTime             Cause      
    ___________    ____________________    ______    __________    ____________________    _________________

    "SouthWest"    01-Feb-2002 12:18:00    458.98    1.8202e+06    07-Feb-2002 16:50:00    "winter storm"   
    "SouthEast"    23-Jan-2003 00:49:00    530.14    2.1204e+05                     NaT    "winter storm"   
    "SouthEast"    07-Feb-2003 21:15:00     289.4    1.4294e+05    17-Feb-2003 08:14:00    "winter storm"   
    "West"         06-Apr-2004 05:44:00    434.81    3.4037e+05    06-Apr-2004 06:10:00    "equipment fault"
    "MidWest"      16-Mar-2002 06:18:00    186.44    2.1275e+05    18-Mar-2002 23:23:00    "severe storm"   
    "West"         18-Jun-2003 02:49:00         0             0    18-Jun-2003 10:54:00    "attack"         
    "West"         20-Jun-2004 14:39:00    231.29           NaN    20-Jun-2004 19:16:00    "equipment fault"
    "West"         06-Jun-2002 19:28:00    311.86           NaN    07-Jun-2002 00:51:00    "equipment fault"
    "NorthEast"    16-Jul-2003 16:23:00    239.93         49434    17-Jul-2003 01:12:00    "fire"           
    "MidWest"      27-Sep-2004 11:09:00    286.72         66104    27-Sep-2004 16:37:00    "equipment fault"

Select and import the variables Region, OutageTime, and Cause into a table and display the first 10 rows.

SelVarNames = {'Region','OutageTime','Cause'};
T_subset = parquetread('outages.parquet','SelectedVariableNames',SelVarNames);
T_subset(1:10,:) 
ans=10×3 table
      Region            OutageTime               Cause      
    ___________    ____________________    _________________

    "SouthWest"    01-Feb-2002 12:18:00    "winter storm"   
    "SouthEast"    23-Jan-2003 00:49:00    "winter storm"   
    "SouthEast"    07-Feb-2003 21:15:00    "winter storm"   
    "West"         06-Apr-2004 05:44:00    "equipment fault"
    "MidWest"      16-Mar-2002 06:18:00    "severe storm"   
    "West"         18-Jun-2003 02:49:00    "attack"         
    "West"         20-Jun-2004 14:39:00    "equipment fault"
    "West"         06-Jun-2002 19:28:00    "equipment fault"
    "NorthEast"    16-Jul-2003 16:23:00    "fire"           
    "MidWest"      27-Sep-2004 11:09:00    "equipment fault"
Open Live Script

Read the data from the file into a timetable, and then use timetable functions to determine if the timetable regular and sorted.

Read data from outages.parquet into a timetable and display the first 10 rows. Use the second variable OutageTime in the data as the time vector for the timetable. 

TT = parquetread('outages.parquet','RowTimes','OutageTime');
TT(1:10,:)
ans=10×5 timetable
         OutageTime           Region        Loss     Customers       RestorationTime             Cause      
    ____________________    ___________    ______    __________    ____________________    _________________

    01-Feb-2002 12:18:00    "SouthWest"    458.98    1.8202e+06    07-Feb-2002 16:50:00    "winter storm"   
    23-Jan-2003 00:49:00    "SouthEast"    530.14    2.1204e+05                     NaT    "winter storm"   
    07-Feb-2003 21:15:00    "SouthEast"     289.4    1.4294e+05    17-Feb-2003 08:14:00    "winter storm"   
    06-Apr-2004 05:44:00    "West"         434.81    3.4037e+05    06-Apr-2004 06:10:00    "equipment fault"
    16-Mar-2002 06:18:00    "MidWest"      186.44    2.1275e+05    18-Mar-2002 23:23:00    "severe storm"   
    18-Jun-2003 02:49:00    "West"              0             0    18-Jun-2003 10:54:00    "attack"         
    20-Jun-2004 14:39:00    "West"         231.29           NaN    20-Jun-2004 19:16:00    "equipment fault"
    06-Jun-2002 19:28:00    "West"         311.86           NaN    07-Jun-2002 00:51:00    "equipment fault"
    16-Jul-2003 16:23:00    "NorthEast"    239.93         49434    17-Jul-2003 01:12:00    "fire"           
    27-Sep-2004 11:09:00    "MidWest"      286.72         66104    27-Sep-2004 16:37:00    "equipment fault"

Determine if the timetable is regular and sorted. A regular timetable has the same time interval between consecutive row times and a sorted timetable has a row time vector is in ascending order.

isregular(TT)
ans = logical
   0


issorted(TT)
ans = logical
   0

Sort the timetable on its row times using the sortrows function and display the first 10 rows of the sorted data.

TT = sortrows(TT);
TT(1:10,:)
ans=10×5 timetable
         OutageTime           Region        Loss     Customers       RestorationTime             Cause       
    ____________________    ___________    ______    __________    ____________________    __________________

    01-Feb-2002 12:18:00    "SouthWest"    458.98    1.8202e+06    07-Feb-2002 16:50:00    "winter storm"    
    05-Mar-2002 17:53:00    "MidWest"      96.563    2.8666e+05    10-Mar-2002 14:41:00    "wind"            
    16-Mar-2002 06:18:00    "MidWest"      186.44    2.1275e+05    18-Mar-2002 23:23:00    "severe storm"    
    26-Mar-2002 01:59:00    "MidWest"      388.04    5.6422e+05    28-Mar-2002 19:55:00    "winter storm"    
    20-Apr-2002 16:46:00    "MidWest"       23141           NaN                     NaT    "unknown"         
    08-May-2002 20:34:00    "SouthWest"    50.732         34481    08-May-2002 22:21:00    "thunder storm"   
    18-May-2002 11:04:00    "MidWest"      1389.1    1.3447e+05    21-May-2002 01:22:00    "unknown"         
    20-May-2002 10:57:00    "NorthEast"    9116.6    2.4983e+06    21-May-2002 15:22:00    "unknown"         
    27-May-2002 09:44:00    "SouthEast"    237.28    1.7101e+05    27-May-2002 16:19:00    "wind"            
    02-Jun-2002 16:11:00    "SouthEast"         0             0    05-Jun-2002 05:55:00    "energy emergency"

Input Arguments

collapse all

Name of Parquet file, specified as a character vector or string scalar. parquetread works with Parquet 1.0 or Parquet 2.0 files.

Depending on the location of the file, filename can take on one of these forms.

Location

Form

Current folder or folder on the MATLAB® path

Specify the name of the file in filename.

Example: 'data.parquet'

File in a folder

If the file is not in the current folder or in a folder on the MATLAB path, then specify the full or relative path name.

Example: 'C:\myFolder\data.parquet'

Example: 'myDir\myFile.ext'

Remote Location

If the file is stored at a remote location, then filename must contain the full path of the file specified as a uniform resource locator (URL) of the form:

scheme_name://path_to_file/my_file.ext

Based on your remote location, scheme_name can be one of the values in this table.

Remote Locationscheme_name
Amazon S3™s3
Windows Azure® Blob Storagewasb, wasbs
HDFS™hdfs

For more information, see Work with Remote Data.

Example: 's3://bucketname/path_to_file/data.parquet'

Data Types: char | string

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: 'OutputType','table' imports the data in the Parquet file as a table.

Output datatype, specified as the comma-separated pair consisting of 'OutputType' and 'auto', 'table', or 'timetable'.

  • 'auto' — Return a table or a timetable. The parquetread detects if the output should be a table or a timetable based on other name-value pairs that you specify. For example, when you set timetable related name-value pairs, then parquetread infers that the output is a timetable. Setting these name-value pairs indicates that the output is a timetable: RowTimes, StartTime, SampleRate, or TimeStep.

  • 'table' — Return a table. For more information on the table datatype, see table.

  • 'timetable' — Return a timetable. For more information on the timetable datatype, see timetable.

Example: 'OutputType','timetable'

Data Types: char | string

Subset of variables to import, specified as the comma-separated pair consisting of 'SelectedVariableNames' and a character vector, string scalar, cell array of character vectors, or a string array.

  • SelectedVariableNames must be a subset of variable names contained in the Parquet file. To get the names of all the variables in the file, use the VariableNames property of the ParquetInfo object.

  • If you do not specify the SelectedVariableNames name-value pair, parquetread reads all the variables from the file.

Data Types: char | string | cell

Row times variable, specified as the comma-separated pair consisting of 'RowTimes' and a variable name or a time vector.

  • Variable name must be a character vector or string scalar containing the name of any variable in the input table that contains datetime or duration values. The variable specified by the variable name provides row time labels for the rows. The remaining variables of the input table become the variables of the timetable.

  • Time vector must be a datetime vector or a duration vector. The number of elements of time vector must equal the number of rows of the input table. The time values in the time vector do not need to be unique, sorted, or regular. All the variables of the input table become variables of the timetable.

Data Types: char | string | datetime | duration

Start time of the row times, specified as the comma-separated pair consisting of StartTime and a datetime scalar or duration scalar.

  • If the start time is a datetime, then the row times of T are datetime values.

  • If the start time is a duration, then the row times of T are duration values.

  • If the time step is a calendar duration, then the start time must be a datetime value.

StartTime is a timetable related parameter. The parquetread function uses StartTime along with SampleRate or TimeStep to define the time vector for the output T.

Data Types: datetime | duration

Sample rate, specified as the comma-separated pair consisting of 'SampleRate' and a positive numeric scalar. The sample rate is the number of samples per second (Hz) of the time vector of the output timetable T.

SampleRate is a timetable related parameter. The parquetread function uses SampleRate along with other timetable parameters to define the time vector for the output T.

Data Types: double

Time step of time vector, specified as the comma-separated pair consisting of 'TimeStep' and a duration scalar.

  • If you specify the time step as a calendar duration (for example, calendar months), then the vector of row times must be a datetime vector.

  • If you specify the time step as a duration (for example, seconds), then the vector of row times can either be a datetime or duration vector.

TimeStep is a timetable related parameter. The parquetread function uses TimeStep along with other timetable parameters to define the time vector for the output T.

Flag to preserve variable names, specified as the comma-separated pair consisting of VariableNamingRule and either true, or false.

  • 'preserve' — Preserve variable names that are not valid MATLAB identifiers such as variable names that include spaces and non-ASCII characters.

  • 'modify' — Convert invalid variable names (as determined by the isvarname function) to valid MATLAB identifiers.

Starting in R2019b, variable names and row names can include any characters, including spaces and non-ASCII characters. Also, they can start with any characters, not just letters. Variable and row names do not have to be valid MATLAB identifiers (as determined by the isvarname function). To preserve these variable names and row names, set the value of VariableNamingRule to 'preserve'.

Data Types: char | string

Output Arguments

collapse all

Output data, returned as a table or timetable. The output of the parquetread function depends on the value of the OutputType name-value pair. For more information, see the name-value pair description for OutputType.

Limitations

In some cases, parquetwrite creates files that do not represent the original array T exactly. If you use parquetread or datastore to read the files, then the result might not have the same format or contents as the original table. For more information, see Apache Parquet Data Type Mappings.

See Also

| | | |

Introduced in R2019a

MATLAB Documentation

Support