This example shows you how to use the MDF Datastore feature of Vehicle Network Toolbox to quickly and efficiently process a data set spread across a collection of multiple MDF files. This workflow is also valuable when there is too much data to fit into available memory.
Find the collection of MDF files representing logged information from multiple test sequences. Note that MDF files to be used by MDF datastore as a set must have the same channel group and channel content structure.
dir('File*.mf4')
File01.mf4 File02.mf4 File03.mf4 File04.mf4 File05.mf4
You create an MDF datastore by selecting a folder location containing a collection of MDF files. In this case, target all files in the current working directory.
mds = mdfDatastore(pwd)
mds =
MDFDatastore with properties:
Files: {
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File01.mf4';
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File02.mf4';
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File03.mf4'
... and 2 more
}
ChannelGroups:
ChannelGroupNumber AcquisitionName Comment ... and 4 more columns
__________________ _______________ _______________
1 '' 'Integer Types'
2 '' 'Float Types'
Channels:
ChannelGroupNumber ChannelName DisplayName ... and 9 more columns
__________________ ____________________________________ ___________
1 'Sigend_Int16_LE_Offset_32' ''
1 'Unsigend_UInt32_LE_Master_Offset_0' ''
2 'Float_32_LE_Offset_64' ''
... and 1 more rows
SelectedChannelNames: {
'Sigend_Int16_LE_Offset_32';
'Unsigend_UInt32_LE_Master_Offset_0'
}
SelectedChannelGroupNumber: 1
ReadSize: 'file'
Multiple options allow control of what data is read from the MDF files and how the reads are performed. In this case, the first channel group is used by default. Note that only one channel group may be selected by the datastore at a time. You can also specify certain channels within the selected channel group to read. In this case, all channels are read by default.
mds.SelectedChannelGroupNumber mds.SelectedChannelNames
ans =
1
ans =
2×1 string array
"Sigend_Int16_LE_Offset_32"
"Unsigend_UInt32_LE_Master_Offset_0"
Using the preview function, you can obtain a quick view of the data available in the file set. Preview always returns up to eight data points from the first file in the datastore.
preview(mds)
ans =
8×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
5 sec 5 5
6 sec 6 6
7 sec 7 7
You can use the readall function to read the entire contents of data in a single call. This is an efficient way to read from many files when the data set fits into available memory. After running readall, the datastore resets to the beginning of the data set.
data = readall(mds); data(1:5,:)
ans =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
You can use the read function to obtain data from the file set. By default, reads from the MDF datastore will read an entire file's worth of data per call. The power of a datastore comes from reading through multiple files sequentially within the file set. As you read, the datastore automatically bridges from one file to the next until all data from all files is read.
for ii = 1:3 data = read(mds); whos('data') data(1:5,:) end
Name Size Bytes Class Attributes
data 10000x2 241767 timetable
ans =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
Name Size Bytes Class Attributes
data 10000x2 241767 timetable
ans =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
Name Size Bytes Class Attributes
data 10000x2 241767 timetable
ans =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
At any time, you can call the reset function to start over again at the beginning of the data set.
reset(mds)
You can use the ReadSize property to specify how much data to read on each call. ReadSize can be specified as a numeric value to read a fixed number of data points. ReadSize lets you control how much data is loaded into memory when you have a data set larger than available memory. We recommend using custom read sizes that are small enough to fit in memory, but still as large as possible to reduce processing overhead and improve performance.
mds.ReadSize = 5 for ii = 1:3 data = read(mds) end
mds =
MDFDatastore with properties:
Files: {
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File01.mf4';
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File02.mf4';
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File03.mf4'
... and 2 more
}
ChannelGroups:
ChannelGroupNumber AcquisitionName Comment ... and 4 more columns
__________________ _______________ _______________
1 '' 'Integer Types'
2 '' 'Float Types'
Channels:
ChannelGroupNumber ChannelName DisplayName ... and 9 more columns
__________________ ____________________________________ ___________
1 'Sigend_Int16_LE_Offset_32' ''
1 'Unsigend_UInt32_LE_Master_Offset_0' ''
2 'Float_32_LE_Offset_64' ''
... and 1 more rows
SelectedChannelNames: {
'Sigend_Int16_LE_Offset_32';
'Unsigend_UInt32_LE_Master_Offset_0'
}
SelectedChannelGroupNumber: 1
ReadSize: 5
data =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
data =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
5 sec 5 5
6 sec 6 6
7 sec 7 7
8 sec 8 8
9 sec 9 9
data =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
______ _________________________ __________________________________
10 sec 10 10
11 sec 11 11
12 sec 12 12
13 sec 13 13
14 sec 14 14
You can also specify ReadSize as a duration to read data points by elapsed time. Note that when the read type is changed, the datastore resets to the beginning of the data set.
mds.ReadSize = seconds(5) for ii = 1:3 data = read(mds) end
mds =
MDFDatastore with properties:
Files: {
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File01.mf4';
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File02.mf4';
' ...\home\jpyle\documents\MATLAB\examples\vnt-ex10761765\File03.mf4'
... and 2 more
}
ChannelGroups:
ChannelGroupNumber AcquisitionName Comment ... and 4 more columns
__________________ _______________ _______________
1 '' 'Integer Types'
2 '' 'Float Types'
Channels:
ChannelGroupNumber ChannelName DisplayName ... and 9 more columns
__________________ ____________________________________ ___________
1 'Sigend_Int16_LE_Offset_32' ''
1 'Unsigend_UInt32_LE_Master_Offset_0' ''
2 'Float_32_LE_Offset_64' ''
... and 1 more rows
SelectedChannelNames: {
'Sigend_Int16_LE_Offset_32';
'Unsigend_UInt32_LE_Master_Offset_0'
}
SelectedChannelGroupNumber: 1
ReadSize: 5 sec
data =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
0 sec 0 0
1 sec 1 1
2 sec 2 2
3 sec 3 3
4 sec 4 4
data =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
_____ _________________________ __________________________________
5 sec 5 5
6 sec 6 6
7 sec 7 7
8 sec 8 8
9 sec 9 9
data =
5×2 timetable
Time Sigend_Int16_LE_Offset_32 Unsigend_UInt32_LE_Master_Offset_0
______ _________________________ __________________________________
10 sec 10 10
11 sec 11 11
12 sec 12 12
13 sec 13 13
14 sec 14 14