Stream Methods

Introduction

Input and output streams are opened using a stream method. A stream method abstracts the stream operations required for reading or writing bytes from the source of those bytes. This makes it possible to read or write from data sources and storage mechanisms such as files, child processes, web servers, and in-memory lists of bytes. A stream method can also perform translations such as compression or encryption. For example, a stream method could allow a user to open a compressed file and read the uncompressed contents. Operating on streams allows operations to be performed on very large files that could otherwise not fit into physical memory.

The Wolfram System supplies stream methods for the most common needs, such as reading and writing from files, web servers, strings, and external commands. New stream methods can also be added to the Wolfram System.

Using Stream Methods

When you open a stream for reading or writing, you can specify the stream method with the (RefLink[Method, paclet:ref/Method]) option, or the Wolfram System can choose the stream method automatically from the stream name.

This opens a stream from a web server, selecting the stream method from the stream name:

Needs["CURLLink`"]
st = OpenRead["http://www.wolfram.com"]

Find[st,"Mathematica"]

Close[st]

This opens a stream from a web server, selecting the stream method with the (RefLink[Method, paclet:ref/Method]) option:

Needs["CURLLink`"]
st = OpenRead["http://www.wolfram.com", Method->"HTTP"]

Find[st,"<body",IgnoreCase->True]

Close[st]

This loads a Wolfram System package directly from a web server, selecting the stream method from the stream name:

Get["http://exampledata.wolfram.com/Collatz.m"]

Collatz[5]

This loads a Wolfram System package directly from a web server, using an explicitly opened input stream:

st=OpenRead["http://exampledata.wolfram.com/Collatz.m"]

ClearAll["Collatz`*"]

Get[st]

Collatz[4]

Close[st]

Input Stream Methods

You can add new input stream methods to the Wolfram System. This would be useful if you wanted to read from a new storage medium, or you wanted to transform the data as it is read, such as uncompressing a file.

Define a stream method that takes its input not from a file but from a list of bytes, passed as a method option "Bytes":

DefineInputStreamMethod["ByteList",{
"ConstructorFunction"->
Function[{name,caller,opts},If[MatchQ[opts,{___,"Bytes"->{_Integer ...},___}],
{True,{0,"Bytes"/.opts}},
{False,$Failed}
]],

"ReadFunction"->
Function[{state,n},
Module[{pos=state[[1]],bytes=state[[2]],bytesRead},
If[pos>=Length[bytes],
{{},state},
bytesRead=Part[bytes,pos+1;;Min[Length[bytes],pos+n]];
{bytesRead,{pos+Length[bytesRead],bytes}}
]
]],

"EndOfFileQFunction"->({#[[1]]>=Length[#[[2]]],#}&)
}]

This tests the new method:

bytes=ToCharacterCode["Title: The Quick Brown Fox\nAbstract: A fox jumps over dogs.\nThe quick brown fox jumped over the lazy dogs.\n"];
st=OpenRead["Quick Brown Fox",Method->{"ByteList","Bytes"->bytes}];
ReadList[st,String]//InputForm
Close[st];

This defines a stream method that performs a rotation cipher on the stream as it reads. A cryptographically strong decryption could be performed in the same fashion:

SetAttributes[rotateByte,Listable]
rotateByte[n_,x_]:=
With[{char=FromCharacterCode[x]},
Which[
UpperCaseQ[char],Mod[(x-65)+n,26]+65,
LowerCaseQ[char],Mod[(x-97)+n,26]+97,
True, x]
]

DefineInputStreamMethod["Rotate",
{
"ConstructorFunction"->
Function[{streamname,caller,opts},
With[{state=Unique["RotateInputStream"]},
state["stream"]=OpenRead[streamname,BinaryFormat->True];
state["positions"]="Positions"/.Join[opts,{"Positions"->13}];
state["eof"]=False;
{True,state}
]],

"CloseFunction"->Function[state,Close[state["stream"]]; ClearAll[state]],

"EndOfFileQFunction"->Function[state,{state["eof"],state}],

"ReadFunction"->
Function[{state,nBytes},
Module[{bytes=If[state["eof"],{},BinaryReadList[state["stream"],"Byte",nBytes]]},
If[!ListQ[bytes],bytes={}];
state["eof"]=Length[bytes]<nBytes;
{rotateByte[state["positions"],bytes],state}
]
]
}]

Test the method by first writing an enciphered test file, and then reading it using the new method:

Export["testrot13.txt","guvf vf n jrnxyl-uvqqra cnffjbeq","Text"]
FilePrint["testrot13.txt"]

st=OpenRead["testrot13.txt",Method->"Rotate"]

The file content differs from what (RefLink[Read, paclet:ref/Read]) returns, because the stream method translates it:

Read[st,String]

Close[st]
DeleteFile["testrot13.txt"]

Output Stream Methods

You can add new output stream methods to the Wolfram System. This would be useful if you wanted to write to a new storage medium, or you wanted to transform the data as it is written, such as compressing a file.

This defines a minimal stream method that prints the number of bytes written:

DefineOutputStreamMethod["Passthrough",
  {
    "ConstructorFunction" -> 
Function[{streamname,isAppend,caller,opts},
With[{state=Unique["PassthroughOutputStream"]},
state["stream"]=OpenWrite[streamname, BinaryFormat -> True];
state["pos"]=0;
{True,state}
]],

"CloseFunction" ->Function[state,Close[state["stream"]]; ClearAll[state]],

"StreamPositionFunction"->Function[state,{state["pos"],state}],

"WriteFunction" ->
Function[{state, bytes},
Module[{result = BinaryWrite[state["stream"], bytes],nBytes},
nBytes=If[result===state["stream"],Length[bytes],0];
Print["Wrote ", nBytes, " bytes"];
state["pos"]+=nBytes;
{nBytes,state}
]
]
    }
 ]

This tests the new method:

st=OpenWrite["test.bin",Method->"Passthrough",BinaryFormat->True]

BinaryWrite[st,Range[0,255]]

StreamPosition[st]

Close[st]

BinaryReadList["test.bin","Byte"]===Range[0,255]

DeleteFile["test.bin"]