Working with String Patterns

Introduction

The general symbolic string patterns in the Wolfram Language allow you to perform powerful string manipulation efficiently. What follows discusses the details of string patterns, including usage and implementation notes. The emphasis is on issues not mentioned elsewhere in the help system.

At the heart of the Wolfram Language is a powerful language for describing patterns in general expressions. This language is used in function definitions, substitutions, and searches, with constructs like x_, a|b, x.., and so on:

MatchQ[{a,b,c,d},{___,x_,x_,___}]

MatchQ[{a,b,c,c,d},{___,x_,x_,___}]

Cases[{a,3,4,b,c,8},_Integer]

A Wolfram Language string pattern uses the same constructs to describe patterns in a text string. You can think of a string as a sequence of characters and apply the principles of general Wolfram Language patterns. In addition there are several useful string-specific pattern constructs:

StringMatchQ["abcd",___~~x_~~x_~~___]

StringMatchQ["abccd",___~~x_~~x_~~___]

StringCases["a34bc8",DigitCharacter]

Regular expressions can be used as an alternative way to specify string patterns. These tend to be more compact, but less readable:

StringMatchQ["abcd",RegularExpression[".*(.)\\1.*"]]

StringMatchQ["abccd",RegularExpression[".*(.)\\1.*"]]

StringCases["a34bc8",RegularExpression["\\d"]]

Here is a list of several functions that recognize string patterns.

Functions that support string patterns.

General String Patterns

A general string pattern is formed from pattern objects similar to the general pattern objects in the Wolfram Language. To join several string pattern objects, use the (RefLink[StringExpression, paclet:ref/StringExpression]) operator ~~:

FullForm["a"~~_]

(RefLink[StringExpression, paclet:ref/StringExpression]) is closely related to (RefLink[StringJoin, paclet:ref/StringJoin]), except nonstrings are allowed and lists are not flattened. For pure strings, they are equivalent:

"aa"~~"bbb"~~"c"

The list of objects that can appear in a string pattern closely matches the list for ordinary Wolfram Language patterns. In terms of string patterns, a string is considered a sequence of characters, that is, "abc" can be thought of as something like (RefLink[String, paclet:ref/String])[a,b,c], to which the ordinary pattern constructs apply.

The following objects can appear in a symbolic string pattern:

The following represent classes of characters:

The following represent positions in strings:

The following determine which match will be used if there are several possibilities:

Some nontrivial issues regarding these objects follow.

The _, __, and ___ wildcards match any characters including newlines. To match any character except newline (analogous to the "." in regular expressions), use (RefLink[Except, paclet:ref/Except])["\n"], (RefLink[Except, paclet:ref/Except])["\n"].., and (RefLink[Except, paclet:ref/Except])["\n"]...:

StringCases["line1\nline2\n",__]

StringCases["line1\nline2\n",Except["\n"]..]

StringCases["line1\nline2\n",RegularExpression[".+"]]

A list of patterns, such as {"a","b","c"}, is equivalent to a list of alternatives, such as "a"|"b"|"c". This is convenient in that functions like (RefLink[Characters, paclet:ref/Characters]) and (RefLink[CharacterRange, paclet:ref/CharacterRange]) can be used to specify classes of characters:

StringReplace["the cat in the hat",x:Characters["aeiou"]:>x<>x]

When (RefLink[Condition, paclet:ref/Condition]) (/;) is used, the patterns involved are treated as strings as far as the rest of the Wolfram Language is concerned, so you need to use (RefLink[ToExpression, paclet:ref/ToExpression]) in some cases:

StringCases["a13 a18 a41 a42","a"~~x:DigitCharacter..~~WordBoundary/;PrimeQ[ToExpression[x]]:>x]

Similar to ordinary Wolfram Language patterns, the function in (RefLink[PatternTest, paclet:ref/PatternTest]) (?) is applied to each individual character:

StringCases["125378132",__?(ToExpression[#]<5&)]

The (RefLink[Whitespace, paclet:ref/Whitespace]) construct is equivalent to (RefLink[WhitespaceCharacter, paclet:ref/WhitespaceCharacter])..:

StringReplace["13   \t 17 \n22   19",Whitespace->","]

You can insert a (RefLink[RegularExpression, paclet:ref/RegularExpression]) object into a general string pattern:

StringCases["a13b12c17a32","a"~~x:RegularExpression["\\d+"]:>x]

This inserts a lookbehind constraint (see "Regular Expressions") to ensure that you only pick words preceded by "the ":

StringCases["the cat in the hat",RegularExpression["(?<=the )"]~~WordCharacter..]

(RefLink[StringExpression, paclet:ref/StringExpression]) objects can be nested:

StringCases["ba3a1a78a2b7ba9","b"~~("a"~~DigitCharacter)..]

The (RefLink[Except, paclet:ref/Except]) construct for string patterns takes a single argument that should represent a single character or a class of single characters.

This deletes all nonvowel characters from the string:

StringReplace["the cat in the hat",Except[Characters["aeiou"]]->""]

When trying to match patterns of variable length (such as __ and patt..), the longest possible match is tried first by default. To force the matcher to try the shortest match first, you can wrap the relevant part of the pattern in (RefLink[Shortest, paclet:ref/Shortest])[ ]:

StringCases["(ab) (cde)","("~~__~~")"]

StringCases["(ab) (cde)",Shortest["("~~__~~")"]]

If for some reason you need a longest match within the short match, you can use (RefLink[Longest, paclet:ref/Longest]):

StringCases["(ab132cd)137(ef576gh)",Shortest["("~~___~~x:DigitCharacter..~~___~~")"]:>x]

StringCases["(ab132cd)(ef576gh)",Shortest["("~~___~~Longest[x:DigitCharacter..]~~___~~")"]:>x]

You could alternatively rewrite this pattern without use of (RefLink[Longest, paclet:ref/Longest]):

StringCases["(ab132cd)(ef576gh)","("~~Shortest[___]~~x:DigitCharacter..~~Shortest[___]~~")":>x]

Regular Expressions

The regular expression syntax follows the underlying Perl Compatible Regular Expressions (PCRE) library, which is close to the syntax of Perl. (See [1] for further information and documentation.) A regular expression in the Wolfram Language is denoted by the head (RefLink[RegularExpression, paclet:ref/RegularExpression]).

The following basic elements can be used in regular expression strings:

The following represent classes of characters:

The following named classes can be used: alnum, alpha, ascii, blank, cntrl, digit, graph, lower, print, punct, space, upper, word, and xdigit.

The following represent positions in strings:

The following set options for all regular expression elements that follow them:

The following are lookahead/lookbehind constructs:

Discussion of a few issues regarding regular expressions follows.

This looks for runs of word characters of length between 2 and 4:

StringCases["a bb ccc dddd eeeee",RegularExpression["\\b\\w{2,4}\\b"]]

With the possessive "+" quantifier, as many characters as possible are grabbed by the matcher, and no characters are given up, even if the rest of the patterns require it:

StringCases["a2 b6",RegularExpression["\\w+\\d"]]

StringCases["a2 b6",RegularExpression["\\w++\\d"]]

StringCases["a2 b6",RegularExpression["\\D++\\d"]]

[[:xdigit:]] corresponds to characters in a hexadecimal number:

StringCases["ff, 13, 1a3, xyz, 3b",RegularExpression["[[:xdigit:]]+"]]

The complete list of characters that need to be escaped in a regular expression consists of ., \, ?, (, ), {, }, [, ], ^, $, *, +, and |. For instance, to write a literal period, use "\\." and to write a literal backslash, use "\\\\".

Inside a character class "[…]", the complete list of escaped characters is ^, -, \, [, and ].

By default, ^ and $ match the beginning and end of the string, respectively. In multiline mode, these match the beginning/end of lines instead:

StringCases["line1\nline2",RegularExpression["^.*"]]

StringCases["line1\nline2",RegularExpression["(?m)^.*"]]

In multiline mode, \\A and \\Z can be used to denote the beginning and end of the string:

StringCases["line1\nline2",RegularExpression["(?m)\\A.*"]]

The (?x) modifier allows you to add whitespace and comments to a regular expression for readability:

StringCases["12.45  bc58.11",
RegularExpression["\<(?x)
\\d+  \\.  #remember to escape the period
\\d+\>"]]

Named subpatterns are achieved by surrounding them with parentheses (subpatt); they then become numbered subpatterns. The number of a given subpattern counts the opening parenthesis, starting from the start of the pattern. You can refer to these subpatterns using \\n for the n(^th) pattern later in the pattern, or by "$n" in the right-hand side of a rule. "$0" refers to all of the matched pattern:

StringCases["a1b6a3b3a3c3a8b8",RegularExpression["(a(\\d))b\\2"]]

StringCases["a1b6a3b3a3c3a8b8",RegularExpression["(a(\\d))b\\2"]->{"$0","$1","Number:$2"}]

If you need a literal $ in this context (when the head of the left-hand side is (RefLink[RegularExpression, paclet:ref/RegularExpression])), you can escape it by using backslashes (for example, "\\$2"):

StringCases["a1b6a3b3a3c3a8b8",RegularExpression["(a(\\d))b\\2"]->{"$0","$1","Number:$2","Literal:\\$2"}]

If you happen to need a single literal backslash followed by a literal $ under these circumstances, you need to be a bit tricky and split into two strings temporarily:

StringCases["a1b6a3b3a3c3a8b8",RegularExpression["(a(\\d))b\\2"]:>{"$0","$1","Number:$2","Literal:\\"<>"\\$2"}]

If you need to group a part of the pattern, but you do not want to count the group as a numbered subpattern, you can use the (?:patt) construct:

StringCases["a11b16c22b77",RegularExpression["(?:a|b)(\\d)\\1"]]

Lookahead and lookbehind patterns are used to ensure a pattern is matched without actually including that text as part of the match.

This picks out words following the string "the ":

StringCases["the cat in the hat",RegularExpression["(?<=the )\\w+"]]

This tries to pick out all even numbers in the string, but it will find matches that include partial numbers:

StringCases["a23b42c63d80, 123",x:RegularExpression["\\d+"]/;Mod[ToExpression[x],2]==0]

Using lookbehind/lookahead, you can ensure that the characters before/after the match are not digits (note that the lookbehind test is superfluous in this particular case):

StringCases["a23b42c63d80, 123",x:RegularExpression["(?<!\\d)\\d+(?!\\d)"]/;Mod[ToExpression[x],2]==0]

RegularExpression versus StringExpression

There is a close correspondence between the various pattern objects that can be used in general symbolic string patterns and in regular expressions. Here is a list of examples of patterns written as regular expressions and as symbolic string patterns.

Pattern objects that can be used in general string patterns, but not in regular expressions, include conditions (/;) and pattern tests (?) that can access general Wolfram Language code during the match.

Some special constructs in regular expressions are not directly available in general string patterns. These include lookahead/lookbehinds and repeats of a given length. They can be embedded into a larger general string pattern by inserting a (RefLink[RegularExpression, paclet:ref/RegularExpression]) object.

String Manipulation Functions

The following discusses some particulars and subtleties in the various string manipulation functions (see the reference pages for more information on these functions).

StringMatchQ

(RefLink[StringMatchQ, paclet:ref/StringMatchQ]) is used to check whether a whole string matches a certain pattern:

StringMatchQ["test","t"~~__~~"t"]

StringMatchQ["tester","t"~~__~~"t"]

(RefLink[StringMatchQ, paclet:ref/StringMatchQ]) is special in that it also allows the metacharacters * and @ to be entered as wildcards (for backward compatibility reasons). * is equivalent to (RefLink[Shortest, paclet:ref/Shortest])[___] ((RefLink[RegularExpression, paclet:ref/RegularExpression])["(?s).*?"]) and @ is equivalent to (RefLink[Except, paclet:ref/Except])[(RefLink[CharacterRange, paclet:ref/CharacterRange])["A","Z"]] ((RefLink[RegularExpression, paclet:ref/RegularExpression])["[^A-Z]"]).

The following three patterns are therefore equivalent:

StringMatchQ["test",_~~"e*"]

StringMatchQ["test",_~~"e"~~Shortest[___]]

StringMatchQ["test",RegularExpression["(?s).e.*?"]]

Note that technically the appearance of (RefLink[Shortest, paclet:ref/Shortest]) does not make a difference here, since we are only looking for a possible match.

If you need to access parts of the string matched by subpatterns in the pattern, use (RefLink[StringCases, paclet:ref/StringCases]) instead.

(RefLink[StringMatchQ, paclet:ref/StringMatchQ]) has a (RefLink[SpellingCorrection, paclet:ref/SpellingCorrection]) option for finding matches allowing for a small number of discrepancies. This only works for patterns consisting of a single literal string:

StringMatchQ["alpha","alpa",SpellingCorrection->True]

StringFreeQ

(RefLink[StringFreeQ, paclet:ref/StringFreeQ]) is used to check whether a string contains a substring matching the pattern. You cannot extract the matching substring; to do this you would use (RefLink[StringCases, paclet:ref/StringCases]):

StringFreeQ["abcde","b"~~__~~"d"]

StringFreeQ["abcde",RegularExpression["b.*d"]]

StringCases

(RefLink[StringCases, paclet:ref/StringCases]) is a general purpose function for finding occurrences of patterns in a string, picking out subpatterns, and processing the results.

Find substrings matching a pattern:

StringCases["a1b2a26d15a42","a"~~_]

Pick apart the matching substring:

StringCases["a1b2a26d15a42","a"~~x:DigitCharacter..->x]

StringCases["a1b2a26d15a42",RegularExpression["a(\\d+)"]->"$1"]

Restrict the number of matches:

StringCases["a b c d e",LetterCharacter,3]

You can use a list of rules:

StringCases["a13bF5b1Aa33",{"a"~~x:DigitCharacter..->f1[x],"b"~~x:(DigitCharacter|CharacterRange["A","F"])..->hex[x]}]

You can also give a list of strings as the first argument for efficient processing of many strings (see "Tips and Tricks for Efficient Matching" for a discussion):

StringCases[{"cat","in","the","hat"},__~~"t"~~EndOfString]

Flatten[%]

The Overlaps Option

The (RefLink[Overlaps, paclet:ref/Overlaps]) option for (RefLink[StringCases, paclet:ref/StringCases]), (RefLink[StringPosition, paclet:ref/StringPosition]), and (RefLink[StringCount, paclet:ref/StringCount]) deals with how the matcher proceeds after finding a match. It has three possible settings: (RefLink[False, paclet:ref/False]), (RefLink[True, paclet:ref/True]), or (RefLink[All, paclet:ref/All]). The default is (RefLink[False, paclet:ref/False]) for (RefLink[StringCases, paclet:ref/StringCases]) and (RefLink[StringCount, paclet:ref/StringCount]), while it is (RefLink[True, paclet:ref/True]) for (RefLink[StringPosition, paclet:ref/StringPosition]).

With (RefLink[Overlaps, paclet:ref/Overlaps])->(RefLink[False, paclet:ref/False]), the matcher continues the match testing at the character following the last matched substring:

StringCases["(a(b)c(d)",Shortest["("~~__~~")"]]

With (RefLink[Overlaps, paclet:ref/Overlaps])->(RefLink[True, paclet:ref/True]), the matcher continues at the character following the first character of the last matched substring (when a single pattern is involved):

StringCases["(a(b)c(d)",Shortest["("~~__~~")"],Overlaps->True]

With (RefLink[Overlaps, paclet:ref/Overlaps])->(RefLink[All, paclet:ref/All]), the matcher keeps starting at the same position until no more new matches are found:

StringCases["(a(b)c(d)",Shortest["("~~__~~")"],Overlaps->All]

If multiple patterns are given in a list, (RefLink[Overlaps, paclet:ref/Overlaps])->(RefLink[True, paclet:ref/True]) will cause the matcher to start at the same position once for each of the patterns before proceeding to the next character:

StringCases["(a(b)c(d)",{Shortest["("~~__~~")"],Shortest["("~~__~~"("]},Overlaps->True]

StringCases["(a(b)c(d)",{Shortest["("~~__~~")"],Shortest["("~~__~~"("]},Overlaps->False]

Note that with (RefLink[Overlaps, paclet:ref/Overlaps])->(RefLink[True, paclet:ref/True]), there can thus be a difference between specifying a list of patterns and using the alternatives operator (|):

StringCases["ab",{_,__},Overlaps->True]

StringCases["ab",_|__,Overlaps->True]

StringPosition

(RefLink[StringPosition, paclet:ref/StringPosition]) works much like (RefLink[StringCases, paclet:ref/StringCases]), except the positions of the matching substrings are returned:

StringPosition["a1b2a26d15a42","a"~~_]

StringTake["a1b2a26d15a42",#]&/@%

The (RefLink[Overlaps, paclet:ref/Overlaps]) option is (RefLink[True, paclet:ref/True]) by default (see the previous section for more details on this option):

StringPosition["(a(b)c(d)",Shortest["("~~__~~")"]]

Note that even empty strings can be matches:

StringPosition["abc",___]

StringCount

(RefLink[StringCount, paclet:ref/StringCount]) returns the number of matching substrings (which are found by (RefLink[StringPosition, paclet:ref/StringPosition]) or (RefLink[StringCases, paclet:ref/StringCases])). It is useful for cases with many matches where memory for storing all the substrings might be an issue:

StringCount["abaababba","a"~~___~~"b",Overlaps->All]

StringCases["abaababba","a"~~___~~"b",Overlaps->All]//Length

Note that (RefLink[Overlaps, paclet:ref/Overlaps])->(RefLink[False, paclet:ref/False]) is the default for (RefLink[StringCount, paclet:ref/StringCount]).

StringReplace

(RefLink[StringReplace, paclet:ref/StringReplace]) is used for substituting substrings matching the given patterns:

StringReplace["abcde",{"a"->"A","cd"->"XX"}]

Named patterns can be used as strings on the right-hand side of the replacement rules. Note the use of (RefLink[RuleDelayed, paclet:ref/RuleDelayed]) (:>) to avoid premature evaluation:

StringReplace["this is a test",x:WordCharacter..:>StringReverse[x]]

When using regular expressions, it is convenient to remember that "$0" on the right-hand side refers to the whole matched substring:

StringReplace["this is a test",RegularExpression["\\w+"]:>StringReverse["$0"]]

You can limit the number of replacements made by specifying a third argument:

StringReplace["this is a test",x:WordCharacter..:>StringReverse[x],1]

Note that the replacement does not have to be a string. If the result is not a string, a (RefLink[StringExpression, paclet:ref/StringExpression]) is returned:

StringReplace["some <b>bold</b> and <i>italics</i>.",Shortest["<"~~x___~~">"]:>Tag[x]]

FullForm[%]

There is limited support for using the old MetaCharacters option in conjunction with general string patterns, but this option is deprecated and its use should be avoided.

StringReplaceList

(RefLink[StringReplaceList, paclet:ref/StringReplaceList]) returns a list of strings where a single string replacement has been made in all possible ways:

StringReplaceList["abaac","a"~~x_:>ToUpperCase[x]]

If a list of strings is given as input, the output is a nested list of results:

StringReplaceList[{"abaac","baaba"},"a"~~x_:>ToUpperCase[x]]

StringSplit

(RefLink[StringSplit, paclet:ref/StringSplit]) is useful for splitting a string into many strings at delimiters matching a pattern. By default, the splits happen at runs of whitespace:

StringSplit["this is a test"]

For instance, to split a normal sentence into words, you need to also include punctuation in the delimiter:

StringSplit["A sentence: with commas, semicolons; etc...!?",Characters[":,;.!? "]..]

By default, empty strings at the beginning and the end of the result are removed:

StringSplit[":a:b:c:",":"]

These can be included by specifying (RefLink[All, paclet:ref/All]) as a third argument:

StringSplit[":a:b:c:",":",All]

The third argument can also be a number giving the maximum number of strings to split into:

StringSplit["this is a test",Whitespace,2]

This splits a string into individual lines:

StringSplit["line1\nthis is line 2\nline3","\n"]

You can also split at patterns that match positions, such as (RefLink[StartOfLine, paclet:ref/StartOfLine]). This keeps the newline characters in the result:

StringSplit["line1\nthis is line 2\nline3",StartOfLine]

You can keep the delimiters, or parts of the delimiters, in the output by using a rule as the second argument:

StringSplit["this is a test"," "->" "]

StringSplit["this is a test"," "->":"]

StringSplit["the <tag1>first</tag1> and the <tag2>second</tag2>",Shortest["<"~~__~~">"]]

StringSplit["the <tag1>first</tag1> and the <tag2>second</tag2>",Shortest["<"~~x__~~">"]:>Tag[x]]

You can give a list of patterns and rules as well; the delimiters matching the patterns will be left out of the result:

StringSplit["the <tag1>first</tag1> and the <tag2>second</tag2>",{Whitespace,Shortest["<"~~x__~~">"]:>Tag[x]}]//InputForm

For Perl Users

Overview

With the addition of general string patterns, the Wolfram Language can be a powerful alternative to languages like Perl and Python for many general, everyday programming tasks. For people familiar with Perl syntax, and the way Perl does string manipulation, the following rough guide shows how to get similar functionality in the Wolfram Language.

Here is an overview of the Wolfram Language functions involved in constructing Perl-like functions.

Following are some common Perl constructs in more detail.

m/.../

The match operator m/regex/ tests whether a string contains a substring matching the regex. For simple matches of this sort in the Wolfram Language, use (RefLink[StringFreeQ, paclet:ref/StringFreeQ]).

Here is a Perl snippet for testing whether a string contains a "b" somewhere after an "a".

$string = "sdakdb";
if ($string =~ m/a.*b/){
  print "Match!";
}

Here is a Wolfram Language version of the same test:

string="sdakdb";
If[!StringFreeQ[string,RegularExpression["a.*b"]],Print["Match!"]]

If parts of the matched string need to be accessed later, using $1, $2, … in Perl, the best Wolfram Language function to use is normally (RefLink[StringCases, paclet:ref/StringCases]).

Here is Perl code for extracting an error message.

$res = "ERROR = paper jam";
if ($res =~ m/ERROR = (.*)/){
  print "Hey, you should check the $1!";
}

Here is a Wolfram Language version:

res="ERROR = paper jam";
With[{test=StringCases[res,RegularExpression["ERROR = (.*)"]->"$1"]},If[test=!={},Print["Hey, you should check the ",test[[1]],"!"]]]

Here is Perl code for extracting several subpatterns at once.

$date = "88/6/13";
($year, $month, $day) = $date =~ m/^(\d+)/(\d+)/(\d+)$/;

In the Wolfram Language, this is done with (RefLink[StringCases, paclet:ref/StringCases]):

date="88/6/13";
{year,month,day}=StringCases[date,RegularExpression["^(\\d+)/(\\d+)/(\\d+)$"]->{"$1","$2","$3"}][[1]]

This is similar to assigning all the matches to an array using the /g modifier.

$text = "128.32.13.117";
@nums = $text =~ m/\d+/g;

The same thing is easily done with (RefLink[StringCases, paclet:ref/StringCases]) in the Wolfram Language:

text="128.32.13.117";
nums=StringCases[text,RegularExpression["\\d+"]]

s/.../.../

The obvious Wolfram Language version of the Perl s/.../.../ substitution operator is (RefLink[StringReplace, paclet:ref/StringReplace]).

$text = "abcagh";
$text =~ s/a./XX/;

The default Perl behavior is to do a single replacement:

text="abcagh";
StringReplace[text,RegularExpression["a."]->"XX",1]

The /g modifier in Perl does global replacement of all matches:

$text =~ s/a./XX/g

StringReplace[text,RegularExpression["a."]->"XX"]

Using the evaluation /e modifier, Perl can use subpatterns as part of the replacement. This is easily done in the Wolfram Language:

$text = "13 27 3";
$text =~ s/(\d+)/$1$1/eg

text="13 27 3";
StringReplace[text,RegularExpression["(\\d+)"]:>"$1$1"]

split(...)

The Perl split command is similar to (RefLink[StringSplit, paclet:ref/StringSplit]) in the Wolfram Language:

$text = "ab:cd:efg";
split(/:/, $text)

text="ab:cd:efg";
StringSplit[text,":"]

You can specify the number of blocks to split into in both Perl and the Wolfram Language:

split(/:/, $text,2)

StringSplit[text,":",2]

A split with capturing parentheses in the pattern, for which the captured substrings are included in the result, can be done in the Wolfram Language using rules in the second argument of (RefLink[StringSplit, paclet:ref/StringSplit]). Compared to Perl, in the Wolfram Language it is easy to then apply a function to these substrings:

$text = "test with <tag1>tags</tag1> and <b>more</b>";
split(/<([^>]*)>/, $text)

text="test with <tag1>tags</tag1> and <b>more</b>";
StringSplit[text,RegularExpression["<([^>]*)>"]->"$1"]//InputForm

text="test with <tag1>tags</tag1> and <b>more</b>";
StringSplit[text,RegularExpression["<([^>]*)>"]:>Tag["$1"]]//InputForm

tr/.../.../

The Perl tr command can be simulated using Wolfram Language (RefLink[StringReplace, paclet:ref/StringReplace]) together with the appropriate list of rules.

Here is the simplest form where the characters "a", "b", and "c" are replaced by "X", "Y", and "Z", respectively.

$text = "abcdef";
$text =~ tr/abc/XYZ/

This generates the appropriate rules in the Wolfram Language using (RefLink[Thread, paclet:ref/Thread]):

text="abcdef";
StringReplace[text,Thread[Rule[Characters["abc"],Characters["XYZ"]]]]

Here is an example where the replacement list is shorter than the character list, so "d", "e", and "f" are all replaced by "Z":

$text = "abcdefghi";
$text =~ tr/abcdef/WXYZ/

text="abcdefghi";
StringReplace[text,Append[Thread[Rule[Characters["abc"],Characters["WXY"]]],Characters["def"]->"Z"]]

Character ranges in Perl are emulated using (RefLink[CharacterRange, paclet:ref/CharacterRange]) in the Wolfram Language:

$text = "this and that";
$text =~ tr/a-z/x/

text="this and that";
StringReplace[text,CharacterRange["a","z"]->"x"]

With the /d modifier, the surplus characters are instead deleted:

$text = "abcdefghi";
$text =~ tr/abcdef/WXYZ/d

text="abcdefghi";
StringReplace[text,Append[Thread[Rule[Characters["abcd"],Characters["WXYZ"]]],Characters["ef"]->""]]

With the /c modifier, the complement of the character list is used:

$text =~ tr/aeh/ /c

StringReplace[text,Except[Characters["aeh"]]->" "]

StringReplace[text,RegularExpression["[^aeh]"]->" "]

The /s modifier squeezes down to one any run of characters translating into the same character.

$text = "abbcccddddeeeeeeffeeded";
$text =~ tr/abcde/ABCD/s

You get the same effect in the Wolfram Language using (RefLink[Repeated, paclet:ref/Repeated]) (..):

text="abbcccddddeeeeeeffeeded";
StringReplace[text,Append[Thread[Rule[Repeated/@Characters["abc"],Characters["ABC"]]],Characters["de"]..->"D"]]

Some Examples

Some brief examples of practical uses of string patterns are presented in this section.

Highlight Patterns

This defines a 1000-base random DNA string:

SeedRandom[12345];dna=StringJoin[Table[{"a","c","g","t"}[[RandomInteger[{1,4}]]],{1000}]]

This highlights parts of the DNA that match a certain pattern:

StringReplace[dna,x:("ag"~~_~~_~~"t"~~_~~"ca"):>"\!\(\*StyleBox[\""<>x<>"\",FontColor->RGBColor[1,0,0],FontSize->18,FontWeight->\"Bold\"]\)"]

Here is the same result using a regular expression:

StringReplace[dna,RegularExpression["ag..t.ca"]:>"\!\(\*StyleBox[\"$0\",FontColor->RGBColor[1,0,0],FontSize->18,FontWeight->\"Bold\"]\)"]

HTML Parsing

String patterns are useful for taking raw HTML and extracting information from it.

Here is the source from www.google.com:

text="\<<html><head><meta http-equiv='content-type' content='text/html;charset=UTF-8'><title>Google</title><style><!--body,td,a,p,.h{font-family:arial,sans-serif;}
.h{font-size:20px;}
.q{color:#0000cc;}
//-->
</style>
<script>
<!--function sf(){document.f.q.focus();}
//-->
</script>
</head><body bgcolor=#ffffff text=#000000 link=#0000cc vlink=#551a8b alink=#ff0000 onLoad=sf()><center><table border=0 cellspacing=0 cellpadding=0><tr><td><img src='/images/logo.gif' width=276 height=110 alt='Google'></td></tr></table><br>
<form action='/search' name=f><script><!--function qs(el) {if (window.RegExp&&window.encodeURIComponent) {var qe=encodeURIComponent(document.f.q.value);if (el.href.indexOf('q=')≠-1) {el.href=el.href.replace(new RegExp('q=[^&$]*'),'q='+qe);} else {el.href+='&q='+qe;}}return 1;}
//-->
</script><table border=0 cellspacing=0 cellpadding=4><tr><td nowrap class=q><font size=-1><b><font color=#000000>Web</font></b>&nbsp;&nbsp;&nbsp;&nbsp;<a id=1a class=q href='/imghp?hl=en&tab=wi' onClick='return qs(this);'>Images</a>&nbsp;&nbsp;&nbsp;&nbsp;<a id=2a class=q href='/grphp?hl=en&tab=wg' onClick='return qs(this);'>Groups</a>&nbsp;&nbsp;&nbsp;&nbsp;<a id=4a class=q href='/nwshp?hl=en&tab=wn' onClick='return qs(this);'>News</a>&nbsp;&nbsp;&nbsp;&nbsp;<a id=5a class=q href='/froogle?hl=en&tab=wf' onClick='return qs(this);'>Froogle</a>&nbsp;&nbsp;&nbsp;&nbsp;<b><a href='/options/index.html' class=q>more&nbsp;&raquo;</a></b></font></td></tr></table>  <table cellspacing=0 cellpadding=0><tr><td width=25%>&nbsp;</td><td align=center><input type=hidden name=hl value=en><span id=hf></span><input type=hidden name=ie value='UTF-8'><input maxLength=256 size=55 name=q value=''><br><input type=submit value='Google Search' name=btnG><input type=submit value='I'm Feeling Lucky' name=btnI></td><td valign=top nowrap width=25%><font size=-2>&nbsp;&nbsp;<a href=/advanced_search?hl=en>Advanced&nbsp;Search</a><br>&nbsp;&nbsp;<a href=/preferences?hl=en>Preferences</a><br>&nbsp;&nbsp;<a href=/language_tools?hl=en>Language Tools</a></font></td></tr></table></form><br><br><font size=-1><a href='/ads/'>Advertising&nbsp;Programs</a>- <a href='/services/'>Business&nbsp;Solutions</a>- <a href=/about.html>About Google</a><span id=hp style='behavior:url(#default#homepage)'></span>
<script>
//<!--if (!hp.isHomePage('http://www.google.com/')) {document.write('<p><a href=\'/mgyhp.html\' onClick=\'style.behavior='url(#default#homepage)';setHomePage('http://www.google.com/');\'>Make Google Your Homepage!</a>');}
//-->
</script></font><p><font size=-2>&copy;2004 Google-Searching 4,285,199,774 web pages</font></p></center></body></html>\>";

StringLength[text]

This extracts all the direct hyperlinks in the source:

StringCases[text,Shortest["<a"~~__~~"href="~~ref__~~(WhitespaceCharacter|">")~~___~~">"]:>ref]

This deletes everything inside tags <…>:

StringReplace[text,Shortest["<"~~___~~">"]->""]

Find Money

Here is some text to scan for strings that look like dollar amounts:

text="This $100 sentence can be bought for $85.00, at 15% discount"

This is one way to do the search using symbolic string patterns:

StringCases[text,"$"~~DigitCharacter..~~(("."~~DigitCharacter..)|"")]

Here is the same search using regular expressions (note that you must remember to escape the dollar sign):

StringCases[text,RegularExpression["\\$\\d+(\\.\\d+)?"]]

There is also a built-in pattern object, (RefLink[NumberString, paclet:ref/NumberString]), for this particular situation:

StringCases[text,"$"~~NumberString]

Find Text in Files

Here is a very simple grep-like function for finding lines in a text file containing text matching a given pattern:

Grep[file_,patt_]:=With[{data=Import[file,"Lines"]},Pick[Transpose[{Range[Length[data]],data}],StringFreeQ[data, patt],False]]

This creates a sample text file:

Export["test.txt",{"this is a line","a line with 2 numbers 5","third line and more","line 4"},"Lines"]

This returns the line numbers and lines in "text.txt" containing any digit characters:

Grep["test.txt",DigitCharacter]//TableForm

This finds lines containing "a" as a standalone word:

Grep["test.txt",RegularExpression["\\ba\\b"]]//TableForm

Tips and Tricks for Efficient Matching

This section addresses some issues involving efficiency in string pattern matching.

StringExpression versus RegularExpression

Since a string pattern written in Wolfram Language syntax is immediately translated to a regular expression and then compiled and cached, there is very little overhead in using the Wolfram Language syntax as opposed to the regular expression syntax directly. An exception to this happens when many different patterns are used a few times; in that case the overhead might be noticeable.

Conditions and PatternTests

If a pattern contains (RefLink[Condition, paclet:ref/Condition]) (/;) or (RefLink[PatternTest, paclet:ref/PatternTest]) (?) statements, the general Wolfram Language evaluator must be invoked during the match, thus slowing it down. If a pattern can be written without such constructs, it will typically be faster:

SeedRandom[1234];test=StringJoin[Table[FromCharacterCode[RandomInteger[{48,80}]],{200}]];

StringCases[test,DigitCharacter..]//Length//Timing

StringCases[test,__?DigitQ]//Length//Timing

Avoid Nested Quantifiers

Because of the nondeterministic finite automaton (NFA) algorithm used in the match, patterns involving nested quantifiers (such as __ and patt.. or the regular expression equivalents) can become arbitrarily slow. Such patterns can usually be "unrolled" into more efficient versions (see Friedl [2] for additional information).

Avoid Many Calls to a Function

If you are searching through a long list of strings for certain matches, it is more efficient to feed the whole list to a string function at once, rather than using something like (RefLink[Select, paclet:ref/Select]) and (RefLink[StringMatchQ, paclet:ref/StringMatchQ]) (see the earlier dictionary example for an illustration). Here is another example that generates a list of 2000 strings with 10 characters each and searches for the strings that start with an "a" and contain "ggg" as a substring:

SeedRandom[1234];test=Table[StringJoin[{"a","c","g","t"}[[#]]&/@Table[RandomInteger[{1,4}],{10}]],{2000}];

Take[test,3]

Here is the slower version, using (RefLink[Select, paclet:ref/Select]) and (RefLink[StringMatchQ, paclet:ref/StringMatchQ]):

Select[test,StringMatchQ[#,"a"~~___~~"ggg"~~___]&]//Timing

If you instead feed the whole list to (RefLink[StringMatchQ, paclet:ref/StringMatchQ]) at once, it will be much faster. Then (RefLink[Pick, paclet:ref/Pick]) can be used to extract the wanted elements:

Pick[test,StringMatchQ[test,"a"~~___~~"ggg"~~___]]//Timing

Alternatively, you could use (RefLink[StringCases, paclet:ref/StringCases]), which is also fast. Note that you need to anchor the pattern using (RefLink[StartOfString, paclet:ref/StartOfString]) to ensure that the "a" is at the start (the (RefLink[EndOfString, paclet:ref/EndOfString]) is superfluous in this particular case):

Flatten[StringCases[test,StartOfString~~"a"~~___~~"ggg"~~___~~EndOfString]]//Timing

Rewrite General Expression Searches as String Searches

Because the string-matching algorithm is different than the algorithm the Wolfram Language uses for general expression matching (string matching can assume a finite alphabet and a flat structure, for instance), there are cases where it is advantageous to translate a normal expression-matching problem to a string-matching problem. A typical case is matching a long list of symbols against a pattern involving several occurrences of __ and ___.

As an example, assume you want to find primes (after prime number 1000000, say) that have at least four identical digits. Using ordinary pattern matching, it could be accomplished like this:

Select[Array[Prime,1000,1000000],MatchQ[IntegerDigits[#],{___,x_,___,x_,___,x_,___,x_,___}]&]//Timing

By converting the list of integers to a string, you can use string matching instead:

Select[Array[Prime,1000,1000000],StringMatchQ[FromCharacterCode[48+IntegerDigits[#]],StringExpression[___,x_,___,x_,___,x_,___,x_,___]]&]//Timing

By using the previous tips of using (RefLink[Pick, paclet:ref/Pick]) or (RefLink[StringCases, paclet:ref/StringCases]), you can speed it up even more:

With[{list=Array[Prime,1000,1000000]},Pick[list,StringMatchQ[FromCharacterCode[48+IntegerDigits[#]]&/@list,StringExpression[___,x_,___,x_,___,x_,___,x_,___]]]]//Timing

Flatten[StringCases[FromCharacterCode[48+IntegerDigits[#]]&/@Array[Prime,1000,1000000],StringExpression[StartOfString,___,x_,___,x_,___,x_,___,x_,___,EndOfString]]]//Timing

For long sequences, the difference can be significant:

test=Range[100];test[[{50,75}]]=5;

Position[test,5]

MatchQ[test,{___,x_,___,x_,___,x_,___}]//Timing

teststr=FromCharacterCode[test];

StringPosition[teststr,FromCharacterCode[5]]

StringMatchQ[teststr,StringExpression[___,x_,___,x_,___,x_,___]]//Timing

Implementation Details

String pattern matching in the Wolfram Language is built on top of the PCRE (Perl Compatible Regular Expressions) library by Philip Hazel [1].

In some cases the pre-5.1 Wolfram Language algorithms are used (for example, when the pattern is just a single, literal string).

Any symbolic string pattern is first translated to a regular expression. You can see this translation by using the internal StringPattern`PatternConvert function:

StringPattern`PatternConvert["a"|""~~DigitCharacter..]//InputForm

The first element returned is the regular expression, while the rest of the elements have to do with conditions, replacement rules, and named patterns.

The regular expression is then compiled by PCRE, and the compiled version is cached for future use when the same pattern appears again. The translation from symbolic string pattern to regular expression only happens once.

Wolfram Language conditions in the pattern are handled by external call-outs from the PCRE library to the Wolfram Language evaluator, so this will slow down the matching.

Explicit (RefLink[RegularExpression, paclet:ref/RegularExpression]) objects embedded into a general string pattern will be spliced into the final regular expression (surrounded by noncapturing parentheses "(?:...)"), so the counting of named patterns can become skewed compared to what you might expect.

Because PCRE currently does not support preset character classes with characters beyond character code 255, the word and letter character classes (such as (RefLink[WordCharacter, paclet:ref/WordCharacter]) and (RefLink[LetterCharacter, paclet:ref/LetterCharacter])) only include character codes in the Unicode range 0–255. Thus (RefLink[LetterCharacter, paclet:ref/LetterCharacter]) and _?(RefLink[LetterQ, paclet:ref/LetterQ]) do not give equivalent results beyond character code 255.

Because of a similar PCRE restriction, case-insensitive matching (for example, with (RefLink[IgnoreCase, paclet:ref/IgnoreCase])->(RefLink[True, paclet:ref/True])) will only apply to letters in the Unicode range 0–127 (that is, the normal English letters "a"–"z" and "A"–"Z").

References

[1] Hazel, P. "PCRE—Perl Compatible Regular Expressions." 2004. www.pcre.org

[2] Friedl, J. E. F. Mastering Regular Expressions, 2nd ed. O'Reilly & Associates, 2002.