Bitvantage.SharpTextFsm 1.3.0

Bitvantage.SharpTextFsm

SharpTextFsm is a .NET implementation of the Google TextFSM Python module. TextFSM templates match semi-formated line delimited text using an articulated regular expression state machine. TextFSM templates are particularly well suited for parsing CLI output.

Installing via NuGet Package Manager

PM> NuGet\Install-Package Bitvantage.SharpTextFsm

TextFSM Resources

This documentation is focused on details that are specific to SharpTextFSM, for general information on TextFSM templates or regular expressions the following resources may be helpful:

• Google TextFSM Wiki
• Getting Started with TextFSM for Python
• Awesome Regex Resources
• Network to Code Text FSM Templates
  An extensive library of templates for many vendors' equipment that can be used directly by SharpTextFSM.

Quick Start

A simple example of how to parse the output from the 'show ip arp' command from a Cisco IOS switch into a C# record.

Combined C# Record Object and TextFSM Template

record ShowIpArp : ITemplate
{
    public string Protocol { get; set; }
    public IPAddress IpAddress { get; set; }
    [Variable(ThrowOnConversionFailure = false)]
    public long? Age { get; set; }
    public string MacAddress { get; set; }
    public string Type { get; set; }
    public string Interface { get; set; }

    string ITemplate.TextFsmTemplate =>
        """
        Value PROTOCOL (\S+)
        Value IP_ADDRESS (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
        Value AGE (-|\d+)
        Value MAC_ADDRESS ([a-f0-9]{4}\.[a-f0-9]{4}\.[a-f0-9]{4})
        Value TYPE (\S+)
        Value INTERFACE (\S+)
            
        Start
         ^Protocol\s+Address\s+Age\(min\)\s+Hardware Addr\s+Type\s+Interface -> Entry
         ^.* -> Error
            
        Entry
         ^${PROTOCOL}\s+${IP_ADDRESS}\s+${AGE}\s+${MAC_ADDRESS}\s+${TYPE}(\s+${INTERFACE})?$$ -> Record
         ^.* -> Error
        """;
}

Example Usage

class Example
{
    public void Test()
    {
        var data = """
            Protocol  Address              Age(min)       Hardware Addr     Type      Interface
            Internet  172.16.233.229       -              0000.0c59.f892    ARPA      Ethernet0/0
            Internet  172.16.233.218       -              0000.0c07.ac00    ARPA      Ethernet0/0
            Internet  172.16.233.19        -              0000.0c63.1300    ARPA      Ethernet0/0
            Internet  172.16.233.209       -              0000.0c36.6965    ARPA      Ethernet0/0
            Internet  172.16.168.11        -              0000.0c63.1300    ARPA      Ethernet0/0
            Internet  172.16.168.254       9              0000.0c36.6965    ARPA      Ethernet0/0
            Internet  10.0.0.0             -              aabb.cc03.8200    SRP-A
            """;

        var template = Template.FromType<ShowIpArp>();
        var results = template.Run<ShowIpArp>(data).ToList();
    }
}

Results

{ShowIpArp { Protocol = Internet, IpAddress = 172.16.233.229,   Age = -, MacAddress = 0000.0c59.f892, Type = ARPA,  Interface = Ethernet0/0 }}
{ShowIpArp { Protocol = Internet, IpAddress = 172.16.233.218,   Age = -, MacAddress = 0000.0c07.ac00, Type = ARPA,  Interface = Ethernet0/0 }}
{ShowIpArp { Protocol = Internet, IpAddress = 172.16.233.19,    Age = -, MacAddress = 0000.0c63.1300, Type = ARPA,  Interface = Ethernet0/0 }}
{ShowIpArp { Protocol = Internet, IpAddress = 172.16.233.209,   Age = -, MacAddress = 0000.0c36.6965, Type = ARPA,  Interface = Ethernet0/0 }}
{ShowIpArp { Protocol = Internet, IpAddress = 172.16.168.11,    Age = -, MacAddress = 0000.0c63.1300, Type = ARPA,  Interface = Ethernet0/0 }}
{ShowIpArp { Protocol = Internet, IpAddress = 172.16.168.254,   Age = 9, MacAddress = 0000.0c36.6965, Type = ARPA,  Interface = Ethernet0/0 }}
{ShowIpArp { Protocol = Internet, IpAddress = 10.0.0.0,         Age = -, MacAddress = aabb.cc03.8200, Type = SRP-A, Interface =             }}

'Value' Bindings

Template TextFSM Values are automatically bound to similarly named public fields and properties within the type.

By default the case and the '_' are ignored. This behavior can be changed by decorating the class with the TemplateAttribute.

For example to match using exact match logic and case-insensitive logic:

[Template(MappingStrategy.Exact | IgnoreCase)]
record Test
{
  ...
}

Explicit 'Value' Binding

A field or property can be explicitly bound using the VariableAttribute. Explicitly bound fields and properties take precedence over automatically bound fields and properties.

[Variable(Name = "MY_VALUE_NAME")]
public long MagicNumber { get; set; }

Ignoring a Field or Property

A field or property can be ignored by setting the Ignore flag in the VariableAttribute.

[Variable(Ignore = true)]
public long MagicNumber { get; set; }

Type Conversion

If a type converter is not specified and the underlying type has a TryParse() or Parse() method, the built-in GenericTryParseConverter or GenericParseConverter type converter will automatically be configured.

Setting an Explicit Type Converter

[Variable(TypeConverter = typeof(MyTypeConverter)]
public long ValueField { get; set; }

Type Conversion Failure

If the type conversion fails, a TemplateTypeConversionException exception will be thrown. This behavior can be changed by setting the ThrowOnConversionFailure property to false.

When the ThrowOnConversionFailure property is set to false, values that fail to parse are set to the type's default value. This value can be changed by setting the DefaultValue property. The default value is specified as a string value and will be converted using the associated type converter.

[Variable(ThrowOnConversionFailure=false)]
public long? Length { get; set; }

[Variable(ThrowOnConversionFailure=false)]
public long MagicNumber { get; set; }

[Variable(ThrowOnConversionFailure=false, DefaultValue="42")]
public long Answer { get; set; }

Built-In Type Converters

Enum Conversion

By default, enums are automatically parsed using the built-in type converter EnumConverter. Member values are matched in a case-insensitive way, and aliases can be attached by using the EnumAliasAttribute.

enum Animal
{
    None,
    Armadillo,
    Blobfish,
    Capybara,
    Fossa,
    [EnumAlias("Ghost Shark")]
    GhostShark,
    [EnumAlias("Goblin Shark")]
    GoblinShark,
    Hagfish,
    Hoatzin,
    Pangolin,
    Platypus,
    [EnumAlias("Sea Hog")]
    [EnumAlias("Sea Pig")]
    [EnumAlias("Sea Piggy")]
    [EnumAlias("Sea Swine")]
    SeaPig,
    Sloth,
    Tarsier,
    Uakari,
}

Custom Type Converters

A custom type converter can be set in the Converter property.

[Variable(Converter=typeof(MyConverter))]
public long MagicNumber { get; set; }

To create a custom type converter extend ValueConverter<T>.

public class MyConverter : ValueConverter<long>
{
    public override bool TryConvert(string value, out long convertedValue)
    {
        if(!int.TryParse(value, out var parsedValue))
        {
            convertedValue = 0;
            return false;
        }

        convertedValue = 42 + parsedValue;
        return true;
    }
}

List Conversion

When the TextFSM value definition has the 'List' option set, the underlying collection is an array, generic List<T>, or a ReadOnlyCollection<T>, and the ListCreator property is not set a list creator is automatically assigned. The list creator is responsible for creating an object of the target type from an array of typed values.

Setting Explicit Type Converters

[Variable(TypeConverter = typeof(ListCreator)]
public string Value { get; set; }

Custom List Creator

To create a custom type converter extend from ListCreator<TList, TItem>.

class CommaSeparatedList : ListCreator<string,string>
{
    public override string Create(string[] values)
    {
        return string.Join(",", values);
    }
}

Built-in List Creators

Transformers

Before a value is converted and assigned, it can be transformed from one value to another using the TransformerAttribute.

[Transformer("old value", "new value")] // "old value" -> "new value"
[Transformer("-", null)]                // Value is not set if the string value is null
[Transformer("*", "")]                  // Value is not set if SkipEmpty is true (default) or empty if SkipEmpty is false
public string Value { get; set; }

Raw Rows

The TextFSM row that is used to generate the record can be included in the record. This can be useful for reference purposes or used by the internal logic of the class.

[RawRow]
public Row ExampleRawRow { get; set; }

Validation and Post Processing

When the ITemplateValidator interface is implemented, a custom method is called after the record is created. The record can be modified, and post-processing tasks can be performed. The raw row that was used to generate the instance of the record is provided for post-processing purposes.

If the function returns false, the record will not be added to the result set.

internal class Test : ITemplate, ITemplateValidator
{
    public long MyProperty { get; set; }

    ...

    bool ITemplateValidator.Validate(Row row)
    {
        if(MyProperty == 0)
            return false;

        if(MyProperty == 5)
            MyProperty = int.Parse((string)row["MyUnboundValue"]);

        if(MyProperty > 10)
            MyProperty = 10;
           
        return true;
    }
}

Regex Value Defenition

Regular expression values function similarly to regular values but do not implicitly capture. They can be used in both rules and other values, enhancing readability and reducing the need for repeating the same expressions.

Value Regex CUTE_ANIMALS (dog|cat|panda|rabit)
Value Regex UGLY_ANIMALS (pug)
Value Regex ALL_ANIMALS (${CUTE_ANIMALS}|${UGLY_ANIMALS})
Value EVERYTHING (${ALL_ANIMALS})

Start
 ^${CUTE_ANIMALS}
 ^${UGLY_ANIMALS}
 ^${ALL_ANIMALS}
 ^${EVERYTHING}

* This feature is not present in the reference implementation of TextFSM.

Library Patterns

Common patterns are available in the built-in Regex library and may be referenced just like they were explicitly defined. Many of the patterns were either borrowed from or inspired by Grok.

Value MAC_ADDRESS (${_MAC_ADDRESS})

Start
 ^MAC_ADDRESS -> Record

Loop Free Continue Line Action with State Change

The reference implementation of TextFSM does not allow combining the 'Continue' line action with a state transition to ensure a loop-free state machine. While this restriction is both reasonable and well intended, it complicates numerous common use cases and discourages using states to validate data.

SharpTextFsm relaxes this restriction by allowing 'Continue' line actions with a state transition provided that doing so cannot produce a loop.

A state machine has a loop if it CAN indefinitely process the same line without advancing to the next line. In practice, this means there cannot be a state with a rule that includes a continue line action and a state transition that jumps to another state that can return to the previous state, either directly or indirectly, through rules that specify the continue line action.

The following is an example of a valid SharpTextFsm template that uses both the Continue line action and a state change:

State1
 ^.* -> Continue State2
State2
 ^.* -> Continue State3
 ^.* -> State1
State3
 ^X.* -> Record
 ^Y.* -> State1

* This feature is not present in the reference implementation of TextFsm.

~Global State

Rules in the ~Global state are evaluated each time a new line is read and before the rules associated with the current state.

Placing rules in the ~Global state can be useful for handling content that appears in numerous states, such as comments, and for transitioning to states when the previous state ends without a clear trigger.

Since rules cannot transition to the ~Global state, rules in the ~Global state can not create a loop, therefore, using the 'Continue' line action with a state transition is always allowed.

Value Test (.*)

~Global
 ^\s*#
 ^X -> Continue XState

Start
 ^.*

XState
 ^X${Test}

* This feature is not present in the reference implementation of TextFsm.

~Global State Filters

State filters can be attached to rules in the ~Global state to limit the states that the rule applies to. The list of states can be negated by prefixing the state list with a '^'.

~Global
 [State1,State2,State3]
 ^X -> Continue State4

 [^State1,State2]
 ^Y -> State1

Metadata Value Option

Using the 'Metadata' value option, state machine data can be included in the result set.

Normally, the last value before a row is recorded will appear in the result set. However, if the List option is specified, an entry is added to the list each time a rule is matched.

Value Metadata LAST_LINE_NUMBER (Line)
Value Metadata,List INPUT_TEXT (Text)

* This feature is not present in the reference implementation of TextFsm.

Metadata Patterns

The metadata pattern is a value from the below list and not a regular expression.

Parsing to a Non-Generic Result Set

To produce an untyped result set similar to the reference implementation of TextFSM.

var results = template.Run(string data)

The results will be a list of rows that contain either a string value or an array of string values.

Parsing to a Dynamic Result Set

To produce an untyped dynamic result set.

var results = template.Run(string data).ToDynamic()

Troubleshooting

Explain() Function

The explain function explains in detail each step of the match process and can help to understand why a template is not working as expected.

var template = Template.FromType<MyTemplate>().

var explain = template.Explain(data);

Known Limitations

In general templates for the reference implementation of TextFSM should work with SharpTextFSM, and templates for SharpTextFSM should work with the reference implementation of TestFSM provided they do not use any extended features.

Known limitations include:

• The .NET regular expression engine is subtly different from the TextFSM regular expression engine, which can create incompatibility between the two implementations.
• Inline capture groups in 'Value' statements are not supported, for example, 'Value INLINE_CAPTURE_GROUP ((?P<FirstGroup>expression) (?P<SecondGroup>expression))'. In the reference implementation, a match would return a separate group inside of each existing 'Value' capture, in .NET regular expressions the syntax for a named capture group is different, and inline capture groups do not map cleanly to type mapping.