Repository files navigation

Modern, minimal, fast, zero allocation, reading and writing of separated values(csv,tsv etc.). Cross-platform, trimmable and AOT/NativeAOT compatible.Featuring an opinionated API design and pragmatic implementation targetted atmachine learning use cases.

⭐ Please star this project if you like it. ⭐

🌃 Modern - utilizes features such asSpan<T>,Generic Math(ISpanParsable<T>/ISpanFormattable),ref struct,ArrayPool<T>and similar from.NET 7+ and C#11+ for a modernand highly efficient implementation.

🔎 Minimal - a succinct yet expressive API with few options and no hiddenchanges to input or output. What you read/write is what you get. E.g. by defaultthere is no "automatic" escaping/unescaping of quotes or trimming of spaces. Toenable this seeSepReaderOptions andUnescaping andTrimming. SeeSepWriterOptions forEscaping.

🚀 Fast - blazing fast with both architecture specific and cross-platformSIMD vectorized parsing incl. 64/128/256/512-bit paths e.g. AVX2, AVX-512 (.NET8.0+), NEON. UsescsFastFloat forfast parsing of floating points. Seedetailedbenchmarks for cross-platform results.

🌪️ Multi-threaded - unparalleled speed with highly efficient parallel CSVparsing that isup to 35x faster thanCsvHelper, seeParallelEnumerate andbenchmarks.

🌀 Async support - efficientValueTask basedasync/await support.Requires C# 13.0+ and for .NET 9.0+ includesSepReader implementingIAsyncEnumerable<>. SeeAsync Support for details.

🗑️ Zero allocation - intelligent and efficient memory management allowingfor zero allocations after warmup incl. supporting use cases of reading orwriting arrays of values (e.g. features) easily without repeated allocations.

✅ Thorough tests - great code coverage and focus on edge case testing incl.randomizedfuzz testing.

🌐 Cross-platform - works on any platform, any architecture supported byNET. 100% managed and written in beautiful modern C#.

✂️ Trimmable and AOT/NativeAOT compatible - no problematic reflection ordynamic code generation. Hence, fullytrimmableandAhead-of-Timecompatible. With a simple console tester program executable possible in just afew MBs. 💾

🗣️ Opinionated and pragmatic - conforms to the essentials ofRFC-4180, but takes an opinionated andpragmatic approach towards this especially with regards to quoting and lineends. See sectionRFC-4180.

Example |Naming and Terminology |API |Limitations and Constraints |Comparison Benchmarks |Example Catalogue |RFC-4180 |FAQ |Public API Reference

vartext="""           A;B;C;D;E;F           Sep;🚀;1;1.2;0.1;0.5           CSV;✅;2;2.2;0.2;1.5           """;usingvarreader=Sep.Reader().FromText(text);// Infers separator 'Sep' from headerusingvarwriter=reader.Spec.Writer().ToText();// Writer defined from reader 'Spec'// Use .FromFile(...)/ToFile(...) for filesvaridx=reader.Header.IndexOf("B");varnms=new[]{"E","F"};foreach(varreadRowinreader)// Read one row at a time{vara=readRow["A"].Span;// Column as ReadOnlySpan<char>varb=readRow[idx].ToString();// Column to string (might be pooled)varc=readRow["C"].Parse<int>();// Parse any T : ISpanParsable<T>vard=readRow["D"].Parse<float>();// Parse float/double fast via csFastFloatvars=readRow[nms].Parse<double>();// Parse multiple columns as Span<T>// - Sep handles array allocation and reuseforeach(refvarvins){v*=10;}usingvarwriteRow=writer.NewRow();// Start new row. Row written on Dispose.writeRow["A"].Set(a);// Set by ReadOnlySpan<char>writeRow["B"].Set(b);// Set by stringwriteRow["C"].Set($"{c*2}");// Set via InterpolatedStringHandler, no allocswriteRow["D"].Format(d/2);// Format any T : ISpanFormattablewriteRow[nms].Format(s);// Format multiple columns directly// Columns are added on first access as ordered, header written when first row written}varexpected="""               A;B;C;D;E;F               Sep;🚀;2;0.6;1;5               CSV;✅;4;1.1;2;15               """;// Empty line at end is for line ending,// which is always written.Assert.AreEqual(expected,writer.ToString());// Above example code is for demonstration purposes only.// Short names and repeated constants are only for demonstration.

Sep uses naming and terminology that is not based onRFC-4180, butis more tailored to usage in machine learning or similar. Additionally, Septakes a pragmatic approach towards names by using short names and abbreviationswhere it makes sense and there should be no ambiguity given the context. Thatis, usingSep forSeparator andCol forColumn to keep code succinct.

Besides being the succinct name of the library,Sep is both the main entrypoint to using the library and the container for a validated separator. That is,Sep is basically defined as:

publicreadonlyrecordstructSep(charSeparator);

The separatorchar is validated upon construction and is guaranteed to bewithin a limited range and not being achar like" (quote) or similar. Thiscan be seen insrc/Sep/Sep.cs. The separator is constrainedalso for internal optimizations, so you cannot use anychar as a separator.

⚠ Note that all types are within the namespacenietras.SeparatedValues and notSep since it is problematic to have a type and a namespace with the same name.

To get started you can useSep as the static entry point to building either areader or writer. That is, forSepReader:

usingvarreader=Sep.Reader().FromFile("titanic.csv");

where.Reader() is a convenience method corresponding to:

usingvarreader=Sep.Auto.Reader().FromFile("titanic.csv");

whereSep? Auto => null; is a static property that returnsnull for anullableSep to signify that the separator should be inferred from the firstrow, which might be a header. If the first row does not contain any of the bydefault supported separators or there are no rows, the default separator will beused.

⚠ Note Sep uses; as the default separator, since this is what was used in aninternal proprietary library which Sep was built to replace. This is also toavoid issues with comma, being used as a decimal separator in some locales.Without having to resort to quoting.

If you want to specify the separator you can write:

usingvarreader=Sep.New(',').Reader().FromFile("titanic.csv");

or

varsep=newSep(',');usingvarreader=sep.Reader().FromFile("titanic.csv");

Similarly, forSepWriter:

usingvarwriter=Sep.Writer().ToFile("titanic.csv");

or

usingvarwriter=Sep.New(',').Writer().ToFile("titanic.csv");

where you have to specify a valid separator, since it cannot be inferred. Tofascillitate easy flow of the separator andCultureInfo bothSepReader andSepWriter expose aSpec property of typeSepSpec that simply defines thosetwo. This means you can write:

usingvarreader=Sep.Reader().FromFile("titanic.csv");usingvarwriter=reader.Spec.Writer().ToFile("titanic-survivors.csv");

where thewriter then will use the separator inferred by the reader, forexample.

In general, both reading and writing follow a similar pattern:

Sep/Spec => SepReaderOptions => SepReader => Row => Col(s) => Span/ToString/ParseSep/Spec => SepWriterOptions => SepWriter => Row => Col(s) => Set/Format

where each continuation flows fluently from the preceding type. For example,Reader() is an extension method toSep orSepSpec that returns aSepReaderOptions. Similarly,Writer() is an extension method toSep orSepSpec that returns aSepWriterOptions.

SepReaderOptions andSepWriterOptions are optionally configurable.That and the APIs for reader and writer is covered in the following sections.

For a complete example, see theexample above or theReadMeTest.cs.

⚠ Note that it is important to understand that SepRow/Col/Cols areref structs(please follow theref struct link and understand how this limits the usage ofthose). This is due to these types being simplefacades or indirections to theunderlying reader or writer. That means you cannot use LINQ or create an arrayof all rows likereader.ToArray(). While for .NET9+ the reader is nowIEnumerable<> sinceref structs can now be used in interfaces that havewhere T: allows ref structthis still does not mean it is LINQ compatible. Hence, if you need store per rowstate or similar you need to parse or copy to different types instead. The sameapplies toCol/Cols which point to internal state that is also reused. Thisis to avoid repeated allocations for each row and get the best possibleperformance, while still defining a well structured and straightforward API thatguides users to relevant functionality. SeeWhy SepReader Was Not IEnumerableUntil .NET 9 and Is Not LINQCompatiblefor more.

⚠ For a full overview of public types and methods seePublic APIReference.

SepReader API has the following structure (in pseudo-C# code):

usingvarreader=Sep.Reader(o=>o).FromFile/FromText/From...;varheader=reader.Header;var_=header.IndexOf/IndicesOf/NamesStartingWith...;foreach(varrowinreader){var_=row[colName/colNames].Span/ToString/Parse<T>...;var_=row[colIndex/colIndices].Span/ToString/Parse<T>...;}

That is, to useSepReader follow the points below:

1. Optionally defineSep or use default automatically inferred separator.
2. Specify reader with optional configuration ofSepReaderOptions. Forexample, if a csv-file does not have a header this can be configured via:

   Sep.Reader(o=>owith{HasHeader=false})

   For all options seeSepReaderOptions.
3. Specify source e.g. file, text (string),TextWriter, etc. viaFromextension methods.
4. Optionally access the header. For example, to get all columns starting withGT_ use:

   varcolNames=header.NamesStarting("GT_");varcolIndices=header.IndicesOf(colNames);

5. Enumerate rows. One row at a time.
6. Access a column by name or index. Or access multiple columns with names andindices.Sep internally handles pooled allocation and reuse of arrays formultiple columns.
7. UseSpan to access the column directly as aReadOnlySpan<char>. Or useToString to convert to astring. Or useParse<T> whereT : ISpanParsable<T> to parse the columnchars to a specific type.

The following options are available:

/// <summary>/// Specifies the separator used, if `null` then automatic detection/// is used based on first row in source./// </summary>publicSep?Sep{get;init;}=null;/// <summary>/// Specifies initial internal `char` buffer length./// </summary>/// <remarks>/// The length will likely be rounded up to the nearest power of 2. A/// smaller buffer may end up being used if the underlying source for <see/// cref="System.IO.TextReader"/> is known to be smaller. Prefer to keep the/// default length as that has been tuned for performance and cache sizes./// Avoid making this unnecessarily large as that will likely not improve/// performance and may waste memory./// </remarks>publicintInitialBufferLength{get;init;}=SepDefaults.InitialBufferLength;/// <summary>/// Specifies the culture used for parsing./// May be `null` for default culture./// </summary>publicCultureInfo?CultureInfo{get;init;}=SepDefaults.CultureInfo;/// <summary>/// Indicates whether the first row is a header row./// </summary>publicboolHasHeader{get;init;}=true;/// <summary>/// Specifies <see cref="IEqualityComparer{T}" /> to use/// for comparing header column names and looking up index./// </summary>publicIEqualityComparer<string>ColNameComparer{get;init;}=SepDefaults.ColNameComparer;/// <summary>/// Specifies the method factory used to convert a column span/// of `char`s to a `string`./// </summary>publicSepCreateToStringCreateToString{get;init;}=SepToString.Direct;/// <summary>/// Disables using [csFastFloat](https://github.com/CarlVerret/csFastFloat)/// for parsing `float` and `double`./// </summary>publicboolDisableFastFloat{get;init;}=false;/// <summary>/// Disables checking if column count is the same for all rows./// </summary>publicboolDisableColCountCheck{get;init;}=false;/// <summary>/// Disables detecting and parsing quotes./// </summary>publicboolDisableQuotesParsing{get;init;}=false;/// <summary>/// Unescape quotes on column access./// </summary>/// <remarks>/// When true, if a column starts with a quote then the two outermost quotes/// are removed and every second inner quote is removed. Note that/// unquote/unescape happens in-place, which means the <see/// cref="SepReader.Row.Span" /> will be modified and contain "garbage"/// state after unescaped cols before next col. This is for efficiency to/// avoid allocating secondary memory for unescaped columns. Header/// columns/names will also be unescaped./// Requires <see cref="DisableQuotesParsing"/> to be false./// </remarks>publicboolUnescape{get;init;}=false;/// <summary>/// Option for trimming spaces (` ` - ASCII 32) on column access./// </summary>/// <remarks>/// By default no trimming is done. See <see cref="SepTrim"/> for options./// Note that trimming may happen in-place e.g. if also unescaping, which/// means the <see cref="SepReader.Row.Span" /> will be modified and contain/// "garbage" state for trimmed/unescaped cols. This is for efficiency to/// avoid allocating secondary memory for trimmed/unescaped columns. Header/// columns/names will also be trimmed. Note that only the space ` ` (ASCII/// 32) character is trimmed, not any whitespace character./// </remarks>publicSepTrimTrim{get;init;}=SepTrim.None;/// <summary>/// Forwarded to <see/// cref="System.Threading.Tasks.ValueTask.ConfigureAwait(bool)"/> or/// similar when async methods are called./// </summary>publicboolAsyncContinueOnCapturedContext{get;init;}=false;

While great care has been taken to ensure Sep unescaping of quotes is bothcorrect and fast, there is always the question of how does one respond toinvalid input.

The below table tries to summarize the behavior of Sep vs CsvHelper and Sylvan.Note that all do the same for valid input. There are differences for how invalidinput is handled. For Sep the design choice has been based on not wanting tothrow exceptions and to use a principle that is both reasonably fast and simple.

· (middle dot) is whitespace to make this visible

¹ CsvHelper withBadDataFound = null

² Sep withUnescape = true inSepReaderOptions

Sep supports trimming by theSepTrim flags enum, whichhas two options as documented there. Below the result of both trimming andunescaping is shown in comparison to CsvHelper. Note unescaping is enabled forall results shown. It is possible to trim without unescaping too, of course.

As can be seen Sep supports a simple principle of trimmingbefore andafterunescaping with trimming before unescaping being important for unescaping ifthere is a starting quote after spaces.

· (middle dot) is whitespace to make this visible

¹ CsvHelper withTrimOptions.Trim | TrimOptions.InsideQuotes

² Sep withSepTrim.All = SepTrim.Outer | SepTrim.AfterUnescape inSepReaderOptions

Debuggability is an important part of any library and while this is still a workin progress for Sep,SepReader does have a unique feature when looking at itand it's row or cols in a debug context. Given the below example code:

vartext="""           Key;Value           A;"1           2           3"           B;"Apple           Banana           Orange           Pear"           """;usingvarreader=Sep.Reader().FromText(text);foreach(varrowinreader){// Hover over reader, row or col when breaking herevarcol=row[1];if(Debugger.IsAttached&&row.RowIndex==2){Debugger.Break();}Debug.WriteLine(col.ToString());}

and you are hovering overreader when the break is triggered then this willshow something like:

String Length=55

That is, it will show information of the source for the reader, in this case astring of length 55.

If you are hovering overrow then this will show something like:

  2:[5..9] = "B;\"Apple\r\nBanana\r\nOrange\r\nPear\""

This has the format shown below.

<ROWINDEX>:[<LINENUMBERRANGE>] = "<ROW>"

Note how this shows line number range[FromIncl..ToExcl], as in C#rangeexpression,so that one can easily find the row in question innotepad or similar. Thismeans Sep has to track line endings inside quotes and is an example of a featurethat makes Sep a bit slower but which is a price considered worth paying.

GitHub doesn't show line numbers in code blocks so consider copying theexample text to notepad or similar to see the effect.

Additionally, if you expand therow in the debugger (e.g. via the smalltriangle) you will see each column of the row similar to below.

00:'Key'   = "B"01:'Value' = "\"Apple\r\nBanana\r\nOrange\r\nPear\""

If you hover overcol you should see:

"\"Apple\r\nBanana\r\nOrange\r\nPear\""

As mentioned earlier Sep only allows enumeration and access to one row at a timeandSepReader.Row is just a simplefacade or indirection to the underlyingreader. This is why it is defined as aref struct. In fact, the followingcode:

usingvarreader=Sep.Reader().FromText(text);foreach(varrowinreader){}

can also be rewritten as:

usingvarreader=Sep.Reader().FromText(text);while(reader.MoveNext()){varrow=reader.Current;}

whererow is just afacade for exposing row specific functionality. That is,row is still basically thereader underneath. Hence, let's look at usingLINQ withSepReader implementingIEnumerable<SepReader.Row> and theRownot being aref struct. Then, you would be able to write something like below:

usingvarreader=Sep.Reader().FromText(text);SepReader.Row[]rows=reader.ToArray();

GivenRow is just a facade for the reader, this would be equivalent towriting:

usingvarreader=Sep.Reader().FromText(text);SepReader[]rows=reader.ToArray();

which hopefully makes it clear why this is not a good thing. The array wouldeffectively be the reader repeated several times. If this would have to besupported one would have to allocate memory for each row always, which wouldbasically be no different than aReadLine approach as benchmarked inComparison Benchmarks.

This is perhaps also the reason why no other efficient .NET CSV parser (known toauthor) implements an API pattern like Sep, but instead let the reader defineall functionality directly and hence only let's you access the current row andcols on that. This API, however, is in this authors opinion not ideal and can bea bit confusing, which is why Sep is designed like it is. The downside is theabove caveat.

The main culprit above is that for exampleToArray() would store aref struct in a heap allocated array, the actual enumeration is not a problem andhence implementingIEnumerable<SepReader.Row> is not the problem as such. Theproblem was that prior to .NET 9 it was not possible to implement this interfacewithT being aref struct, but with C# 13allows ref struct and .NET 9having annotated such interfaces it is now possible and you can assignSepReader toIEnumerable, but most if not all of LINQ will still not work asshown below.

vartext="""           Key;Value           A;1.1           B;2.2           """;usingvarreader=Sep.Reader().FromText(text);IEnumerable<SepReader.Row>enumerable=reader;// Currently, most LINQ methods do not work for ref types. See below.//// The type 'SepReader.Row' may not be a ref struct or a type parameter// allowing ref structs in order to use it as parameter 'TSource' in the// generic type or method 'Enumerable.Select<TSource,// TResult>(IEnumerable<TSource>, Func<TSource, TResult>)'//// enumerable.Select(row => row["Key"].ToString()).ToArray();

CallingSelect should in principle be possible if this was annotated withallows ref struct, but it isn't currently.

If you want to use LINQ or similar you have to first parse or transform the rowsinto some other type and enumerate it. This is easy to do and instead ofcounting lines you should focus on how such enumeration can be easily expressedusing C# iterators (akayield return). With local functions this can be doneinside a method like:

vartext="""           Key;Value           A;1.1           B;2.2           """;varexpected=new(stringKey,doubleValue)[]{("A",1.1),("B",2.2),};usingvarreader=Sep.Reader().FromText(text);varactual=Enumerate(reader).ToArray();CollectionAssert.AreEqual(expected,actual);staticIEnumerable<(stringKey,doubleValue)>Enumerate(SepReaderreader){foreach(varrowinreader){yieldreturn(row["Key"].ToString(),row["Value"].Parse<double>());}}

Now if instead refactoring this to something LINQ-compatible by defining acommonEnumerate or similar method it could be:

vartext="""           Key;Value           A;1.1           B;2.2           """;varexpected=new(stringKey,doubleValue)[]{("A",1.1),("B",2.2),};usingvarreader=Sep.Reader().FromText(text);varactual=Enumerate(reader,    row=>(row["Key"].ToString(),row["Value"].Parse<double>())).ToArray();CollectionAssert.AreEqual(expected,actual);staticIEnumerable<T>Enumerate<T>(SepReaderreader,SepReader.RowFunc<T>select){foreach(varrowinreader){yieldreturnselect(row);}}

In fact, Sep provides such a convenience extension method. And, discounting theEnumerate method, this does have less boilerplate, but not really moreeffective lines of code. The issue here is that this tends to favor factoringcode in a way that can become very inefficient quickly. Consider if one wantedto only enumerate rows matching a predicate onKey which meant only 1% of rowswere to be enumerated e.g.:

vartext="""           Key;Value           A;1.1           B;2.2           """;varexpected=new(stringKey,doubleValue)[]{("B",2.2),};usingvarreader=Sep.Reader().FromText(text);varactual=reader.Enumerate(    row=>(row["Key"].ToString(),row["Value"].Parse<double>())).Where(kv=>kv.Item1.StartsWith('B')).ToArray();CollectionAssert.AreEqual(expected,actual);

This means you are still parsing the double (which is magnitudes slower thangetting just the key) for all rows. Imagine if this was an array of floatingpoints or similar. Not only would you then be parsing a lot of values you wouldalso be allocated 99x arrays that aren't used after filtering withWhere.

Instead, you should focus on how to express the enumeration in a way that isboth efficient and easy to read. For example, the above could be rewritten as:

vartext="""           Key;Value           A;1.1           B;2.2           """;varexpected=new(stringKey,doubleValue)[]{("B",2.2),};usingvarreader=Sep.Reader().FromText(text);varactual=Enumerate(reader).ToArray();CollectionAssert.AreEqual(expected,actual);staticIEnumerable<(stringKey,doubleValue)>Enumerate(SepReaderreader){foreach(varrowinreader){varkeyCol=row["Key"];if(keyCol.Span.StartsWith("B")){yieldreturn(keyCol.ToString(),row["Value"].Parse<double>());}}}

To accomodate this Sep provides an overload forEnumerate that is similar to:

staticIEnumerable<T>Enumerate<T>(thisSepReaderreader,SepReader.RowTryFunc<T>trySelect){foreach(varrowinreader){if(trySelect(row,outvarvalue)){yieldreturnvalue;}}}

With this the above customEnumerate can be replaced with:

vartext="""           Key;Value           A;1.1           B;2.2           """;varexpected=new(stringKey,doubleValue)[]{("B",2.2),};usingvarreader=Sep.Reader().FromText(text);varactual=reader.Enumerate((SepReader.Rowrow,out(stringKey,doubleValue)kv)=>{varkeyCol=row["Key"];if(keyCol.Span.StartsWith("B")){kv=(keyCol.ToString(),row["Value"].Parse<double>());returntrue;}kv=default;returnfalse;}).ToArray();CollectionAssert.AreEqual(expected,actual);

Note how this is pretty much the same length as the previous customEnumerate.Also worse due to how C# requires specifying types forout parameters whichthen requires all parameter types for the lambda to be specified. Hence, in thiscase the customEnumerate does not take significantly longer to write and is alot more efficient than using LINQ.Where (also avoids allocating a string forkey for each row) and is easier to debug and perhaps even read. All examplesabove can be seen inReadMeTest.cs.

There is a strong case for having an enumerate API though and that is forparallelized enumeration, which will be discussed next.

As discussed in the previous section Sep providesEnumerate convenienceextension methods, that should be used carefully. Alongside these there areParallelEnumerate extension methods that provide very efficient multi-threadedenumeration. Seebenchmarks for numbers andPublicAPI Reference.

ParallelEnumerate is build on top of LINQAsParallel().AsOrdered() and willreturn exactly the same asEnumerate but with enumeration parallelized. Thiswill use more memory during execution and as many threads as possible via the.NET thread pool. When usingParallelEnumerate one should, therefore (asalways), be certain the provided delegate does not refer to or change anymutable state.

ParallelEnumerate comes with a lot of overhead compared to single-threadedforeach orEnumerate and should be used carefully based on measuring anypotential benefit. Sep goes a long way to make this very efficient by usingpooled arrays and parsing multiple rows in batches, but if the source only has afew rows then any benefit is unlikely.

Due toParallelEnumerate being based on batches of rows it is also importantnot to "abuse" it in-place of LINQAsParallel. The idea is to use it forparsing rows, not for doing expensive per row operations like loading an imageor similar. In that case, you are better off usingAsParallel() afterParallelEnumerate orEnumerate similarly to:

usingvarreader=Sep.Reader().FromFile("very-long.csv");varresults=reader.ParallelEnumerate(ParseRow).AsParallel().AsOrdered().Select(LoadData)// Expensive load.ToList();

As a rule of thumb if the time per row exceeds 1 millisecond consider moving theexpensive work to afterParallelEnumerate/Enumerate,

SepWriter API has the following structure (in pseudo-C# code):

usingvarwriter=Sep.Writer(o=>o).ToFile/ToText/To...;foreach(vardatainEnumerateData()){usingvarrow=writer.NewRow();var_=row[colName/colNames].Set/Format<T>...;var_=row[colIndex/colIndices].Set/Format<T>...;}

That is, to useSepWriter follow the points below:

1. Optionally defineSep or use default automatically inferred separator.
2. Specify writer with optional configuration ofSepWriterOptions.For all options seeSepWriterOptions.
3. Specify destination e.g. file, text (string viaStringWriter),TextWriter, etc. viaTo extension methods.
4. MISSING:SepWriter currently does not allow you to define the header upfront. Instead, header is defined by the order in which column names areaccessed/created when defining the row.
5. Define new rows withNewRow. ⚠ Be sure to dispose any new rows beforestarting the next! For convenience Sep provides an overload forNewRow thattakes aSepReader.Row and copies the columns from that row to the new row:

   usingvarreader=Sep.Reader().FromText(text);usingvarwriter=reader.Spec.Writer().ToText();foreach(varreadRowinreader){usingvarwriteRow=writer.NewRow(readRow);}

6. Create a column by selecting by name or index. Or multiple columns viaindices and names.Sep internally handles pooled allocation and reuse ofarrays for multiple columns.
7. UseSet to set the column value either as aReadOnlySpan<char>,stringor via an interpolated string. Or useFormat<T> whereT : IFormattableto formatT to the column value.
8. Row is written whenDispose is called on the row.

   Note this is to allow a row to be defined flexibly with both columnremoval, moves and renames in the future. This is not yet supported.

The following options are available:

/// <summary>/// Specifies the separator used./// </summary>publicSepSep{get;init;}/// <summary>/// Specifies the culture used for parsing./// May be `null` for default culture./// </summary>publicCultureInfo?CultureInfo{get;init;}/// <summary>/// Specifies whether to write a header row/// before data rows. Requires all columns/// to have a name. Otherwise, columns can be/// added by indexing alone./// </summary>publicboolWriteHeader{get;init;}=true;/// <summary>/// Disables checking if column count is the/// same for all rows./// </summary>/// <remarks>/// When true, the <see cref="ColNotSetOption"/>/// will define how columns that are not set/// are handled. For example, whether to skip/// or write an empty column if a column has/// not been set for a given row./// <para>/// If any columns are skipped, then columns of/// a row may, therefore, be out of sync with/// column names if <see cref="WriteHeader"/>/// is true./// </para>/// As such, any number of columns can be/// written as long as done sequentially./// </remarks>publicboolDisableColCountCheck{get;init;}=false;/// <summary>/// Specifies how to handle columns that are/// not set./// </summary>publicSepColNotSetOptionColNotSetOption{get;init;}=SepColNotSetOption.Throw;/// <summary>/// Specifies whether to escape column names/// and values when writing./// </summary>/// <remarks>/// When true, if a column contains a separator/// (e.g. `;`), carriage return (`\r`), line/// feed (`\n` or quote (`"`) then the column/// is prefixed and suffixed with quotes `"`/// and any quote in the column is escaped by/// adding an extra quote so it becomes `""`./// Note that escape applies to column names/// too, but only the written name./// </remarks>publicboolEscape{get;init;}=false;/// <summary>/// Forwarded to <see/// cref="System.Threading.Tasks.ValueTask.ConfigureAwait(bool)"/> or/// similar when async methods are called./// </summary>publicboolAsyncContinueOnCapturedContext{get;init;}=false;

Escaping is not enabled by default in Sep, but when it is it gives the sameresults as other popular CSV librares as shown below. Although, CsvHelperappears to be escaping spaces as well, which is not necessary.

Separator/delimiter is set to semi-colon; (default for Sep)

· (middle dot) is whitespace to make this visible

\r,\n are carriage return and line feed special characters to make these visible

¹ Sep withEscape = true inSepWriterOptions

Sep supports efficientValueTask based asynchronous reading and writing.

However, given bothSepReader.Row andSepWriter.Row areref structs, asthey point to internal state and should only be used one at a time,async/await usage is only supported on C# 13.0+ as this has support for"refand unsafe in iterators and async methods" as covered inWhat's new in C#13. Pleaseconsult details in that for limitations and constraints due to this.

Similarly,SepReader only implementsIAsyncEnumerable<SepReader.Row> (andIEnumerable<SepReader.Row>) for .NET 9.0+/C# 13.0+ since then the interfaceshave been annotated withallows ref struct forT.

Async support is provided on the existingSepReader andSepWriter typessimilar to howTextReader andTextWriter support both sync and async usage.This means you as a developer are responsible for calling async methods andusingawait when necessary. See below for a simple example and consult testson GitHub for more examples.

vartext="""           A;B;C;D;E;F           Sep;🚀;1;1.2;0.1;0.5           CSV;✅;2;2.2;0.2;1.5           """;// Empty line at end is for line endingusingvarreader=awaitSep.Reader().FromTextAsync(text);awaitusingvarwriter=reader.Spec.Writer().ToText();awaitforeach(varreadRowinreader){awaitusingvarwriteRow=writer.NewRow(readRow);}Assert.AreEqual(text,writer.ToString());

Note how forSepReader theFromTextAsync is suffixed withAsync toindicate async creation, this is due to the reader having to read the first rowof the source at creation to determine both separator and, if file has a header,column names of the header. TheFrom*Async call then has to beawaited.After that rows can be enumerated asynchronously simply by puttingawaitbeforeforeach. If one forgets to do that the rows will be enumeratedsynchronously.

ForSepWriter the usage is kind of reversed.To* methods have noAsyncvariants, since creation is synchronous. That is,StreamWriter is created by asimple constructor call. Nothing is written until a header or row is defined andDispose/DisposeAsync is called on the row.

For reader nothing needs to be asynchronously disposed, sousing does notrequireawait. However, forSepWriter dispose may have to write/flush datato underlyingTextWriter and hence it should be usingDisposeAsync, so youmust useawait using.

To support cancellation many methods have overloads that accept aCancellationToken like theFrom*Async methods for creating aSepReader orfor exampleNewRow forSepWriter. ConsultPublic APIReference for full set of available methods.

Additionally, bothSepReaderOptions andSepWriterOptions feature thebool AsyncContinueOnCapturedContext option that is forwarded to internalConfigureAwait calls, see theConfigureAwaitFAQ for details onthat.

Sep is designed to be minimal and fast. As such, it has some limitations andconstraints:

• Comments# are not directly supported. You can skip a row by:

  foreach(varrowinreader){// Skip row if starts with #if(!row.Span.StartsWith("#")){// ...}}

  This does not allow skipping lines before a header row starting with#though. InExample Catalogue a full example is givendetailing how to skip lines before header.

To investigate the performance of Sep it is compared to:

• CsvHelper -the most commonlyused CSV library with a staggering downloads on NuGet. Fullyfeatured and battle tested.
• Sylvan - is well-known and haspreviously been shown to bethe fastest CSV libraries forparsing(Sep changes that 😉).
• ReadLine/WriteLine - basic naive implementations that read line by lineand split on separator. While writing columns, separators and line endingsdirectly. Does not handle quotes or similar correctly.

All benchmarks are run from/to memory either with:

• StringReader orStreamReader + MemoryStream
• StringWriter orStreamWriter + MemoryStream

This to avoid confounding factors from reading from or writing to disk.

When usingStringReader/StringWriter eachchar counts as 2 bytes, whenmeasuring throughput e.g.MB/s. When usingStreamReader/StreamWritercontent is UTF-8 encoded and eachchar typically counts as 1 byte, as contentusually limited to 1 byte per char in UTF-8. Note that in .NET forTextReaderandTextWriter data is converted to/fromchar, but for reading suchconversion can often be just as fast asMemmove.

By default onlyStringReader/StringWriter results are shown, if a result isbased onStreamReader/StreamWriter it will be called out. Usually, resultsforStreamReader/StreamWriter are in line withStringReader/StringWriterbut with half the throughput due to 1 byte vs 2 bytes. For brevity they are notshown here.

For all benchmark results, Sep has been defined as theBaseline inBenchmarkDotNet. This meansRatio will be 1.00for Sep. For the othersRatio will then show how manytimes faster Sep isthan that. Or how manytimes more bytes are allocated inAlloc Ratio.

Disclaimer: Any comparison made is based on a number of preconditions andassumptions. Sep is a new library written from the ground up to use the latestand greatest features in .NET. CsvHelper has a long history and has to takeinto account backwards compatibility and still supporting older runtimes, somay not be able to easily utilize more recent features. Same goes for Sylvan.Additionally, Sep has a different feature set compared to the two. Performanceis a feature, but not the only feature. Keep that in mind when evaluatingresults.

The following runtime is used for benchmarking:

• NET 9.0.X

NOTE:Garbage CollectionDATASmode is disabled since this severely impacts (e.g.1.7xslower) performance for somebenchmarks due to the bursty accumulated allocations. That is,GarbageCollectionAdaptationMode is set to0.

The following platforms are used for benchmarking:

• AMD EPYC 7763 (Virtual) X64 Platform Information

  OS=Ubuntu 22.04.5 LTS (Jammy Jellyfish)AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores

• AMD Ryzen 7 PRO 7840U (Laptop on battery) X64 Platform Information

  OS=Windows 11 (10.0.22631.4460/23H2/2023Update/SunValley3)AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores

• AMD 5950X (Desktop) X64 Platform Information

  OS=Windows 10 (10.0.19044.2846/21H2/November2021Update)AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores

• Apple M1 (Virtual) ARM64 Platform Information

  OS=macOS Sonoma 14.7.1 (23H222) [Darwin 23.6.0]Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores

The following reader scenarios are benchmarked:

• NCsvPerf fromThe fastest CSVparser in.NET
• Floats as for example in machine learning.

Details for each can be found in the following. However, for each of these 3different scopes are benchmarked to better assertain the low-level performanceof each library and approach and what parts of the parsing consume the mosttime:

• Row - for this scope only the row is enumerated. That is, for Sep allthat is done is:

  foreach(varrowinreader){}

  this should capture parsing both row and columns but without accessing these.Note that some libraries (like Sylvan) will defer work for columns to whenthese are accessed.
• Cols - for this scope all rows and all columns are enumerated. Ifpossible columns are accessed as spans, if not as strings, which then mightmean a string has to be allocated. That is, for Sep this is:

  foreach(varrowinreader){for(vari=0;i<row.ColCount;i++){varspan=row[i].Span;}}

• XYZ - finally the full scope is performed which is specific to each ofthe scenarios.

Additionally, as Sep supports multi-threaded parsing viaParallelEnumeratebenchmarks results with_MT in the method name are multi-threaded. These showSep provides unparalleled performance compared to any other CSV parser.

The overhead of Sep async support is also benchmarked and can be seen with_Async in the method name. Note that this is the absolute best case for asyncgiven there is no real IO involved and hence no actual asynchronous work orcontinuations (thus noTask allocations) since benchmarks run from memoryonly. This is fine as the main purpose of the benchmark is to gauge the overheadof the async code path.

NCsvPerf fromThe fastest CSVparser in.NET is abenchmark which inJoel Verhagen ownwords was defined with:

My goal was to find the fastest low-level CSV parser. Essentially, all Iwanted was a library that gave me a string[] for each line where each field inthe line was an element in the array.

What is great about this work is it tests a whole of 35 different libraries andapproaches to this. Providing a great overview of those and their performance onthis specific scenario. Given Sylvan is the fastest of those it is used as theone to beat here, while CsvHelper is used to compare to the most commonly usedlibrary.

The source used for this benchmarkPackageAssetsBench.cs is aPackageAssets.csvwith NuGet package information in 25 columns with rows like:

75fcf875-017d-4579-bfd9-791d3e6767f0,2020-11-28T01:50:41.2449947+00:00,Akinzekeel.BlazorGrid,0.9.1-preview,2020-11-27T22:42:54.3100000+00:00,AvailableAssets,RuntimeAssemblies,,,net5.0,,,,,,lib/net5.0/BlazorGrid.dll,BlazorGrid.dll,.dll,lib,net5.0,.NETCoreApp,5.0.0.0,,,0.0.0.075fcf875-017d-4579-bfd9-791d3e6767f0,2020-11-28T01:50:41.2449947+00:00,Akinzekeel.BlazorGrid,0.9.1-preview,2020-11-27T22:42:54.3100000+00:00,AvailableAssets,CompileLibAssemblies,,,net5.0,,,,,,lib/net5.0/BlazorGrid.dll,BlazorGrid.dll,.dll,lib,net5.0,.NETCoreApp,5.0.0.0,,,0.0.0.075fcf875-017d-4579-bfd9-791d3e6767f0,2020-11-28T01:50:41.2449947+00:00,Akinzekeel.BlazorGrid,0.9.1-preview,2020-11-27T22:42:54.3100000+00:00,AvailableAssets,ResourceAssemblies,,,net5.0,,,,,,lib/net5.0/de/BlazorGrid.resources.dll,BlazorGrid.resources.dll,.dll,lib,net5.0,.NETCoreApp,5.0.0.0,,,0.0.0.075fcf875-017d-4579-bfd9-791d3e6767f0,2020-11-28T01:50:41.2449947+00:00,Akinzekeel.BlazorGrid,0.9.1-preview,2020-11-27T22:42:54.3100000+00:00,AvailableAssets,MSBuildFiles,,,any,,,,,,build/Microsoft.AspNetCore.StaticWebAssets.props,Microsoft.AspNetCore.StaticWebAssets.props,.props,build,any,Any,0.0.0.0,,,0.0.0.075fcf875-017d-4579-bfd9-791d3e6767f0,2020-11-28T01:50:41.2449947+00:00,Akinzekeel.BlazorGrid,0.9.1-preview,2020-11-27T22:42:54.3100000+00:00,AvailableAssets,MSBuildFiles,,,any,,,,,,build/Akinzekeel.BlazorGrid.props,Akinzekeel.BlazorGrid.props,.props,build,any,Any,0.0.0.0,,,0.0.0.0

ForScope = Asset the columns are parsed into aPackageAsset class, whichconsists of 25 properties of which 22 arestrings. Each asset is accumulatedinto aList<PackageAsset>. Each column is accessed as astring regardless.

This means this benchmark is dominated by turning columns intostrings for thedecently fast parsers. Hence, the fastest libraries in this test employ stringpooling. That is, basically a custom dictionary fromReadOnlySpan<char> tostring, which avoids allocating a newstring for repeated values. And as canbe seen in the csv-file there are a lot of repeated values. Both Sylvan andCsvHelper do this in the benchmark. So does Sep and as with Sep this is anoptional configuration that has to be explicitly enable. For Sep this means thereader is created with something like:

usingvarreader=Sep.Reader(o=>owith{HasHeader=false,CreateToString=SepToString.PoolPerCol(maximumStringLength:128),}).From(CreateReader());

What is unique for Sep is that it allows defining a pool per column e.g. viaSepToString.PoolPerCol(...). This is based on the factthat often each column has its own set of values or strings that may be repeatedwithout any overlap to other columns. This also allows one to define per columnspecific handling ofToString behavior. Whether to pool or not. Or even to usea statically defined pool.

Sep supports unescaping via an option, seeSepReaderOptionsandUnescaping. Therefore, Sep has two methods being tested. ThedefaultSep without unescaping andSep_Unescape where unescaping is enabled.Note that only if there are quotes will there be any unescaping, but to supportunescape one has to track extra state during parsing which means there is aslight cost no matter what, most notably for theCols scope. Sep is still thefastest of all (by far in many cases).

The results below show Sep isthe fastest .NET CSV Parser (for thisbenchmark on these platforms and machines 😀). While for pure parsing allocatingonly a fraction of the memory due to extensive use of pooling and theArrayPool<T>.

This is in many aspects due to Sep having extremely optimized string pooling andoptimized hashing ofReadOnlySpan<char>, and thus not really due the thecsv-parsing itself, since that is not a big part of the time consumed. At leastnot for a decently fast csv-parser.

WithParallelEnumerate (MT) Sep is>2x faster than Sylvan and up to 9xfaster than CsvHelper.

The package assets benchmark (ScopeAsset) has a very high base load in theform of the accumulated instances ofPackageAsset and since Sep is so fast theGC becomes a significant bottleneck for the benchmark, especially formulti-threaded parsing. Switching toSERVERGCcan, therefore, provide significant speedup as can be seen below.

WithParallelEnumerate and server GC Sep is>4x faster than Sylvan and up to18x faster than CsvHelper. Breaking 4 GB/s parsing speed on package assets on5950X.

NCsvPerf does not examine performance in the face of quotes in the csv. Thisis relevant since some libraries like Sylvan will revert to a slower (not SIMDvectorized) parsing code path if it encounters quotes. Sep was designed toalways use SIMD vectorization no matter what.

Since there are two extrachars to handle per column, it does have asignificant impact on performance, no matter what though. This is expected whenlooking at the numbers. For each row of 25 columns, there are 24 separators(here,) and one set of line endings (here\r\n). That's 26 characters.Adding quotes around each of the 25 columns will add 50 characters or almosttriple the total to 76.

Here again are benchmark results with server garbage collection, which providessignificant speedup over workstation garbage collection.

Similar to the benchmark related to quotes here spaces and quotes" areadded to relevant columns to benchmark impact of trimming and unescape on lowlevel column access. That is, basically" is prepended and appended to eachcolumn. This will test the assumed most common case and fast path part oftrimming and unescaping in Sep. Sep is about 10x faster than CsvHelper for this.Sylvan does not appear to have support automatic trimming and is, therefore, notincluded.