Repository files navigation

Dynamic Data is a portable class library which brings the power of Reactive Extensions (Rx) to collections.

Rx is extremely powerful but out of the box provides nothing to assist with managing collections. In most applications there is a need to update the collections dynamically. Typically a collection is loaded and after the initial load, asynchronous updates are received. The original collection will need to reflect these changes. In simple scenarios the code is simple. However, typical applications are much more complicated and may apply a filter, transform the original dto and apply a sort. Even with these simple every day operations the complexity of the code is quickly magnified. Dynamic data has been developed to remove the tedious code of dynamically maintaining collections. It has grown to become functionally very rich with at least 60 collection based operations which amongst other things enable filtering, sorting, grouping, joining different sources, transforms, binding, pagination, data virtualisation, expiration, disposal management plus more.

The concept behind using dynamic data is you maintain a data source (eitherSourceCache<TObject, TKey> orSourceList<TObject>), then chain together various combinations of operators to declaratively manipulate and shape the data without the need to directly manage any collection.

As an example the following code will filter trades to select only live trades, creates a proxy for each live trade, and finally orders the results by most recent first. The resulting trade proxies are bound on the dispatcher thread to an observable collection. Also since the proxy is disposableDisposeMany() will ensure the proxy is disposed when no longer used.

ReadOnlyObservableCollection<TradeProxy>list;varmyTradeCache=newSourceCache<Trade,long>(trade=>trade.Id);varmyOperation=myTradeCache.Connect().Filter(trade=>trade.Status==TradeStatus.Live).Transform(trade=>newTradeProxy(trade)).Sort(SortExpressionComparer<TradeProxy>.Descending(t=>t.Timestamp)).ObserveOnDispatcher().Bind(outlist).DisposeMany().Subscribe()

The magic is that asmyTradeCache is maintained the target observable collection looks after itself.

This is a simple example to show how using Dynamic Data's collections and operators make in-memory data management extremely easy and can reduce the size and complexity of your code base by abstracting complicated and often repetitive operations.

• Sample WPF project trading projectDynamic Trader
• Various unit tested examples of many different operatorsSnippets
• Tail Blazer for tailing files

If you have any questions, want to get involved or would simply like to keep abreast of developments, you are welcome to join the slack communityReactive UI Slack.

• Dynamic Data
  • Sample Projects
  • Get in touch
• Table of Contents
• Create Dynamic Data Collections
  • The Observable List
  • The Observable Cache
• Creating Observable Change Sets
  • Connect to a Cache or List
  • Create an Observable Change Set from an Rx Observable
  • Create an Observable Change Set from an Rx Observable with an Expiring Cache
  • Create an Observable Change Set from an Observable Collection
  • Create an Observable Change Set from an Binding List
  • Using the ObservableChangeSet static class
• Consuming Observable Change Sets
• Observable list vs observable cache
• History of Dynamic Data
• Want to know more?

Create an observable list like this:

varmyInts=newSourceList<int>();

The observable list provides the direct edit methods you would expect. For example:

myInts.AddRange(Enumerable.Range(0,10000));myInts.Add(99999);myInts.Remove(99999);

TheAddRange,Add andRemove methods above will each produce a distinct change notification. In order to increase efficiency when making multiple amendments, the list provides a means of batch editing. This is achieved using the.Edit method which ensures only a single change notification is produced.

myInts.Edit(innerList=>{innerList.Clear();innerList.AddRange(Enumerable.Range(0,10000));});

IfmyInts is to be exposed publicly it can be made read only using.AsObservableList

IObservableList<int>readonlyInts=myInts.AsObservableList();

which hides the edit methods.

The list's changes can be observed by callingmyInts.Connect() like this:

IObservable<IChangeSet<int>>myIntsObservable=myInts.Connect();

This creates an observable change set for which there are dozens of operators. The changes are transmitted as an Rx observable, so they are fluent and composable.

Create an observable cache like this:

varmyCache=newSourceCache<TObject,TKey>(t=>key);

There are direct edit methods, for example

myCache.Clear();myCache.AddOrUpdate(myItems);

TheClear andAddOrUpdate methods above will each produce a distinct change notification. In order to increase efficiency when making multiple amendments, the cache provides a means of batch editing. This is achieved using the.Edit method which ensures only a single change notification is produced.

myCache.Edit(innerCache=>{innerCache.Clear();innerCache.AddOrUpdate(myItems);});

IfmyCache is to be exposed publicly it can be made read only using.AsObservableCache

IObservableCache<TObject,TKey>readonlyCache=myCache.AsObservableCache();

which hides the edit methods.

The cache is observed by callingmyCache.Connect() like this:

IObservable<IChangeSet<TObject,TKey>>myCacheObservable=myCache.Connect();

This creates an observable change set for which there are dozens of operators. The changes are transmitted as an Rx observable, so they are fluent and composable.

As stated in the introduction of this document, Dynamic Data is based on the concept of creating and manipulating observable change sets.

The primary method of creating observable change sets is to connect to instances ofISourceCache<T,K> andISourceList<T>. There are alternative methods to produce observables change sets however, depending on the data source.

CallingConnect() on aISourceList<T> orISourceCache<T,K> will produce an observable change set.

varmyObservableChangeSet=myDynamicDataSource.Connect();

Given either of the following observables:

IObservable<T>myObservable;IObservable<IEnumerable<T>>myObservable;

an observable change set can be created like by calling.ToObservableChangeSet like this:

varmyObservableChangeSet=myObservable.ToObservableChangeSet(t=>t.key);

The problem with the example above is that the internal backing cache of the observable change set will grow in size forever.To counter this behavior, there are overloads of.ToObservableChangeSet where a size limitation or expiry time can be specified for the internal cache.

To create a time expiring cache, call.ToObservableChangeSet and specify the expiry time using the expireAfter argument:

varmyConnection=myObservable.ToObservableChangeSet(t=>t.key,expireAfter: item=>TimeSpan.FromHours(1));

To create a size limited cache, call.ToObservableChangeSet and specify the size limit using the limitSizeTo argument:

varmyConnection=myObservable.ToObservableChangeSet(t=>t.key,limitSizeTo:10000);

There is also an overload to specify expiration by both time and size.

varmyObservableCollection=newObservableCollection<T>();

To create a cache based observable change set, call.ToObservableChangeSet and specify a key selector for the backing cache

varmyConnection=myObservableCollection.ToObservableChangeSet(t=>t.Key);

or to create a list based observable change set call.ToObservableChangeSet with no arguments

varmyConnection=myObservableCollection.ToObservableChangeSet();

This method is only recommended for simple queries which act only on the UI thread asObservableCollection is not thread safe.

varmyBindingList=newBindingList<T>();

To create a cache based observable change set, call.ToObservableChangeSet and specify a key selector for the backing cache

varmyConnection=myBindingList.ToObservableChangeSet(t=>t.Key);

or to create a list based observable change set call.ToObservableChangeSet with no arguments

varmyConnection=myBindingList.ToObservableChangeSet();

This method is only recommended for simple queries which act only on the UI thread asObservableCollection is not thread safe.

There is also another way to create observable change sets, and that is to use theObservableChangeSet static class. This class is a facsimile of the Rx.NetObservable static class and provides an almost identical API.

An observable list can be created as follows:

varmyObservableList=ObservableChangeSet.Create<int>(observableList=>{//some code to load data and subscribevarloader=myService.LoadMyDataObservable().Subscribe(observableList.Add);varsubscriber=myService.GetMySubscriptionsObservable().Subscribe(observableList.Add);//dispose of resourcesreturnnewCompositeDisposable(loader,subscriber);});

and creating a cache is almost identical except a key has to be specified

varmyObservableCache=ObservableChangeSet.Create<Trade,int>(observableCache=>{//code omitted},trade=>trade.Id);

There are several overloadsObservableChangeSet.Create which match the overloads whichObservable.Create provides.

The examples below illustrate the kind of things you can achieve after creating an observable change set.Now you can create an observable cache or an observable list, here are a few quick fire examples to illustrate the diverse range of things you can do. In all of these examples the resulting sequences always exactly reflect the items is the cache i.e. adds, updates and removes are always propagated.

This example shows how you can create derived collections from an observable change set. It applies a filter to a collection, and then creates a new observable collection that only contains items from the original collection that pass the filter.This pattern is incredibly useful when you want to make modifications to an existing collection and then expose the modified collection to consumers.

Even though the code in this example is very simple, this is one of the most powerful aspects of Dynamic Data.

Given a SourceList

varmyList=newSourceList<People>();

You can apply operators, in this case theFilter() operator, and then create a new observable list withAsObservableList()

varoldPeople=myList.Connect().Filter(person=>person.Age>65).AsObservableList();

The resulting observable list, oldPeople, will only contain people who are older than 65.

The same pattern can be used with SourceCache by using.AsObservableCache() to create derived caches.

As an alternative to.Bind(out collection) you can use.BindToObservableList(out observableList) for bothSourceList &SourceCache. This is useful for getting derived read-only lists from sources that use.AutoRefresh(), since collections do not support refresh notifications.

Filter the observable change set by using theFilter operator

varmyPeople=newSourceList<People>();varmyPeopleObservable=myPeople.Connect();varmyFilteredObservable=myPeopleObservable.Filter(person=>person.Age>50);

or to filter a change set dynamically

IObservable<Func<Person,bool>>observablePredicate=...;varmyFilteredObservable=myPeopleObservable.Filter(observablePredicate);

Sort the observable change set by using theSort operator

varmyPeople=newSourceList<People>();varmyPeopleObservable=myPeople.Connect();varmySortedObservable=myPeopleObservable.Sort(SortExpressionComparer.Ascending(p=>p.Age));

or to dynamically change sorting

IObservable<IComparer<Person>>observableComparer=...;varmySortedObservable=myPeopleObservable.Sort(observableComparer);

For more information on sorting seewiki

TheGroupOn operator pre-caches the specified groups according to the group selector.

varmyOperation=personChangeSet.GroupOn(person=>person.Status)

The value of the inner group is represented by an observable list for each matched group. When values matching the inner grouping are modified, it is the inner group which produces the changes.You can also useGroupWithImmutableState which will produce a grouping who's inner items are a fixed size array.

TheTransform operator allows you to map objects from the observable change set to another object

varmyPeople=newSourceList<People>();varmyPeopleObservable=myPeople.Connect();varmyTransformedObservable=myPeopleObservable.Transform(person=>newPersonProxy(person));

TheTransformToTree operator allows you to create a fully formed reactive tree (only available for observable cache)

varmyPeople=newSourceCache<Person,string>(p=>p.Name);varmyTransformedObservable=myPeople.Connect().TransformToTree(person=>person.BossId);

Flatten a child enumerable

varmyOperation=personChangeSet.TransformMany(person=>person.Children)

TheCount,Max,Min,Avg, andStdDev operators allow you to perform aggregate functions on observable change sets

varmyPeople=newSourceList<People>();varmyPeopleObservable=myPeople.Connect();varcountObservable=myPeopleObservable.Count();varmaxObservable=myPeopleObservable.Max(p=>p.Age);varminObservable=myPeopleObservable.Min(p=>p.Age);varstdDevObservable=myPeopleObservable.StdDev(p=>p.Age);varavgObservable=myPeopleObservable.Avg(p=>p.Age);

More aggregating operators will be added soon.

TheAnd,Or,Xor andExcept operators allow you to perform logical operations on observable change sets

varpeopleA=newSourceCache<Person,string>(p=>p.Name);varpeopleB=newSourceCache<Person,string>(p=>p.Name);varobservableA=peopleA.Connect();varobservableB=peopleB.Connect();varinBoth=observableA.And(observableB);varinEither=observableA.Or(observableB);varinOnlyOne=observableA.Xor(observableB);varinAandNotinB=observableA.Except(observableB);

A recent and very powerful feature is dynamic logical operators. From version 4.6 onwards you can dynamically include and exclude collections from the resulting list.

varlist1=newSourceList<int>();varlist2=newSourceList<int>();varlist3=newSourceList<int>();varcombined=newSourceList<ISourceList<int>>();//child lists can be added or removed any timecombined.Add(list1);combined.Add(list2);combined.Add(list3);//The operators look after themselvesvarinAll=combined.And();varinAny=combined.Or();varinOnlyOne=combined.Xor();varinFirstAndNotAnyOther=combined.Except();

For more information on grouping seewiki

TheDisposeMany operator ensures that objects are disposed when removed from an observable stream

varmyPeople=newSourceList<People>();varmyPeopleObservable=myPeople.Connect();varmyTransformedObservable=myPeopleObservable.Transform(person=>newDisposablePersonProxy(person)).DisposeMany();

TheDisposeMany operator is typically used when a transform function creates disposable objects.

TheDistinctValues operator will select distinct values from the underlying collection

varmyPeople=newSourceList<People>();varmyPeopleObservable=myPeople.Connect();varmyDistinctObservable=myPeopleObservable.DistinctValues(person=>person.Age);

Virtualise data to restrict by index and segment size

IObservable<IVirtualRequest>request;//request streamvarvirtualisedStream=someDynamicDataSource.Virtualise(request)

Virtualise data to restrict by index and page size

IObservable<IPageRequest>request;//request streamvarpagedStream=someDynamicDataSource.Page(request)

In either of the above, the result is re-evaluated when the request stream changes

Top is an overload ofVirtualise() and will return items matching the first 'n' items.

vartopStream=someDynamicDataSource.Top(10)

If the collection is made up of objects that implementINotifyPropertyChanged then the following operators are available

TheWhenValueChanged operator returns an observable of the value of the specified property when it has changed

varageChanged=peopleDataSource.Connect().WhenValueChanged(p=>p.Age)

TheWhenPropertyChanged operator returns an observable made up of the value of the specified property as well as it's parent object when the specified property has changed

varageChanged=peopleDataSource.Connect().WhenPropertyChanged(p=>p.Age)

TheWhenAnyPropertyChanged operator returns an observable of objects when any of their properties have changed

varpersonChanged=peopleDataSource.Connect().WhenAnyPropertyChanged()

Binding is a very small part of Dynamic Data. The above notify property changed overloads are just an example when binding. If you have a domain object which has children observables you can useMergeMany() which subscribes to and unsubscribes from items according to collection changes.

varmyoperation=somedynamicdatasource.Connect().MergeMany(trade=>trade.SomeObservable());

This wires and unwiresSomeObservable as the collection changes.

I get asked about the differences between these a lot and the answer is really simple. If you have a unique id, you should use an observable cache as it is dictionary based which will ensure no duplicates can be added and it notifies on adds, updates and removes, whereas list allows duplicates and only has no concept of an update.

There is another difference. The cache side of dynamic data is much more mature and has a wider range of operators. Having more operators is mainly because I found it easier to achieve good all round performance with the key based operators and do not want to add anything to Dynamic Data which inherently has poor performance.

Even before Rx existed I had implemented a similar concept using old fashioned events but the code was very ugly and my implementation full of race conditions so it never existed outside of my own private sphere. My second attempt was a similar implementation to the first but using Rx when it first came out. This also failed as my understanding of Rx was flawed and limited and my design forced consumers to implement interfaces. Then finally I got my design head on and in 2011-ish I started writing what has become dynamic data. No inheritance, no interfaces, just the ability to plug in and use it as you please. All along I meant to open source it but having so utterly failed on my first 2 attempts I decided to wait until the exact design had settled down. The wait lasted longer than I expected and ended up taking over 2 years but the benefit is it has been trialled for 2 years on a very busy high volume low latency trading system which has seriously complicated data management. And what's more that system has gathered a load of attention for how slick and cool and reliable it is both from the user and IT point of view. So I present this library with the confidence of it being tried, tested, optimised and mature. I hope it can make your life easier like it has done for me.

I could go on endlessly but this is not the place for full documentation. I promise this will come but for now I suggest downloading my WPF sample app (links at top of document) as I intend it to be a 'living document' and I promise it will be continually maintained.

Also, if you follow me on Twitter you will find out when new samples or blog posts have been updated.

Additionally, if you have read up to here and not pressed star then why not? Ha. A star may make me be more responsive to any requests or queries.