The SQL Sentry Console is now 64-bit capable

I wanted to take a departure from the language-based posts I’ve done in the past and relay some information regarding the next release of our product. Starting with the next point release of our software (currently slated to be 4.3) The SQL Sentry Console will be able to run in native 64-bit mode. The reason for this blog post is to describe some of the reasons why this is available now and has not been in the past, as well as what it means to the end-user.

SQL Sentry has two components, the SQL Sentry Console, and the SQL Sentry Server Service. Each one of these talks to a number of different systems and uses many supporting dlls/assemblies to provide support for external systems. The SQL Sentry Server Service has run natively in 64-bit mode for some time now. The console, however has been forced to run in 32-bit mode. The reason for this is that in Windows, any 64-bit process can only load 64-bit dlls. You can’t mix and match 64-bit and 32-bit code.

SQL Sentry was conceived when SQL Server 2000 was the dominant SQL Server version. As such we decided early on to support reading SQL Server Enterprise Manager registrations into the console using SQL-DMO. We also found we could (with some work) show Enterprise Manager property windows and dialogs using the SQL-NS library. Both of these libraries were COM libraries, and since SQL Server 2000 was written at a time when 32-bit was your only OS option, the COM libraries were 32-bit. Eventually Microsoft added support for 64-bit operating systems in SQL Server 2000, but the client tools, running in 32-bit mode anyway, had no need to get these updates.

Because the console linked directly to these dlls to support reading registrations and showing SQL Server 2000 dialogs, the console was forced to run in 32-bit mode, lest it not be able to load the 32-bit dlls. This wasn’t a major issue to do, as it’s easy to flag an executable to run in x86 mode (vs. the default automatic detection mode for managed applications) by just setting that as a compilation option. It worked and that’s how SQL Sentry has been able to interoperate with SQL Server 2000 until now.

In 4.2 we wanted to support reading SQL Server 2008 registrations. This proved to be quite an engineering task as we found that Microsoft changed many of the APIs used to read in this registration data. Because of this we decided to go back to the drawing board and reengineer the way we read in registration information. As stated previously, before we linked directly to the registration COM objects (for SQL Server 2000) and SMO assemblies (for 2005). For 4.2 we elected to use an intermediate layer. We abstracted all the functionality we needed into a set of interfaces that we could read/write to, and then created plugins, one for 2000, one for 2005, and one for 2008. That allowed us to independently manage all the plugins and their corresponding references and decouple their implementations from the console. It was an undertaking but it paid off, because we have a much better separation of code layers, and we also will be able to support the next version of SQL Server without any major work.

Around the time when I’d looked at the feasibility of making the console work in 64-bit mode I quickly realized that doing so would require such a decoupling, because I had to get the 32-bit dlls out of the console process. It just so happens that the work done for 4.2 did just that.

In 4.3 the default build will not have the 32-bit flag in the manifest, so it will launch using whatever architecture the processor/OS supports. The side effect of this is if you are using the SQL Sentry Console on a 64-bit system and running it in 64-bit mode, you will be unable to read in SQL Server 2000 registrations, because it will not load the 2000 plugin (if it did it wouldn’t work due to the 64-bit/32-bit process/dll interoperability issue). The workaround for this is that we’re also shipping an x86 SQL Sentry Console that is flagged to run in x86 mode. This console will only be installed on 64-bit operating systems. If you install SQL Sentry on a 64-bit OS starting with 4.3 you’ll see two executable shortcuts in the start menu:

SQL Sentry Console and
SQL Sentry Console (x86)

If you require interoperation with SQL Server Enterprise Manager registrations, you should run the x86 version, otherwise, run the regular SQL Sentry Console, which will run in 64-bit mode on a 64-bit OS.

Dictionary Replication

It is very common in programming to need to do a replication of data. In SQL Sentry when we pull objects from remote systems we need to figure out if they exist in our database. We index objects by specific keys, and then during the synchronization process use these keys and compare them to the remote objects keys to figure out what objects are new, what objects have been deleted, and what objects have changed. The details of everything we do are beyond the scope of this article, but I find myself doing enough of these dictionary replications that I decided to blog about it.

Take the following scenario:
You have two dictionaries,

Dictionary<K,VTarget> targetDictionary
and
Dictionary<K,VSource> sourceDictionary

You want to in one call update targetDictionary with all the contents of sourceDictionary, and be able to generate callbacks for new entries, deleted entries, and changed entries. The value types are different, as you plan to map VSource to VTarget (more on that later), but they share a common key data type (we could later expand that to be different too if it suited us). It seems a bit overwhelming but it’s a perfect example of where generics can make your life a lot easier. Traditionally I’d find myself writing three loops. In the first loop I update changed items and add new items that are in the source but not in the target. In the second l find the keys that are in the target but not in the source. These are deleted items. I cant remove them yet though because I’m inside an enumerator and that would generate an error, so I use a temp collection then do one more loop at the end to remove them from the target.

   1: // Initialize the collections

   2: Dictionary<int, string> peopleByIDTarget = new Dictionary<int, string>();

   3: Dictionary<int, string> peopleByIDSource = new Dictionary<int, string>();

   4:  

   5: peopleByIDTarget.Add(1, "Brooke");

   6: peopleByIDTarget.Add(2, "Tommy");

   7:  

   8: peopleByIDSource.Add(2, "Rick");

   9: peopleByIDSource.Add(3, "Dom");

  10:  

  11: // Loop through the source

  12: foreach (KeyValuePair<int, string> sourceKeyValuePair in peopleByIDSource)

  13: {

  14:     string existingName;

  15:     

  16:     if (peopleByIDTarget.TryGetValue(sourceKeyValuePair.Key, out existingName))

  17:     {

  18:         // ID Exists. Change Name!

  19:         peopleByIDTarget[sourceKeyValuePair.Key] = sourceKeyValuePair.Value;

  20:     }

  21:     else

  22:     {

  23:         // ID doesn't exist, so add with name.

  24:         peopleByIDTarget.Add(sourceKeyValuePair.Key, sourceKeyValuePair.Value);

  25:     }

  26: }

  27:  

  28: // Create a temp list to hold items we need to remove. You cant remove items

  29: // while enumerating or you get an error.

  30: List<int> keysToRemove = new List<int>();

  31:  

  32: // Loop through the target to see what items dont exist in the source

  33: foreach (KeyValuePair<int, string> targetKeyValuePair in peopleByIDTarget)

  34: {

  35:     // The target item doesnt exist in the source so we must remove it.

  36:     // Add it to the removal list.

  37:     if (!peopleByIDSource.ContainsKey(targetKeyValuePair.Key))

  38:     {

  39:         keysToRemove.Add(targetKeyValuePair.Key);

  40:     }

  41: }

  42:  

  43: // Remove the keys we marked for removal

  44: foreach (int key in keysToRemove)

  45: {

  46:     peopleByIDTarget.Remove(key);

  47: }

This works but it’s quite a bit of code, especially if this is happening with a lot of collections. I like to promote code reuse so the goal was to make this routine generic so that I could simply call:

peopleByIDTarget.Merge(peopleByIDSource);

Another thing the above example lacks is support for the callbacks I mentioned earlier. I’d like to know when an item is removed, added, or changed, and specify it in a easily defined way, like

peopleByIDTarget.Merge(peopleByIDSource, itemAddedCallback, itemChangedCallback, itemRemovedCallback)

and get the item that was removed, added, or changed.

It’s pretty straightforward to convert the above code into a generic method. The following is the most advanced version, supporting lots of options, callbacks, comparisons to see whether two values are the same (you may not always wish to fire the changed event if the instance of V didn’t have any properties that are different).

First we need to define some helper classes for the callbacks:

   1: /// <summary>

   2: /// Provides event arguments for items that are added to a dictionary.

   3: /// </summary>

   4: /// <typeparam name="K">The Key type</typeparam>

   5: /// <typeparam name="V">The Value type</typeparam>

   6: public class DictionaryItemAddedEventArgs<K, V> : EventArgs

   7: {

   8:     /// <summary>

   9:     /// The Key

  10:     /// </summary>

  11:     public K Key { get; private set; }

  12:  

  13:     /// <summary>

  14:     /// The new value

  15:     /// </summary>

  16:     public V NewValue { get; private set; }

  17:  

  18:     /// <summary>

  19:     /// Creates a new DictionaryItemAddedEventArgs

  20:     /// </summary>

  21:     /// <param name="key">The key</param>

  22:     /// <param name="newValue">The new value</param>

  23:     public DictionaryItemAddedEventArgs(K key, V newValue)

  24:     {

  25:         this.Key = key;

  26:         this.NewValue = newValue;

  27:     }

  28: }

  29:  

  30: /// <summary>

  31: /// Provides event arguments for items that are changed in a dictionary.

  32: /// </summary>

  33: /// <typeparam name="K">The Key type</typeparam>

  34: /// <typeparam name="V">The Value type</typeparam>

  35: public class DictionaryItemChangedEventArgs<K, V> : EventArgs

  36: {

  37:     /// <summary>

  38:     /// The Key

  39:     /// </summary>

  40:     public K Key{ get; private set; }

  41:  

  42:     /// <summary>

  43:     /// The previous value

  44:     /// </summary>

  45:     public V OldValue { get; private set; }

  46:  

  47:     /// <summary>

  48:     /// The new value

  49:     /// </summary>

  50:     public V NewValue { get; private set; }

  51:  

  52:     /// <summary>

  53:     /// Creates a new DictionaryItemChangedEventArgs

  54:     /// </summary>

  55:     /// <param name="key">The key</param>

  56:     /// <param name="oldValue">The previous value</param>

  57:     /// <param name="newValue">The new value</param>

  58:     public DictionaryItemChangedEventArgs(K key, V oldValue, V newValue)

  59:     {

  60:         this.Key = key;

  61:         this.OldValue = oldValue;

  62:         this.NewValue = newValue;

  63:     }

  64: }

  65:  

  66: /// <summary>

  67: /// Provides event arguments for items that are deleted from a dictionary.

  68: /// </summary>

  69: /// <typeparam name="K">The Key type</typeparam>

  70: /// <typeparam name="V">The Value type</typeparam>

  71: public class DictionaryItemDeletedEventArgs<K, V> : EventArgs

  72: {

  73:     /// <summary>

  74:     /// The Key

  75:     /// </summary>

  76:     public K Key { get; private set; }

  77:  

  78:     /// <summary>

  79:     /// The previous value

  80:     /// </summary>

  81:     public V OldValue { get; private set; }

  82:  

  83:     /// <summary>

  84:     /// Creates a new DictionaryItemDeletedEventArgs

  85:     /// </summary>

  86:     /// <param name="key">The key</param>

  87:     /// <param name="oldValue">The previous value</param>

  88:     public DictionaryItemDeletedEventArgs(K key, V oldValue)

  89:     {

  90:         this.Key = key;

  91:         this.OldValue = oldValue;

  92:     }

  93: }

Then we can get to the actual dictionary extensions class. The primary work starts on line 180 and I've included a couple other helper extensions I added for other uses:

   1: /// <summary>

   2: /// Provides extention methods to the dictionary class

   3: /// </summary>

   4: public static class DictionaryExtensions

   5: {

   6:     /// <summary>

   7:     /// Creates a new Dictionary with the key and value of the current dictionary reversed.

   8:     /// This method should be not used when duplicate values are expected because collisions will occur.

   9:     /// </summary>

  10:     /// <typeparam name="K">The Key type</typeparam>

  11:     /// <typeparam name="V">The Value type</typeparam>

  12:     /// <param name="dictionary">The dictionary to use</param>

  13:     /// <returns>A Dictionary with the keys and values of the current dictionary reversed</returns>

  14:     public static Dictionary<V, K> CreateDictionaryOfValueAndKey<K, V>(this Dictionary<K, V> dictionary)

  15:     {

  16:         Dictionary<V, K> result = new Dictionary<V,K>();

  17:         foreach (KeyValuePair<K, V> keyValuePair in dictionary)

  18:         {

  19:             result.Add(keyValuePair.Value, keyValuePair.Key);

  20:         }

  21:  

  22:         return result;

  23:     }

  24:  

  25:     /// <summary>

  26:     /// Creates a new MultiDictionary with the key and value of the current dictionary reversed.

  27:     /// This method should be used when duplicate values are expected.

  28:     /// </summary>

  29:     /// <typeparam name="K">The Key type</typeparam>

  30:     /// <typeparam name="V">The Value type</typeparam>

  31:     /// <param name="dictionary">The dictionary to use</param>

  32:     /// <returns>A MultiDictionary with the keys and values of the current dictionary reversed</returns>

  33:     public static MultiDictionary<V, K> CreateMultiDictionaryOfValueAndKey<K, V>(this Dictionary<K, V> dictionary)

  34:     {

  35:         MultiDictionary<V, K> result = new MultiDictionary<V, K>();

  36:         foreach (KeyValuePair<K, V> keyValuePair in dictionary)

  37:         {

  38:             result.Add(keyValuePair.Value, keyValuePair.Key);

  39:         }

  40:  

  41:         return result;

  42:     }

  43:  

  44:     /// <summary>

  45:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

  46:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

  47:     /// setting them to the value in sourceDictionary

  48:     /// </summary>

  49:     /// <typeparam name="K">The Key type</typeparam>

  50:     /// <typeparam name="V">The Value type of the source and target dictionaries</typeparam>        

  51:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

  52:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

  53:     public static void Merge<K, V>(

  54:         this Dictionary<K, V> targetDictionary,

  55:         Dictionary<K, V> sourceDictionary)

  56:     {

  57:         Merge(targetDictionary, sourceDictionary, null, null, null, null, null);

  58:     }

  59:  

  60:     /// <summary>

  61:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

  62:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

  63:     /// setting them to the value in sourceDictionary

  64:     /// </summary>

  65:     /// <typeparam name="K">The Key type</typeparam>

  66:     /// <typeparam name="VTarget">The Value type of the target dictionary</typeparam>

  67:     /// <typeparam name="VSource">The Value type of the source dictionary</typeparam>

  68:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

  69:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

  70:     public static void Merge<K, VTarget, VSource>(

  71:         this Dictionary<K, VTarget> targetDictionary,

  72:         Dictionary<K, VSource> sourceDictionary)

  73:     {

  74:         Merge(targetDictionary, sourceDictionary, null, null, null, null, null);

  75:     }

  76:  

  77:     /// <summary>

  78:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

  79:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

  80:     /// setting them to the value in sourceDictionary

  81:     /// </summary>

  82:     /// <typeparam name="K">The Key type</typeparam>

  83:     /// <typeparam name="V">The Value type of the source and target dictionaries</typeparam>

  84:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

  85:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

  86:     /// <param name="valueUpdater">The action to use to update values that share the same key</param>

  87:     public static void Merge<K, V>(

  88:         this Dictionary<K, V> targetDictionary,

  89:         Dictionary<K, V> sourceDictionary,

  90:         Func<V, V, bool> valueUpdater)

  91:     {

  92:         Func<V, V> valueMapper = x => x;

  93:         Merge(targetDictionary, sourceDictionary, valueMapper, valueUpdater, null, null, null);

  94:     }

  95:  

  96:     /// <summary>

  97:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

  98:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

  99:     /// setting them to the value in sourceDictionary

 100:     /// </summary>

 101:     /// <typeparam name="K">The Key type</typeparam>

 102:     /// <typeparam name="VTarget">The Value type of the target dictionary</typeparam>

 103:     /// <typeparam name="VSource">The Value type of the source dictionary</typeparam>

 104:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

 105:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

 106:     /// <param name="valueMapper">The transform to convert VSource to VTarget</param>

 107:     /// <param name="valueUpdater">The action to use to update values that share the same key</param>

 108:     public static void Merge<K, VTarget, VSource>(

 109:         this Dictionary<K, VTarget> targetDictionary,

 110:         Dictionary<K, VSource> sourceDictionary,

 111:         Func<VSource, VTarget> valueMapper,

 112:         Func<VTarget, VTarget, bool> valueUpdater)

 113:     {

 114:         Merge(targetDictionary, sourceDictionary, valueMapper, valueUpdater, null, null, null);

 115:     }

 116:  

 117:     /// <summary>

 118:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

 119:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

 120:     /// setting them to the value in sourceDictionary

 121:     /// </summary>

 122:     /// <typeparam name="K">The Key type</typeparam>

 123:     /// <typeparam name="V">The Value type of the source and target dictionaries</typeparam>

 124:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

 125:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

 126:     /// <param name="itemAddedEventHandler">The event to fire for items that were added</param>

 127:     /// <param name="itemChangedEventHandler">The event to fire for items that were changed</param>

 128:     /// <param name="itemDeletedEventHandler">The event to fire for items that were deleted</param>

 129:     public static void Merge<K, V>(

 130:         this Dictionary<K, V> targetDictionary,

 131:         Dictionary<K, V> sourceDictionary,

 132:         EventHandler<DictionaryItemAddedEventArgs<K, V>> itemAddedEventHandler,

 133:         EventHandler<DictionaryItemChangedEventArgs<K, V>> itemChangedEventHandler,

 134:         EventHandler<DictionaryItemDeletedEventArgs<K, V>> itemDeletedEventHandler)

 135:     {

 136:         Func<V, V> valueMapper = x => x;

 137:         Merge(targetDictionary, sourceDictionary, valueMapper, null, itemAddedEventHandler, itemChangedEventHandler, itemDeletedEventHandler);

 138:     }

 139:  

 140:     /// <summary>

 141:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

 142:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

 143:     /// setting them to the value in sourceDictionary

 144:     /// </summary>

 145:     /// <typeparam name="K">The Key type</typeparam>

 146:     /// <typeparam name="VTarget">The Value type of the target dictionary</typeparam>

 147:     /// <typeparam name="VSource">The Value type of the source dictionary</typeparam>

 148:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

 149:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

 150:     /// <param name="valueMapper">The transform to convert VSource to VTarget</param>

 151:     /// <param name="itemAddedEventHandler">The event to fire for items that were added</param>

 152:     /// <param name="itemChangedEventHandler">The event to fire for items that were changed</param>

 153:     /// <param name="itemDeletedEventHandler">The event to fire for items that were deleted</param>

 154:     public static void Merge<K, VTarget, VSource>(

 155:         this Dictionary<K, VTarget> targetDictionary,

 156:         Dictionary<K, VSource> sourceDictionary,

 157:         Func<VSource, VTarget> valueMapper,

 158:         EventHandler<DictionaryItemAddedEventArgs<K, VTarget>> itemAddedEventHandler,

 159:         EventHandler<DictionaryItemChangedEventArgs<K, VTarget>> itemChangedEventHandler,

 160:         EventHandler<DictionaryItemDeletedEventArgs<K, VTarget>> itemDeletedEventHandler)

 161:     {

 162:         Merge(targetDictionary, sourceDictionary, valueMapper, null, itemAddedEventHandler, itemChangedEventHandler, itemDeletedEventHandler);

 163:     }        

 164:  

 165:     /// <summary>

 166:     /// Merges the targetDictionary with the sourceDictionary, deleting items that arent in sourceDictionary, 

 167:     /// adding items that are in sourceDictionary but not in the targetDictionary, and updating items that are in both, 

 168:     /// setting them to the value in sourceDictionary

 169:     /// </summary>

 170:     /// <typeparam name="K">The Key type</typeparam>

 171:     /// <typeparam name="VTarget">The Value type of the target dictionary</typeparam>

 172:     /// <typeparam name="VSource">The Value type of the source dictionary</typeparam>

 173:     /// <param name="targetDictionary">The current dictionary to merge entries into</param>

 174:     /// <param name="sourceDictionary">The new dictionary with the most recent data</param>

 175:     /// <param name="valueMapper">The transform to convert VSource to VTarget</param>

 176:     /// <param name="valueUpdater">The action to use to update values that share the same key</param>

 177:     /// <param name="itemAddedEventHandler">The event to fire for items that were added</param>

 178:     /// <param name="itemChangedEventHandler">The event to fire for items that were changed</param>

 179:     /// <param name="itemDeletedEventHandler">The event to fire for items that were deleted</param>

 180:     public static void Merge<K, VTarget, VSource>(

 181:         this Dictionary<K, VTarget> targetDictionary,

 182:         Dictionary<K, VSource> sourceDictionary,

 183:         Func<VSource, VTarget> valueMapper,

 184:         Func<VTarget, VTarget, bool> valueUpdater,

 185:         EventHandler<DictionaryItemAddedEventArgs<K, VTarget>> itemAddedEventHandler,

 186:         EventHandler<DictionaryItemChangedEventArgs<K, VTarget>> itemChangedEventHandler,

 187:         EventHandler<DictionaryItemDeletedEventArgs<K, VTarget>> itemDeletedEventHandler)

 188:     {

 189:         foreach (var keyValuePair in sourceDictionary)

 190:         {

 191:             VTarget newValue = valueMapper(keyValuePair.Value);

 192:  

 193:             VTarget oldValue;

 194:             if (targetDictionary.TryGetValue(keyValuePair.Key, out oldValue))

 195:             {

 196:                 bool changed = true;

 197:                 if (valueUpdater == null)

 198:                 {

 199:                     targetDictionary[keyValuePair.Key] = newValue;

 200:                 }

 201:                 else

 202:                 {

 203:                     changed = valueUpdater(oldValue, newValue);

 204:                 }

 205:  

 206:                 if (itemChangedEventHandler != null && changed)

 207:                 {

 208:                     itemChangedEventHandler(targetDictionary, new DictionaryItemChangedEventArgs<K, VTarget>(keyValuePair.Key, oldValue, newValue));

 209:                 }

 210:             }

 211:             else

 212:             {

 213:                 targetDictionary.Add(keyValuePair.Key, newValue);

 214:                 if (itemAddedEventHandler != null)

 215:                 {

 216:                     itemAddedEventHandler(targetDictionary, new DictionaryItemAddedEventArgs<K, VTarget>(keyValuePair.Key, newValue));

 217:                 }

 218:             }

 219:         }

 220:  

 221:         List<KeyValuePair<K, VTarget>> itemsToDelete = new List<KeyValuePair<K, VTarget>>();

 222:         foreach (var keyValuePair in targetDictionary)

 223:         {

 224:             if (!sourceDictionary.ContainsKey(keyValuePair.Key))

 225:             {

 226:                 itemsToDelete.Add(keyValuePair);

 227:             }

 228:         }

 229:  

 230:         foreach (var keyValuePair in itemsToDelete)

 231:         {

 232:             targetDictionary.Remove(keyValuePair.Key);

 233:             if (itemDeletedEventHandler != null)

 234:             {

 235:                 itemDeletedEventHandler(targetDictionary, new DictionaryItemDeletedEventArgs<K, VTarget>(keyValuePair.Key, keyValuePair.Value));

 236:             }

 237:         }

 238:     }

 239: }

It’s essentially the original code, just made generic with the callbacks included. There are some additional Func<> delegates. valueMapper is used to transform the source value type to the target value type if they are different. Note there are overloads that do not require this and they just use a simple x=>x mapping. Also, there is a valueUpdater delegate as well. It is used to compare two values to see if they are really the same, which is useful if the value type is a class with properties and you want to see if those have changed prior to calling the changed callback.

Taking the initial example we can now do this:

   1: int countRemoved = 0;

   2: int countAdded = 0;

   3: int countChanged = 0;

   4:  

   5: peopleByIDTarget.Merge(

   6: peopleByIDSource,

   7: (dictionary, itemAddedArgs) =>

   8: {

   9:     countAdded++;

  10:     System.Diagnostics.Debug.WriteLine("Item " + itemAddedArgs.NewValue + " Added");

  11: },

  12: (dictionary, itemChangedArgs) =>

  13: {

  14:     countChanged++;

  15:     System.Diagnostics.Debug.WriteLine("Item " + itemChangedArgs.OldValue + " changed to " + itemChangedArgs.NewValue);

  16: },

  17: (dictionary, itemRemovedArgs) =>

  18: {

  19:     countRemoved++;

  20:     System.Diagnostics.Debug.WriteLine("Item " + itemRemovedArgs.OldValue + " Removed");

  21: });

  22:  

  23: System.Diagnostics.Debug.WriteLine(countAdded.ToString() + " Items Added");

  24: System.Diagnostics.Debug.WriteLine(countChanged.ToString() + " Items Changed");

  25: System.Diagnostics.Debug.WriteLine(countRemoved.ToString() + " Items Removed");

Generics combined with closures allows for some very rapid development.

Mapping Parents to Children Using Anonymous Types and LINQ

After writing my last post I spent some time thinking about certain places where the DeepEnumerator class helped me, and one of the uses that I came across was based on a particular challenge. I had a class that represented the tree structure that I talked about in my last post, namely it adhered to the following principle:

   1: class T

   2: {

   3:     IEnumerable<T> Children { get; }

   4: }

What I discovered is that it’s fairly common to have a situation where the parent defines it’s children in the aforementioned manner, but often in such cases, the children do not define their parents. What I wanted to do is to handle this case by creating a new class that holds the parent and the child and do it in a generic way.

For the purpose of illustrating anonymous types let’s do it first with anonymous types. Note: The following example uses LINQ for terseness and expressiveness, but it’s not required.

First, given the aforementioned class T, let’s define the anonymous type, given an instance of T called parent.

   1: // assume parent which is the top most

   2: // element in the tree is defined

   3: T parent = new T();

   4: var root = new {Parent=default(T), Element = parent};

This code may look odd but all we really are doing here is defining the concept of a root, and a root is an item that has no parent. We use default(T) to define the type. To enumerate, we know we want to enumerate starting with root, which internally represents a tuple of a parent and child, but the challenge is mapping root to IEnumerable<AnonymousType> where anonymous type is the type of root that we just defined.

To do so let’s use linq:

   1: elementAndParent =>

   2:     {

   3:         var query =

   4:             from child in elementAndParent.Element.Children

   5:             select new { Parent = elementAndParent.Element, Element = child };

   6:  

   7:         return query;

   8:     }

All this does is it takes the elementAndParent and maps it to a query that takes all the children under elementAndParent.Element(which is really the element that we are using currently on the tree) and returns a new anonymous type with that child as Element, and the current element as Parent.

Let’s look at it in the full context. Note I’ve defined our cute dog Kadi here as a person. Call it artistic license:

   1: public class Person

   2: {

   3:     private List<Person> _Children = new List<Person>();

   4:  

   5:     public int Age { get; set; }

   6:     public string FirstName { get; set; }

   7:     public string LastName { get; set; }

   8:  

   9:     public List<Person> Children

  10:     {

  11:         get { return _Children; }

  12:     }

  13: }

  14: private void TestParents()

  15: {

  16:     Person parent = new Person();

  17:     parent.FirstName = "Bob";

  18:     parent.LastName = "Philpott";

  19:  

  20:     Person child1 = new Person();

  21:     child1.FirstName = "Brooke";

  22:     child1.LastName = "Philpott";

  23:  

  24:     Person child2 = new Person();

  25:     child2.FirstName = "Stephen";

  26:     child2.LastName = "Philpott";

  27:  

  28:     Person pet1 = new Person();

  29:     pet1.FirstName = "Kadi";

  30:     pet1.LastName = "Philpott";

  31:  

  32:     parent.Children.Add(child1);

  33:     parent.Children.Add(child2);

  34:  

  35:     child1.Children.Add(pet1);

  36:  

  37:     var root = new { Parent = default(Person), Element = parent };

  38:     var allElementAndParents =

  39:             Intercerve.Collections.Generic.DeepEnumerator.Enumerate(

  40:             root,

  41:             elementAndParent =>

  42:             {

  43:                 var query =

  44:                     from child in elementAndParent.Element.Children

  45:                     select new { Parent = elementAndParent.Element, Element = child };

  46:  

  47:                 return query;

  48:             });

  49:  

  50:     foreach (var elementAndParent in allElementAndParents)

  51:     {

  52:         if (elementAndParent.Parent == null)

  53:         {

  54:             continue;

  55:         }

  56:  

  57:         string message = string.Format("{0} is the parent of {1}", elementAndParent.Parent.FirstName, elementAndParent.Element.FirstName);

  58:         System.Diagnostics.Debug.WriteLine(message);

  59:     }

  60: }

Note I filtered out entries that don’t have parents, as those are roots and I don’t really care about them.

The only drawback to this approach is that anonymous types have to be local in scope, which prevents me from reusing this code (obviously not ideal). To fix this let’s declare the anonymous type as a real type:

   1: class ElementAndParent<T>

   2: {

   3:     public T Element { get; set; }

   4:     public T Parent { get; set; }

   5: }

Once we have this helper class we can define a function like so:

   1: private static IEnumerable<ElementAndParent<T>> GetElementsWithParentsDeep<T>(

   2:     T parent, 

   3:     Func<T, IEnumerable<T>> childMappingFunction)

   4: {

   5:     var root = new ElementAndParent<T> { Parent = default(T), Element = parent };

   6:     var allElementAndParents =

   7:             Intercerve.Collections.Generic.DeepEnumerator.Enumerate(

   8:             root,

   9:             elementAndParent =>

  10:             {

  11:                 var query =

  12:                     from child in childMappingFunction(elementAndParent.Element)

  13:                     select new ElementAndParent<T> { Parent = elementAndParent.Element, Element = child };

  14:  

  15:                 return query;

  16:             })

  17:             .Where(item=>item.Parent != null);

  18:  

  19:     return allElementAndParents;

  20: }

What if we wanted to be able to walk all the way up the parents from the current node. Pretty easy actually. We just make a couple modifications. First let us define a class:

   1: private class ElementAndParentExtended<T>

   2:     {

   3:         public T Element { get; set; }

   4:         public ElementAndParentExtended<T> Parent { get; set; }

   5:     }

This class basically makes parent the same type as the type itself, allowing us to get the parent’s parent. We can then modify the enumeration like so:

   1: private static IEnumerable<ElementAndParent<T>> GetElementsWithParentsDeep<T>(

   2:     T parent, 

   3:     Func<T, IEnumerable<T>> childMappingFunction)

   4: {

   5:     var root = new ElementAndParent<T> { Parent = default(T), Element = parent };

   6:     var allElementAndParents =

   7:             Intercerve.Collections.Generic.DeepEnumerator.Enumerate(

   8:             root,

   9:             elementAndParent =>

  10:             {

  11:                 var query =

  12:                     from child in childMappingFunction(elementAndParent.Element)

  13:                     select new ElementAndParent<T> { Parent = elementAndParent.Element, Element = child };

  14:  

  15:                 return query;

  16:             })

  17:             .Where(item=>item.Parent != null);

  18:  

  19:     return allElementAndParents;

  20: }

To test, let’s make a new method:

   1: private static void TestParents2()

   2: {

   3:     Person parent = new Person();

   4:     parent.FirstName = "Bob";

   5:     parent.LastName = "Philpott";

   6:  

   7:     Person child1 = new Person();

   8:     child1.FirstName = "Brooke";

   9:     child1.LastName = "Philpott";

  10:  

  11:     Person child2 = new Person();

  12:     child2.FirstName = "Stephen";

  13:     child2.LastName = "Philpott";

  14:  

  15:     Person pet1 = new Person();

  16:     pet1.FirstName = "Kadi";

  17:     pet1.LastName = "Philpott";

  18:  

  19:     parent.Children.Add(child1);

  20:     parent.Children.Add(child2);

  21:  

  22:     child1.Children.Add(pet1);

  23:  

  24:     var parentsExtended = GetElementsWithParentsExtendedDeep(parent, item => item.Children);

  25:     foreach (var item in parentsExtended)

  26:     {

  27:         if (item.Parent == null)

  28:         {

  29:             continue;

  30:         }

  31:  

  32:         StringBuilder message = new StringBuilder(item.Element.FirstName);

  33:         var itemParent = item.Parent;

  34:         while (itemParent != null)

  35:         {

  36:             message.Append(" is the child of " + itemParent.Element.FirstName);

  37:             itemParent = itemParent.Parent;

  38:         }

  39:  

  40:         System.Diagnostics.Debug.WriteLine(message);

  41:     }

  42: }

If you run the method you’ll see that each element can now walk up it’s hierarchy to get it’s ancestry.

These methods could be extended to support all the options that the deep enumerator does, but I’ve not done so in the interest of brevity. Note I also could have used the Tuple class for this, but I chose to create my own for clarity.

Deep enumeration

Over the course of my programming career I've found myself working with tree data structures quite a bit. There are many examples of APIs that use trees, like tree controls, directories of a file system, etc. It is often desirable to iterate over the contents of the nodes in the tree and do a pattern match. Unfortunately there isn't a straightforward way to do this in c# without writing some custom code. I don't like writing a custom function each time I need to do something like this, and the great thing about generics is that allows sufficient abstraction of types and functions to allow generalizing a pattern like this.

Let us define a basic abstraction of a tree element

   1: IElement

   2: {

   3:     IEnumerable<IElement> Children { get; }

   4: }

In a nutshell that's all you need. A tree node is a structure where each element has children that can be iterated over, and the hierarchy of all the elements defines the tree structure. This is of course a generalization, as we could put additional constraints such as each element being only available once in the tree, but I'll leave it up to the developer to enforce such constraints. I'm merely trying to iterate over the tree nodes.

With that in mind, we can define a function that maps

   1: IElement -> IEnumerable<IElement>

   2:  

   3: Func<IElement, IEnumerable<IElement>> childrenMappingFunction;

There are two common ways to do recursion: function recursion where a function points to itself, or stack-based recursion where we use a stack to keep a list of evaluation arguments on a stack and evaluate them as need be. We'll use stack-based iteration because it tends to be more scalable.

Without further ado let’s get to the meat of the code:

   1: public static IEnumerable<T> Enumerate<T>(

   2:     T rootItem, 

   3:     Func<T, IEnumerable<T>> childrenMappingFunction, 

   4:     bool returnRoot)

   5: {

   6:     Stack<T> workStack = new Stack<T>();

   7:     workStack.Push(rootItem);

   8:  

   9:     bool isRoot = true;

  10:  

  11:     while (workStack.Count > 0)

  12:     {

  13:         T item = workStack.Pop();

  14:  

  15:         if (isRoot)

  16:         {

  17:             if (returnRoot)

  18:             {

  19:                 yield return item;

  20:             }

  21:  

  22:             isRoot = false;

  23:         }

  24:         else

  25:         {

  26:             yield return item;

  27:         }

  28:  

  29:         IEnumerable<T> children = childrenMappingFunction(item);

  30:  

  31:         foreach (T child in children)

  32:         {

  33:             workStack.Push(child);

  34:         }

  35:     }

  36: }

Let’s look at what the code does. It first creates a stack to hold the objects that need to be evaluated. It then pushes the root item onto the stack. The loop is quite simple. While the stack has items in it, first pop an item off the stack, yield it using the c# iterator pattern, then call childrenMappingFunction to get the children (IEnumerable<T>) from T item. Then, for each child in children, add the child to the stack, so that it is evaluated on the next iteration of the while loop. That’s pretty much it. Because of the elegance of c# iterators, it’s easy to use this function to map a tree to a flatter IEnumerable<T> without ever having to concern ourselves with the details again.

I’ve added some additional overloads and parameters to allow customizing how we get the data back. The only one that might need a bit of explanation is leftToRight. What I discovered when first using this class is that because of the way items are added to the stack, the elements returned from the function are not necessarily in the same order that they were in each nodes IEnumerable. This is not always a problem, but certainly can be in certain use cases so I added a modifier, leftToRight, that ensures that the data is added to the stack in reverse order, so that when it is popped off the stack, it’s in the order that the elements were returned from IEnumerable. It’s not the most efficient way to do this and I recognize that, but I don’t use that option most of the time and in the cases where I did use it performance was not a major concern. If anyone wants to provide a cleaner implementation I welcome feedback. Here is the class in it’s entirety:

   1: public static class DeepEnumerator

   2: {

   3:     public static IEnumerable<T> Enumerate<T>(

   4:         IEnumerable<T> rootItems, 

   5:         Func<T, IEnumerable<T>> childrenMappingFunction)

   6:     {

   7:         return Enumerate(rootItems, childrenMappingFunction, true, true);

   8:     }

   9:  

  10:     public static IEnumerable<T> Enumerate<T>(

  11:         IEnumerable<T> rootItems, 

  12:         Func<T, IEnumerable<T>> childrenMappingFunction, 

  13:         bool returnRootItems, 

  14:         bool enumerateDeep)

  15:     {

  16:         foreach (T rootItem in rootItems)

  17:         {

  18:             foreach (T item in Enumerate(rootItem, childrenMappingFunction, returnRootItems, enumerateDeep, true))

  19:             {

  20:                 yield return item;

  21:             }

  22:         }

  23:     }

  24:  

  25:     public static IEnumerable<T> Enumerate<T>(

  26:         T rootItem, 

  27:         Func<T, IEnumerable<T>> childrenMappingFunction)

  28:     {

  29:         return Enumerate(rootItem, childrenMappingFunction, true, true, true);

  30:     }

  31:  

  32:     public static IEnumerable<T> Enumerate<T>(

  33:         T rootItem, 

  34:         Func<T, IEnumerable<T>> childrenMappingFunction, 

  35:         bool returnRoot, 

  36:         bool enumerateDeep, 

  37:         bool leftToRight)

  38:     {

  39:         if (enumerateDeep)

  40:         {

  41:             Stack<T> workStack = new Stack<T>();

  42:             workStack.Push(rootItem);

  43:  

  44:             bool isRoot = true;

  45:  

  46:             while (workStack.Count > 0)

  47:             {

  48:                 T item = workStack.Pop();

  49:  

  50:                 if (isRoot)

  51:                 {

  52:                     if (returnRoot)

  53:                     {

  54:                         yield return item;

  55:                     }

  56:  

  57:                     isRoot = false;

  58:                 }

  59:                 else

  60:                 {

  61:                     yield return item;

  62:                 }

  63:  

  64:                 IEnumerable<T> children = childrenMappingFunction(item);

  65:  

  66:                 if (leftToRight)

  67:                 {

  68:                     children = EnumerateListBackwards(children.ToList());

  69:                 }

  70:  

  71:                 foreach (T child in children)

  72:                 {

  73:                     workStack.Push(child);

  74:                 }

  75:             }

  76:         }

  77:         else

  78:         {

  79:             yield return rootItem;

  80:             foreach (T item in childrenMappingFunction(rootItem))

  81:             {

  82:                 yield return item;

  83:             }

  84:         }

  85:     }

  86:  

  87:     private static IEnumerable<T> EnumerateListBackwards<T>(List<T> list)

  88:     {

  89:         for (int index = list.Count - 1; index >= 0; index--)

  90:         {

  91:             yield return list[index];

  92:         }

  93:     }

  94: }

Let’s look at a use case:

Given a .Net Winforms TreeView, find all nodes in the entire tree that start with E.

Because Node.Nodes does not implement IEnumerable<Node> we need a helper function, so we’ll use that with the deep enumerator.

   1: private static Enumerable<TreeNode> IterateNodes(TreeNodeCollection treeNodes)

   2: {

   3:     foreach (TreeNode childNode in treeNodes)

   4:     {

   5:         yield return childNode;

   6:     }

   7: }

   8:  

   9: TreeView treeView = new TreeView();

  10: foreach (TreeNode treeNode in 

  11:    DeepEnumerator.Enumerate(

  12:        IterateNodes(treeView.Nodes), 

  13:        node => IterateNodes(node.Nodes)))

  14: {

  15:    if (treeNode.Text.StartsWith("E"))

  16:    {

  17:         Do Something.

  18:    }

  19: }

For a second example let us define a slightly cleaner interface:

   1: private interface ITextNode

   2: {

   3:     string Text { get; }

   4:     IEnumerable<ITextNode> Children { get; }

   5: }

What if we wanted to find all nodes in List<ITextNode> nodes where one or more of the children (root included) start with the letter E.

Using LINQ this is pretty straightforward:

   1: List<ITextNode> textNodes = new List<ITextNode>();

   2: var nodesWithOneOrMoreChildrenStartingWithLetterE =

   3:     from textNode in textNodes

   4:     let children = DeepEnumerator.Enumerate(textNode, node => node.Children)

   5:     let countWithLetterE = children.Where(node => node.Text.StartsWith("E")).Count()

   6:     where countWithLetterE > 0

   7:     select textNode;

These are just a few examples. I hope you find this class useful. Deep tree enumeration is something I find myself doing fairly often, and this class has eliminated a lot of redundant code for me.

About the Title

The never-ending staircase is one of the elements of an illustration by M. C. Escher called "Ascending and Descending". The reason I chose such a title for my blog is because to me programming has always seemed like an infinite series of steps. There seems to be no limit to what we can learn or how far we can push ourselves given enough time and effort.

The purpose of this blog is to to share with the community some of those steps of mine. In addition, occasionally (at least in my experience) there seem to be moments of extreme clarity, where paradigm shifts occur and many steps are taken in a short amount of time. At times in retrospect I wish I had documented such moments before and I hope this blog can become a channel for some of those ideas. While I expect the majority of my posts will cover code and language, some may cover architecture as well.

I hope everyone finds my posts to be useful and I strongly encourage feedback from anyone who happens to stumble upon this blog.