Kimball Method Slowly Changing Dimension Component v1.5 Released!

nd it's about ^%#@ time!  I simply can't believe it's been over eight months - all for a few new features.  Well, okay, maybe not just a few...
What Is The Kimball Method Slowly Changing Dimension Component?
The Kimball Method Slowly Changing Dimension component is a superior alternative to the Slowly Changing Dimension Wizard in SQL Server Integration Services.  Both components implement a very common data warehousing process.  If you have a data warehouse ETL system, chances are you're using the SCD Wizard...  Or you got frustrated and resorted to a more performant but less maintainable Script, tens of data flow transforms, or a T-SQL MERGE.
I - like many others - was a little disappointed by the poor performance and lack of flexibility in the included SCD wizard in SSIS.  Not that the wizard is horrible - it encapsulates the SCD process very nicely - but it's just "not good enough" for me.  I have to count myself lucky, I know a little .Net and happen to be a glutton for up-front punishment - so I investigated the extensibility of Integration Services and ended up creating a custom component that made some different architectural design choices than the SCD wizard.  This different design has many advantages, requiring very few compromises - in my opinion, of course.
All you need to do to use it in place of the SCD wizard is to download and install it, add it to your BIDS toolbox, then drop it on your data flow design surface.  If you're familiar with processing slowly changing dimensions for a Kimball method data warehouse, I think you'll be using it in seconds.
What Does The Kimball Method SCD Component Do?
Basically, the same thing as the SCD wizard - just differently, and with quite a few more configurable options.  The component "processes" slowly changing dimensions.  "Processing" means that it compares the current information from your line-of-business system information with a dimension table in your data warehouse.  (This line-of-business system is also known as your OLTP or "source" system.)  It detects changes that have occurred in the OLTP system, identifies new business entities that need to be inserted into the dimension table, retroactive changes that need to be made to the dimension table, or changes that require historical "versioning" of those business entities.

How Is The Kimball Method SCD "Better" Than The SCD Wizard?
There are quite a few differences that I think make the KSCD superior to the SCD wizard, but the "big two" reasons are that you can actually edit the KSCD without destroying your data flow logic, and it operates about 100x faster.  There are many more things differentiating them, but for now I'll explore just those two.  And really - aren't those two things alone worth spending a few minutes trying this out?
Non-Destructive Edits
First, I'll talk about the "issue" with the built-in SCD wizard.  The SCD wizard steps you through "configuring" your SCD processing logic quite nicely.  It does use some "foreign" terminology for someone that's read up on the Kimball methodology for describing SCD processing - but nothing too odd.  But once you get to the end of the wizard, it doesn't place just one component in the data flow, it places multiple components!  There's the SCD component itself, plus some derived columns, union alls, and OLE DB commands.  Some of the choices you made in the wizard are encoded in the SCD component itself, but quite a few are represented in those other components and the way they're connected up to each other. 
You can modify some of those properties directly as you need to (as many experts recommend you do) in order to have the process operate a little more efficiently.  The drawback of doing this is that you can't change everything you might want to that way.  Should you need to change a column's method of "change tracking" from "historical" (type 2) to "changing" (type 1), or a new column is introduced to your dimension table... you have to run the SCD wizard all over again.  Unfortunately, this results in the destruction and recreation of all the components that made up the processing logic - even if you've made tweaks to them.  All that optimizing and customizing... destroyed in seconds!
Good news for you - the KSCD is completely editable, and doesn't espouse a scorched-earth policy on the design surface.  Edit the component all you want - the worst that will happen is that columns will get added or dropped from the outputs just like you'd expect, requiring no more effort to adjust your data flow than any other component.
Performance
The other large obstacle in the SCD wizard is it's (frankly) underwhelming performance.  It's completely understandable once you know the design choice that was made (see below) - but that choice makes it basically unusable well before you have dimensions that get into the hundred thousand row range.
The KSCD component trades off higher memory usage in order to blow the freakin' doors off the CPU, powering through dimensions in about 1% of the time the SCD wizard takes.  (That 100x speed advantage is actually conservative, in my opinion.)
My thought when I was starting this project was - if you're processing a hundred-thousand-row-plus dimension, you probably have a hundred MB of RAM you wouldn't mind using to make it go faster.
What Were The Design Choices That Made This Possible?
As I've already alluded - the major design choices were a commitment to encapsulate SCD processing into one component only, and a tradeoff of memory versus lookups.  Everything else the component has going for it, I think the SCD wizard could do - if the SSIS team had some free time on their hands.
Multiple Versus Single Components
The first is puzzling to me - the SSIS team could have encapsulated all of the SCD processing logic into one component had they made that decision (with time and resources to back it up - but probably not that much).  Every other component in the toolbox works that way, why not the SCD?  Perhaps it was a time constraint issue after all...
Lookup Versus Caching
The second choice was a significant one that constrains both the SCD wizard and the KSCD to opposite sides of a coin - there are definite consequences to each design that can't be ignored.
The SCD wizard included in SSIS takes only one input - the rows coming from your OLTP or "source" system.  It uses a (limited choice of) Connection Manager to perform lookups on your data warehouse's dimension table.  The benefit of this is that the resulting component operates synchronously, doesn't require any memory overhead, and might be able to process multiple "changes" per business entity in one pass.  But due to the lookup implementation inside the SCD wizard - it isn't as performant or flexible as the logic in the standalone Lookup component.  It's a very basic "uncached" lookup - and this means that it suffers a complete round-trip SQL query penalty for each and every row sent through the component.  Think about that.  Take your best possible performance profile for a round trip lookup query, be it 20, 10 or even 5ms (!)... then multiply that by the number of rows in your OLTP table.  Completely ignoring any other performance penalties with SQL query round trips, a 10ms cost per row on a 100,000 row flow is over 16 minutes.  Not too horrible, but you'll soon realize how unscalable that math is, and how impervious to how big or expensive the hardware is you might throw at it.  It doesn't scale up or out - at all.
The KSCD component uses a different approach.  It takes the same OLTP "source" system input as the SCD wizard - but instead of a Connection Manager to perform lookups, it asks you to provide another input to it with the entire contents of your dimension table.  The benefit to this design is that the component can use the full power of however many cores your SSIS box has in order to make multithreaded comparisons of two in-memory data sets.  The flipside to that benefit is that it requires memory in order to hold those rows in cache - and depending on how "badly" your data flow is optimized, the component could theoretically have to cache one of the input streams completely.  In a worst-case scenario, for a 100,000 row dimension with a rediculous 8K row size, that could mean almost 800MB of RAM.  Again, not particularly scary - even bumping it up to a million row (of 8K rows!) dimension only means 8GB of RAM.  Under my (basic) tests, instead of 16 minutes for 100,000 rows or over 2.5 hours for one million, you'd be looking at times well under ten minutes, probably less than two.  And seriously - if you can't tweak your data flow to cut that memory usage in half, I'll come and fire you, because you have tons of options for improving the KSCD cache's performance.
The Sensible Way To Process Slowly Changing Dimensions
Review the pros and cons of the Kimball Method Slowly Changing Dimension component, and I think you'll find that it gives the SCD wizard, scripting, using multiple derived columns and conditional splits, and even the T-SQL MERGE statement a massive beat-down.  Each one of those ways of processing SCDs has a single strength: the SCD wizard is easy to set up, scripting is flexible, using multiple components is easily maintainable, and MERGE is performs great.  But none of them deliver on all those aspects like the KSCD does - it's easy to set up, flexible, easy to maintain, and wicked fast.  But don't believe me - check out the reviews people submitted about the previous version.  Version 1.5 delivers a lot more flexibility and usability on top of what they've experienced. 
I would also very much like to thank the community that's grown around this project.  I have had discussions with scores of colleagues online and in person that have resulted in me learning much more about "practical" data warehouse ETL processes.  Those SSIS experts have spent an inordinate amount of time diagnosing problems, educating me, and fixing code with me.  You have prompted almost all of the feature improvements inside v1.5 and continue to energize me about extending SSIS - thanks.
I'll be posting up some walkthroughs and more videos soon as well... stay tuned!