Tainter's Composite

Introduction

First let me start by defining my terms.  Dr. Joseph Tainter, is an anthropologist who looked into the question of why societies collapse.  A composite is combining multiple things into a single image.  I believe using Tainer's mode of thinking, one can create a model or system including for software and organizations.  This model, using a lot of systems thinking includes the work of the Dreyfus model, The Gervais Principle and general economics, but the framework all rests on is from Tainter.  I will explain Tainer's ideas, however, if you don't know about the Dreyfus model or Gervais Principle, you will need to read up on those before continuing further.  In fact, without those you will be lost.  I realize it is a huge amount of reading just to read this post, but I promise you it is worth your while if you want to understand the inner workings of business culture.

Tainter asserts that societies become more complex as the societies needs for solutions to problems increase.  That is to say, in order to solve a problem, the society adds complexity.  For instance, when a tax loophole is discovered, society might create a rule around that loophole to end it.  This complexity means that someone must create the law, the people who are affected must learn of it, there must be a means of enforcing the law, someone has to interpret the law and when violated someone must enforce the penalty.  Most laws might add minor complexity, but at some point the laws become uncountable, making it very difficult to follow the law.  Now this by itself might not be a problem, unless the value received from additional complexity declines compared to the cost.  When enough of these sorts of mounting complexities cost more to the society than the society produces, eventually the society will fall.

A Corporate Example

Let me consider a different form of this, one that is very easy to understand.  Let us suppose that you have a piece of software that requires 9 engineers but has an income stream sufficient to pay for 10 engineers.  Customers keep demanding new features, and the complexity of the system rises.  Each new features has both cross-cutting concerns and interlaces with existing features.  More bugs are found, but few new customers are added.  Eventually, those whom knew the project will leave, but the complexity of the project does not.  The company finds itself needing more engineers in order to support the product.  The new engineers conclude that the best thing to do is to rewrite the application.  Even if you ignore Joel's words (from the last link about refactoring vs rewrite), you know that then they had to hire even more engineers to support the old application while the rewrite was in progress.  The company had not been making that much money, didn't appear to be able to make much more and so the obvious thing to do is the close shop.  A sociopath from The Gervais Principle would clearly do so.  They would not feel bad for the engineers who lost their jobs, for the customers who lost the product or even the history of the company.

This scenario has not happened to me personally, but I know that at least twice my entire automation code set was abandoned because the person whom replaced me was either not an expert or was at least less experienced compared to me in their programming skills and the company had no one else beside myself who knew the automation code base.  Instead of trying to learn the code base, the less knowledgeable people threw it away and started over.  Why do that?  My guess is the person doesn't understand the complexity in the system and decide that the last guy or gal must have been an idiot.  I too have inherited code bases, and not once have I started over from scratch.  I have thrown away pieces and parts, but never the whole, so I can't directly answer it.  I suppose, I once discover there was former automation scripts about 6 months after I started a project written in a different programming language.  In that particular case, the code developed had already outstripped all the functionality from the existing discovered code.  Even in this case, I still reviewed the work to ensure we had capture the same set of solutions.  This leads into an interesting question of losing complexity by forgetting, however, I am not going to cover that topic.

Company Growth & Avoid Responsibility

This also applies to big companies and as I alluded to earlier, governments.  According to Tainer, complexity growth without equal or greater forms of production end up with an eventual collapse.  Consider HR policies of different organizations.  With self-employment, there is no HR, thus zero complexity.  With a company of just a few people there is likely still no HR, but rather just a group of people all working towards a unified goal.  Often these people are very close.  Eventually there is a need for someone to manage the complexity and you hire someone into HR.  As you keep growing, someone does a boneheaded thing, like not taking time off and not call in sick for a week and then show up expecting that they still have a job.  There is no policy saying they will be fired and so they demand that they either keep their job or get unemployment.  So a policy is added, no big deal...

Then you get to be a mega corp with 100,000 employees.  You have divisions bigger than most medium sized companies and you have a handbook big enough to use as a deadly weapon.  Big organizations are often not run nearly as kindly towards employees because (in part) no one is empowered enough to go do that or if they are, it is spread unevenly causing jealousy, which causes things to then be 'not allowed'.  Having a pizza party for success now makes 500 others unhappy as they smell your reward.  You made 50 people happy and 500 grumpy.  An expert would have seen the problem ahead of time and planned on making it outdoors or away from the other employees, but the person who had the party didn't know better. So the company decides to make a rule saying you can't do that.  Eventually all these rules add up.  They hurt morale because the Dreyfus expert knows its stupid but has to think about how to get around the rule, making their 'expert' skill less valuable (as the point of the expert is they don't use rules).  They, like most Gervais losers (economic losers) flee, making such companies lose their best employees. It also encourages people to not try to make work better as that would break norms and likely added more rules to the already overly large set of rules they have to deal with.  Finally it encourages people to quit thinking because there is a rule, making them even less effective.

There is yet another reason that rules are created.  As The Gervais Principle describes, the sociopaths are always interested in getting the clueless to accept as much responsibility as possible while giving limited credit or authority.  The clueless are a buffer for the sociopaths so that they get as much value as possible from the losers without having to deal with the losers.  This is a form of protection that a zealous clueless person is willing to do, however, the clueless often don't know how to actually run an organization.  The clueless are in a position in which they use baby talk to pretend they know what they are doing.  Since they don't actually know what they are doing, the sociopaths use rules to force the clueless to do what the sociopath thinks is right.  These types of rules create a different type of complexity.  This is a social complexity, where the sociopath must juggle personalities to get the most production.  The clueless is actually a cost, but a small cost considering they are a sort of hedge that if things go wrong, the clueless can be blamed.  In a larger sense, this means that the person with bad ideas but who is good at manipulation can maintain power for a long time in spite of making bad choices because those choices aren't attributed to that person.  This can eat away at an organization or society, building up costs from complex institutions with mild production value, further corrupting the society.

Cat & Mouse

Capitalism is an attempt to alleviate the ineffective manager who somehow remains employed via competition.  The problem with capitalism, is like any system, the complexity builds up to protect the system from being hacked.  This can be anything from monopoly laws to defenses from regulatory capture.  However, this sort of question is mostly capture by Tainter himself, so I won't dive into it.

Ultimately the Tainter Composite describes how complexity not only infects society but the institutions around society.  The final piece I want to describe is how this leads to a cat and mouse set of activities.  Often rules are made because there is a real and legitimate concern.  Some of my examples might seem silly but are actually true (I have experienced some of them).  However, even 'smart' rules can get you into trouble.

Consider black hat crackers and the white hat hackers.  Code is in fact just a form of rules, often very useful rules.  You are currently reading this using a system with more lines of rules than one person could memorize.  The trouble is, there is always a group who wants to enjoy finding ways around these rules for their own purposes, just like some of the troubles in capitalism.  It might be someone who just adds everyone to your friends list or it might be to take all the money out of your bank account.  As security gets better, more rules have to be applied to protect the system.  Will code ever get to the point where maintaining and adding rules are too expensive for one person ?  Will it ever get to the point where building up all the rules is more costly than accepting the attacks?  To some degree that has happened where most people have traded up freedom (complexity) for a walled garden (less value).  Once again, this gets into reducing complexity, which is a topic for another day.

Rather than telling you how this applies to testing, effectively creating rules for your brain, I am going to try not to add complexity and let you interpret it for yourself.  If you found this useful, please write a comment and I will write more on the topic, otherwise, I will leave this dense, difficult topic alone for a while.