Theory of Computation Course

A friend of mine has asked for some tutoring on the standard CS undergrad course. This post is my syllabus for my work with him.

Theory of Computation

Lectures
Harvard Extension Course CSCI E-207
accessible via  http://www.seas.harvard.edu/courses/cs121/

Syllabus
The syllabus to match the Harvard Extension lectures can be found at
http://www.seas.harvard.edu/courses/cs121/syllabus.html

Schedule
The schedule may need to be adjusted to accomodate  your availability but here is a tentative plan:
There are 24 lectures. Ideally you would view 2 a week making it a 12 week course. But this would probably require meeting twice a week for discussion. This would be a minimum of 4 hours a week and that may be too much for your schedule. I suspect we will start and proceed as long as you have the time and patience and then stop once either has been exhausted.

Text
 Introduction to the Theory of Computation, by Michael Sipser, first edition, 1997, Thomson Course Technology


Homework
None planned but may be required to work through important concepts 

drug demand management

I have a friend who is committed to the challenge of reducing the demand for drugs in the world. I support her effort but we have our differences of opinion. She'd be happy to know that she has opened my eyes to the broader scope of this at the world-level. However from my perspective I still think that many agencies are still blind to some basic human issues that politicize this struggle to mitigate the harm from these substances. What is needed is a clear-eyed acceptance of the limits any agency will have in changing human desires and vice.

When it comes to any other drug other than marijuana (M), I am happy with her position. To various degrees, these other drugs lead to significant health problems, whether physical or mental, and have various degrees of addiction. While I am not always in favor of prosecuting and incarcerating harmless drug users, I will accept a certain amount of social harm in the broader war against these drugs. What I cannot accept is how the war against drugs sweeps up M and its users into this broad attempt. What I hope to do in this essay is lay out my view clearly.

We accept that human nature includes some vices. Alcohol and tobacco have been accepted substances for many generations and are well known to have negative affects on health. We have made considerable progress in reducing the use of one but not the other over the past few generations. This progress has been in spite of the legal status and even active promotion of these substances. The promotion of a healthy and responsible lifestyle does not seem to require the force of law. Is it really necessary for M?

Being a member of the boomer generation and probably the first generation that saw a widespread acceptance of drugs, I cannot say that I have seen much of a change in demand among young people for M in my lifetime. I cannot speak for generations before me and I may not have the best evidence for the generations younger than me. However I have known a great number of people who used M in their youth and many who continued the habit well into adulthood. I have witnessed a general reduction of use with age but no appreciable change in attitude about the drug.

This intimate knowledge of long-term users has done nothing to make me believe that there is any severe long-term damage done by the use of M. Any damage seems to be far less than for either alcohol or tobacco. Yet each user is a criminal and occasionally is caught up in the legal system due to this use. What appears to be even worse is that class increases the chance that a user will be caught up in a legal process and have their lives ruined or at least dramatically impacted by this use. It is a sad commentary on our society that money insulates you from the law and gives you far greater opportunities to indulge your vices. The poor and disadvantaged in our society have far more pressure on them to lead faultless lives and conform to the most strict moral codes if they are to have any chance of advancement for themselves or their children. Money confers great opportunities. But there are some opportunities that should not be bought and the opportunity to indulge our vices should not be one of them. As long as drug laws are not enforced as effectively against the privileged to the same extent they are among the poor, I see them as immoral more than the immorality of the drug use itself. As I said already, I can accept this for the other drugs but its use as justified tool in the war on drugs for M is beyond the pale in my opinion.

If you make something illegal that everyone uses, you make criminals of them all. While M use is hardly universal, it is so prevalent in our society that it strains reason to see why there should be a law against it. While enforcement is careful not to antagonize the influential people in society with their heavy handed tactics, the poor are often targeted as a tactic as a way of leveraging information and access to their real targets. So you not only force all citizens to resort to illegal distributors to gain a product that has high demand but you selectively alienate a segment of the population who are either otherwise innocent users of the drug or who engage in petty distribution of it for a profit. Either way, when use and demand is so widespread and police forces do not have the means or will to truly stop the use of the drug, you make a mockery of the law and inure people to their law breaking. For me, this in itself is a serious problem if you expect people to respect the other laws.

I begin to strenuously object to our current policies when I look at their consequences. A common justification for including M in the category of illegal drugs is that it is a gateway drug to harder drugs. I believe this has twisted the true cause and effect. By forcing people to procure their product from illegal distributors, you empower these distributors to promote other, higher profit products. If you are already buying M from a dealer, it is likely that deal also promotes E, C and many other substances that he can get from his supplier. Why do we want to put our citizens in harms way in this manner? Isn't it enough that we are powerless to help people live healthy, profitable lives and avoid the effects of alcohol and tobacco (to say nothing of fast food and sugary drinks)? Why must we also force them to be tempted to try yet another harmful substance and risk serious health consequences?

Pushing otherwise law abiding citizens into the arms of an criminal distribution chain also has the unintended consequence of strengthening this distribution chain. We already see the harmful effects of corruption and violence that these criminal enterprises bring. M is a major component of this profit. With every illegal purchase made by some suburban skateboard kid we are putting money into the pockets of a violent and unreachable criminal class that corrupts the rule of law and morality for a large number of people throughout this distribution chain. You may argue that there is harm to the teenager by his use of M. But what of the harm to the mules, and peasants who see their local police corrupted by the bribes and intimidation from these criminals? Are our suburban children really so precious that we must countenance the wholesale corruption of entire societies that are unable to afford the kind of enforcement to control the supply chain? Why must our desire to reduce demand for M be borne by people who are struggling for basic necessities?

I can appreciate how abhorrent the idea of legalization of M is to many people. But what is the more honest and moral choice? To continue this epic battle that has become a stalemate? or to accept this as a public health issue and a call to accept humanity with its faults rather than struggle to remake people into the puritan mold we wish they could fit into? There are so many benefits to this reasonable step that I scarcely know where to begin.

The criminal elements than ensure the smooth transportation of product from growth to consumption derives a significant portion of their revenue from this product. If the product becomes legal, this revenue source is eliminated and the enterprise will be forced to shrink. This will lead to fewer criminals, fewer guns, fewer pushers, fewer arrests, fewer prosecutions. The savings to the public treasury could be either used to reduce taxation or to fund new programs to discourage the use of the product. What is even more important is that making it legal creates the opportunity to regulate and tax the product. Demand can be manipulated by taxation and regulation of supply. As long as the taxation and ultimate street price remains close to or below the prior illegal price, the criminal distribution chain must either accept a reduced profit from the sale of M even as they see many of their consumers move to the legal channels of distribution. The most likely result will be their exit from the market.

Another concern that is relieved with the introduction of a legal distribution chain is the control over the purity and strength of the product. In my lifetime I have seen an unbelievable growth in the strength of the product available on the street. Longer distribution chains from exotic locations, the application of agricultural techniques to improve yield and potency have all but eradicated the M that I grew up with in the mid-west. Even worse, there are many stories of unscrupulous vendors who have adulterated their products or unwittingly promoted such products. The hard inherent in these trends and practices are eliminated by the creation of a legal source.

The biggest objection to legalization is a slippery slope argument. I will not even bother shooting this down. But a more cogent objection comes from the shape of the legal distribution channel. People who cynically see big business moving into to capitalize on this large market for a high-profit product rightly fear what could be brought forth. I share their concern but do not see it as a destiny. The legal advertising for hard liquor is a recent phenomenon. Many states till maintain package stores which are the only source of liquor. If such restrictions can be maintained for liquor, I fail to see how public policy would fail in the introduction of a controversial product. Perhaps in some future where the societal concerns are found baseless will we see M marketed on billboards with slick campaigns. But in the short term I don't see this as anything more than fear mongering.

I don't see the legalization of M in the US market as being the fearsome thing its opponent suggest. I think it would do us tremendous good in our relationship with Mexico and the other Latin American countries who are bearing the brunt of our poorly conceived and executed "war on drugs". It should lead to an immediate reduction in the crimes that sap our judicial system and make the true use of the common substance more visible and hopefully lead to a better balance between approbation and condemnation. It will also lead to safer streets in the towns in Mexico most embroiled in the war between the narco-traffikers. Yes, the puritans will recoil. But in the end we are ensuring a more moral world and an honest acceptance of our faults and not a continued adherence to some romantic and hypocritical ideal we will never achieve.

Mind the Gap

For anyone who has visited London, the title immediately brings to mind that continual reminder that the London subway system was built many years ago and there are sometimes some very significant gaps between the platform and the train. An unwary traveler can easily step into that gap and find themselves embarrassed at a minimum or even seriously injured as they fall halfway to the tracks. While it is drummed into your sub-consciousness in your time spent in their "tubes", I don't recall ever hearing that message on a continuous loop. But here in a airport in Pittsburg, built at least a century after the earliest London line, I hear a comparable warning on a continuous loop to no effect except to annoy people waiting nearby. I find this a sadly common situation in American society and design and I would like to find a name for it.

An innovation in airports and other transportation centers is the addition of moving walkways. It is hard to believe that there is still an American citizen left who isn't familiar with them by now. Yet the walkway near my gate has speakers at the end of each walkway that plays the loop "Caution...moving walk is nearing its end. Please watch your step." with only a two second pause between repetitions. Now when someone is on the walkway near its end, it makes perfect sense. However this system has no sensor so the loop plays even though there are no walkers, and in fact have not been for more than 10 minutes. I can just imagine why this has come to be and it is that image that keeps me at a slow burn with each repetition.

A continual theme from the political right is the nanny state. While I disagree on where this came from,

The Failure of Reductionist Thinking in the Creation of Software Intensive Systems

My career was dedicated to helping large organizations build software intensive systems that would solve their business needs. This was an honorable calling and one that had many moments of intellectual joy. With each level achieved, I perceived that the projects were often doomed to sub-optimal achievement by decisions made earlier in the process. I was drawn inexorably to these higher levels like a moth to a light. Since my capabilities allowed me to ascend toward management ranks I was allowed to see how these decisions were arrived at and to make my attempt to avoid the pitfalls I had witnessed in earlier projects. The culmination for me was to be project manager for several large, complex, high-risk projects, work that I eventually found soul crushing and caused an existential crisis that I am only now recovering from. But this experience has left me with a clarity of vision on how management and the engineering staffs they employ come from such different cultures that to find they achieve anything at all is a testament to the underlying unity of human society.

Projects are funded to solve business problems. This goal directed nature of these efforts gives them a vitality that is lost if the goals cannot be succinctly expressed or are not shared by the stakeholders. This deep understanding of the goals is the sole province of the leaders of that organization. Many projects fail at this level do to the annunciation of commandments or the articulation of such laudable, but ultimately vacuous goals as to be useless for the guidance to a satisfactory conclusion. At this level, at a minimum, a project must be able to articulate how a successful conclusion will be recognized. There is no great harm in allowing untestable or abstract statements. However management must recognize that their job is not complete until the project can articulate a set of concrete business objectives that will justify the capital and expense of this project. It must also accept that this is a contract with the project team and the project stakeholders that constrain them to accept this as a stated end position and only change it with the full cooperation of the project team. After all, things change and knowledge is learned during the articulation of the goals. But too many projects lose focus and support when the vague goals they started with are never reified into something tangible enough to drive decisions and the most energetic members of management move on to other more alluring goals leaving the project with the heavy lifting of ensuring at least some reasonable goal is achieved with the resources expended.

Traditional waterfall methodologies inherited from the success of very large well disciplined organizations to manage large complex projects may never have worked very well but where they were employed with thought, consistency with well trained and stable staffs, they worked better than anything else that had previously been tried. The dominance of the model of project process led most managements to view the creation of software systems as more akin to the creation of an automobile on the assembly line than the creation of a work of art with inscrutable processes and unpredictable outcomes. Yet the experience of the past 40 years suggests that the latter may be a closer model of systems creation than the former.

The most recent understandings of project failures involve the consequences of the earliest design decisions on some of the most valuable properties of the system to be created, the emergent qualities of the complete ensemble. This view is supported by the thesis of SEI which posits that most, if not all, "non-functional" qualities of a software system derive from the was the system is decomposed and constituted during the design process. If is oft stated that you can't tack on (take your pick here: usability, security, performance, etc) on the end of an already existing product. It must be a design goal from the beginning due to the cross cutting concerns. That is to say that qualities are orthogonal to the functional needs. If you think about it, it cannot be otherwise since we know we can refactor code in any number of ways without making any change in its dynamic behavior. Yet one or two inappropriate decompositions can make the achievement of some desirable qualities difficult or impossible. Yet these earliest design decisions are made long before there is any significant understanding of even the problem-domain, no less the solution-domain for this project. This reality puts architecture-centric techniques into a tension with Agile techniques which are embraced by managements for their fast delivery of tangible results. To balance these two competing forces requires a management that can weigh short-term gain against long-term investment; that can participate in the technological vision of an architect and create a safe-space in which difficult concepts can be given enough time to mature without engaging in a blind trust that time and money will necessarily result in a better product.

Another danger faced by management at these early stages of the project is the lack of maturity both within management and within the engineering community to properly analyze the problem-space. From an engineering perspective, the requirements engineering phase is at its best if it results in a set of models which express views of the problem-space in a way that enables the designer to project a new and different version that can be understood by both the business and the development organization(s). Yet no matter how powerful, these models are often the source of a great deal of damage. At this point I always have Magritte's famous "pipe" painting in mind which proclaims "this is not a pipe." The point made is that the painting is a depiction of a pipe, not an actual pipe. In software even more than in art, the error of reification is so easy to make that it is almost impossible to avoid. No model of human or organizational behavior can ever capture the true richness of behavior that it is capable of. But rather than embrace, or even acknowledge this truth, management will attempt to use a model to constrain behavior in an attempt to achieve uniformity, uniformity that may be driven by a desire to reduce the skill set needed by the worker, to ensure that different organizations create an identical product or merely to believe that they are achieving some business goal that is never expressed. This reduction of human work to a standardized process has been studied since the dawn of the assembly line and doesn't need to be rehashed. But what is important here is how this mentality can cause the project to mistakenly believe that the model of the organization into which a system will be crafted really expresses the behavior of the people and the jobs they do rather than allow for the extra-system work that in most cases allows the system adapt and remain resilient to unexpected input. The model is not the process.

The biggest dangers in systems development are seen when the team believes they understand the problem in sufficient detail to fully articulate the solution specification. This is the realm of design. This design exists on a continuum from the selection of previous designs that satisfy this requirements, as if the problem is a burnt out bulb and the solution is the replacement with a functionally equivalent one, to creation of a novel and innovative solution to a problem that had never been tackled before, such as the creation of a spacecraft to support colonization on Mars. Too often a project is funded as if the problem space is burnt out light bulb while the stakeholders want to go on a mission to Mars. Project management, with support from senior management, must ensure that everyone on the project is guided to the same place on this continuum and that this scope is normalized to the budget it is given.

Another pitfall during this phase of the project is the abdication of management decision making. Design is all about tradeoffs. I haven't met a business manager yet that will not quip when asked if they want it fast or cheap will not answer "yes." It is not unreasonable for business managers to seek a quality product that is instantly available, does everything they could ever think of for that product and for it to be free. But we all know that it is impossible. Decisions must be made and if the business will not make them, the designers will. For the business to not engage in this decision making (assuming that the designers do not usurp that responsibility) is a clear abdication of their responsibility. While a talented designer may actually make more astute decisions than the business, the business is left poorer for its ownership of those decisions and it leaves a flaw in the crystalline connections of forces that connect the problem to the solution via a line of empowered managers. What is sought is a product that once delivered is known to represent the best decisions possible from this organization and one that cannot be disowned by those managers.

This brings me to what I believe is perhaps the most intractable danger in systems development; the achievement of emergent properties in the system. The drumbeat for "quality software" has been growing over the years and will surely increase as we attempt ever more complex systems in ever more demanding areas like health and transportation. Yet what we are grappling with is an engineering approach to achieving properties that have more in common with chaos and stochastic processes than they do with Newtonian mechanics. The Alexandrian pattern languages will enable us to evolve solutions through trial and error. Yet to an engineer this is an unsatisfactory place to be. By what principles, by what mathematics can be envision some emergent property and then back into the simple rules that will allow this property to be expressed? Kurtzweil suggests we will see singularity before 2050 but even if we do, I don't think it will do us much good. Reaching the tipping point where our manufactured logic machines have comparable statistics to the wetware in our heads does not guarantee that they will become HAL-like. Unless and until we know more about the brain than the number of neurons, dendrites, axons or even their wiring pattern, I don't think the hardware achievement will mean much beyond the new economics of computation. There is still a new mathematics waiting to be created out there, one that does not fall prey to reductionist thinking.

Are They Really Requirements?

The paradigm we get from the waterfall model is that a client states their requirements and they lead unidirectionally to the specification for the system. We already know from the popularity of the Agile methods that waterfall is not the correct model, that for many projects it is more productive to approach it as an artistic product and make many iterations that can be shared with the client. The benefits of this approach have been well documented but it still leaves the idea that there are some set of requirements that just need to be uncovered and refined. I now believe this is a fatal flaw in this part of even the Agile methods.

Iterative methods gain their power from the inclusion of the client in the design process. In the older waterfall approaches, after the requirements were documented and signed off, the client would often see little progress (except bills and excuses why the project was behind schedule) until an advanced level of testing allowed the client to see the complete system as it is intended to work. The pain this caused when it was realized that what was built was not an acceptable solution to the client's problem was always significant since so much had already been sunk into the wrong solution. Agile at least boxes the risk into a single iteration and if they are small enough, the damage can be limited. One good trait of all these Agile methods is that they support faster failure of dysfunctional projects.

Clients have always had misgivings about requirements signoffs. While they could not articulate it, they felt they were being setup for failure. Their business expertise did not prepare them for the task of directly specifying the product they needed nor is the average business analyst prepared to completely understand the business function that is within the scope of development. Neither is generally prepared for the design tradeoffs that are often done without any direct client involvement, even if the team is articulate enough to explain how a particular design decision impacts the various emergent properties of the system under construction. The current methodologies simply don't work during the requirements elicitation and analysis phases of even the most Agile methods. Instead they substitute what I like to call the waving of the hands form of requirements where everyone simply talks about what is needed, very often in "I'll recognize it when I see it" kind of terms. The design team does their best to interpret these statements, goes away and comes back with something that can be critiqued. The progressive elaboration of models and prototypes will allow an experienced team to eventually drive to a solution.

So what is wrong with this approach? After all, it does produce a solution. Is it optimal? Who knows? Is there any traceability to the design decisions made? Probably not. Does this sound like engineering? Emphatically no. We can do better.

What this approach lacks is the top-sight and planning that enable an architecture centric approach. Many solutions do not depend upon an architecture centric approach to achieve business success. When the solution is self-contained, highly derivative of a prior effort or does not have exceptional quality requirements, the emergent properties of the system will probably not be difficult to achieve when there is little or no "big analysis up front." But this is not the forefront of software engineering today and the solution to these challenges has not yet been found.

Emergent properties of a software system do not derive from one or even a small number of design decisions made when designing the product. Rather they emerge from a collection of many, most or sometimes all of the elements that comprise the system that have cross-cutting concerns addressed in a manner that allows the ensemble to achieve the collective goal. When the top-level decomposition of responsibilities has been properly done, these emergent properties can be achieved by independent developers working on sub-systems within the larger ensemble. But too often poor choices made in the first few design decisions can block the achievement of an emergent behavior even when all purely functional properties exist in the finished product. If the emergent properties that exist do not satisfy the business need, the product is rejected. But if the tradeoffs provide a product that "satifices", the product will be accepted and since the knowledge of what might have been is at best speculative (at worst, vindictive), the missed opportunity will never be known.

This can be improved by recognizing the key decision making role the client plays throughout the design process. The key challenge faced in trying to include the client, though, can be easily seen in a near universal scenario. The client has established a timeline and budget for the project. Then the requirements gathering begins. At some point, an astute engineer will observe that it is possible to get the needed functions and qualities a, b, and c at levels of x, y, and z and this combination is judged unacceptable. The engineer turns to the client and says, which are you willing to sacrafice. The client says "none. I want them all." What is really being said here is that the client is abrogating their responsiblity to be a decision maker in the project. If the engineer is correct that the qualities cannot all be achieved simultaneously (time and budget being two of those qualities), then the decision is foisted upon a team member to make a decision for the client. Just as the client felt cornered when asked to sign off on the requirements document, so too does the designer feel abandoned just when he needs the client the most. This is, after all, a business decision, a business decision that can have real business consequences. What is needed is for the client to engage in a form of negotiation to explore the possibilities, ensure that it really is impossible to achieve all goals simultaneously and then to take responsibility for the ultimate responsibility with the help of the business analyst, architect or whoever is in the lead design role.

So you can see that it is not as if requirements are completely elaborated at the time the project team ordinarily asks the business to sign off on them. This is an important first step and one that cannot be taken lightly. But neither can this be seen as the final word for business input. For an architecture centric approach, it must include sufficient elaboration of the qualities that can only be achieved with cross-cutting concerns that span a large portion of the code base to be developed. For any development it must at least be accurate even if it is not complete. But the impact of the aspirational goals cannot be known until the design progresses to a lower level of design and only then can the implications of that set of aspirational goals be known. As soon as possible thereafter the client must be brought back into the design process to negotiate the tradeoffs that should be made.

This process is much closer to advocacy than it is to specification. As leverage points appear there is someone to argue for the most advantageous tradeoffs that achieve their client's goals. For the client representative(s), it is to collaborate with the chief designer. For the project team, it is to do the most professional job possible in predicting the properties that will be present in the final product extended from this design.

from

Tim Bender to me

show details Sep 10 (2 days ago)

In discussing a large software project with an artist, I conjured the analogy that a software product is like a painting. Each engineer is an artist with their own style and they must paint a small portion of a large masterpiece, often without ever being able to see the whole thing. Recently, I watched a movie which made me recall this analogy and I pondered it further as a way of explaining in quite a simple way some of the rather complex interactions that occur in software engineering.

The idea centers around giving groups the common task of creating the simple image of a house with a lawn, a small family, the sun, a bird, and a flower. The image would need to be simple enough to be easily reproduced by an individual, but complex enough to offer varying entry points for concept learning opportunities.

Some of these concepts are weak and need some fleshing out.
1. Requirement solicitation:
Scenario: Give a team a sheet of paper and some coloring pencils. Express to them the importance of this drawing and that it must look exactly like what is being requested. Tell them to draw "a house with a lawn, a small family, the sun, and perhaps a bird and flower or something". Being purposefully vague and leaving them to either create only the initial description, or go ask follow-up questions.
Challenge: Requesting help and/or more information.

2. Specifying interfaces:
Scenario: Give a team the exact image they are to produce and a collection of transparencies attached to construction paper (making them non-transparent). Tell the team that each person must draw something on their transparency. Inform that the transparencies will be stacked to create the complete image.
Challenge: Specifying interfaces clearly to minimize integration failures.

3. Organizing for a task:

Scenario: Give a team the exact image they are to produce and a blank canvas with some pens. Tell them that each of them must contribute something and that they will be asked to state their contribution. From there, let them self-organize.

Challenge: Skill auditing. Some members will be excellent at sketching/drawing. Some may be terrible. Those that are not gifted in this area could take on a small portion of the image or provide administrative support. A variation might include purposefully leaving out supplies that a team member must requisition.

I was curious if you would have any thoughts on this.

***********************

Me? Have opinions? Perish the thought!

This is an interesting metaphor but I probably don't see it the same way you do.

On my visit to Vienna I had a curated tour of the Kusthistoriches (Art History) Museum. In the pre-20th century tradition of large art studios, it was common for the studio to accept commissions in very must the way you describe and in a way that does have some interesting parallels to software engineering. Each studio is headed by a master who is the creative genius that gave the studio its fame and provides the marketing effort to keep it employed. But given the output of the studio, it was not possible for the master to personally paint all the canvases. Raphael ran such a studio and we spent time talking about how to tell a Raphael from his hand versus one that he may have never touched.

In the studio system the various artists had their individual talents. The less talented might be relegated to painting landscape backgrounds, another buildings, etc. Only the most talented would paint the main subject which was always a human being. The ability to achieve life-like form and tone was prized and only the very best could capture the "essence" of a human subject. For historical paintings that had many figures, the master might paint a few of the figures but would allow the studio to fill in the remainder. This form or organization seems to be very similar to how a software project develops. There is someone with experience who is responsible for the early launch and concept. But what is different is that unless it is an architecture driven development, there is no chief designer who will oversee the design from start to finish. With each phase transition it is the project manager who oversees the progress toward the goal and not an architect. A project manager is usually more focused on the more mundane aspects of the project such as time and budget rather than the less externally visible properties of the system under development.

There is a fundamental difference between art and software. For art, the effect of the completed product is what is valued and not the qualities of the individual pieces, no matter how good. So it is with a software system. A property like security is not solely dependent upon any single component, although that cross-cutting concern will be seen in many of the modules. The lack of attention to even one point in the design can doom the security of the entire system. To one extent or another, this can be true of all the quality (non-functional) requirements placed upon the system. Performance is famously lost through a weak-link in the processing as is availability; modifiability by the inappropriate decomposition of the modules; usability by the lack of a facility for an un-do in the transactions. These qualities can be reduced to an engineering model that provides quantitative, or at least, testable properties. This is not true of art which is often said to lie in the eye of the beholder; not a good quality in an engineering project.

Ironically, I have seen many project proceed as if they were art project. The project management, the client, the business analysts, and the coders all proceed with the functional requirements only and wing it with the crap that is often offered as "non-functional requirements." This forces the team to make decisions for the client as to what qualities and what relative priorities to give to those qualities. This causes the client to not perceive the lack of some quality until they see an early prototype of the product, or worse, the finished product. Then, and only then, does the performance/availability/fault tolerance/usability/security get the attention that it deserved often with disastrous consequences to the design concepts that had guided the many hands that crafted the code.

The number 2 scenario sounds exactly like the way cell animation is done. I think the key difference here is that the transparencies are opaque and therefore block what is behind. This makes the "interface" irrelevant since there is no need for coordination between Bambi and the forest trees behind her. As long as there is no interleaving between two transparencies, there is no need for the careful coordination that is suggested with interfaces. Software, of course, is very different since most any decomposition of a function into smaller modules will create some form of interface that must be spec'd.

For any product that will be created by many hands I cannot believe that self-organization, as it is often understood, can work. I think most people take self-organization to be some form of egalitarian 'let's all get along' I do not believe this is what the Agile manifesto suggests. Rather I believe the Agile manifesto suggests that experienced and skilled professionals are more effective of organizing themselves for the task without an explicit plan from some "manager" who is not as experienced with the technology. However all the same roles will be filled. Group dynamics ensures that a leader will be selected and that the group will go through the steps of forming, storming, norming and performing. Self-organization will also do nothing to prevent the group dysfunctions that affect traditional forms of management as well.

Going back to your example, if a group of artists are given a commission and left to self-organize, the leader that emerges is not guaranteed to be the best artist. It is often true in engineering circles as well. Arrogance, self-confidence, domineering personalities are the most likely to be the group leaders. When this is also the most talented individual and has the leadership skills to bring the team along, the results can be striking. But it can just as easily lead to disaster or a mediocre product too.