Quantity versus Quality: Fifty Pounds of Innovative Yet Broken Clay Pots
There is a story retold in the virtual reality community which emphasizes reaching perfection through a quantity approach over a quality approach. The text originally came from the book Art and Fear, which is about the process of making art. I like Derek Sivers’ shortened version, so I’ll repeat it here.
The ceramics teacher announced he was dividing his class into two groups. All those on the left side of the studio would be graded solely on the quantity of work they produced, all those on the right graded solely on its quality.
His procedure was simple: on the final day of class he would weigh the work of the “quantity” group: 50 pounds of pots rated an A, 40 pounds a B, and so on. Those being graded on “quality”, however, needed to produce only one pot – albeit a perfect one – to get an A.
Well, come grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity!
It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.
Sure, you have to question the authenticity of the story, but for most people, the lesson rings true. This is the lesson that we should walk away with, right? Quantity trumps a quality approach when trying to reach perfection?
No. Not at all. It is critical to understand the story in its original context.
QUANTITY VERSUS QUALITY IN LEARNING
These students aren’t exploring the cutting edge of clay pot design, finding new ways of pushing mounds of clay into innovative and problem-solving symmetrical shapes. They’re introducing themselves to a very well known skill. They gaining mastery in a known art. Nothing more.
What is the real moral of the story? It couldn’t be simpler.
Practice makes perfect.
So many people have repeated this story and walked away with the wrong moral, which is unfortunate. You know what is worse than learning the wrong moral to a story? Learning the wrong moral to a story and then applying it to the wrong situation.
Are you learning a new computer language or figuring out how to use a new tool? Quantity trumps quality may be the wrong lesson, but it is the right approach because practice makes perfect. Despite its flaws, sometimes the story works.
QUANTITY VERSUS QUALITY IN PROBLEM SOLVING
Are you figuring out the best approach to solve a new problem? Quantity trumps quality, right? No. It isn’t that simple. If you’re letting a short story guide your problem-solving methodology, you’re probably not doing it right. Even worse, you are basing it off of a parable that has nothing to do with solving new problems or innovation.
Still, when someone applies this particular story to the topic of design, what they are trying to say is that you should not to get caught up in finding the optimum approach. Try something and learn from your mistakes. Use the experimental approach.
That’s fine, but you shouldn’t take that advice to the exclusion of everything else. The opposite side of the coin is true, too.
Don’t get caught up in making lots of small changes and iterative improvements when you have more to gain by rethinking your whole approach. A good design will increase the efficiency of your problem solving efforts. A great design will change the entire nature of the problem that you are trying to solve.
My message to you is this: in problem solving, one solution doesn’t fit all, and don’t take any one solution to excess. Weigh your options. There is more than one way to reach a solution, and each way has its own advantages and disadvantages. Use the right one for each situation. (Please don’t make me drag out that quote from Chet Faliszek of Valve which tells you to question the common wisdom of what others are doing in VR development.)
QUANTITY VERSUS QUALITY IN THE METAVERSE
I’ve seen this quote applied to metaverse implementation, but in a different way. The idea is that time spent on designing how a metaverse should work is wasted in comparison to actually going out there and building one and iterating on it. You know. Stupid eggheads thinking about the problem rather than going out and doing something, right?
In particular, a quote from John Carmack is used to support this view.
It is almost painful for me to watch some of the VRML initiatives. It just seems so obviously the wrong way to do something. All of this debating, committee forming, and spec writing, and in the end, there isn’t anything to show for it. Make something really cool first, and worry about the spec after you are sure it’s worth it! –John Carmack Answers (Slashdot)
Let’s keep in mind that this particular quote about a single design effort is now sixteen years old. There is little question that VRML was flawed. Shortly after that quote, John had his own opportunity to pitch an implementation of the metaverse, and it failed to gain traction within his own circle. Other implementations have since come and gone, and despite all of the experiments and iterative improvements, none have really succeeded.
We’ve spent years with the experimental approach and, really, Second Life in its heyday has been the pinnacle of what has been achieved. What has prevented them from experimenting and refining their approach? Why did nobody arrive at the ideal solution and then publish the spec as Carmack suggests?
It goes back to those students and their clay pots.
CLAY POTS IN THE METAVERSE
The story of the clay pots works in the specific scenario for which it was designed. Students learning a known craft. We can assume that the labor of the students is absolutely free. We can assume that the clay is provided at no additional charge. We can assume that there is no penalty for making a mistake and throwing it away.
That is how education works. That isn’t how the real world works.
What if Second Life decided that they might be able to make more money and attract a larger set of users by charging for traffic rather than renting virtual land?
In the real world, we have opportunity costs, both explicit and implicit. We have the labor, hardware, and software required for platform development. We have a reputation as a steady platform to build upon. We have partners who are ready to jump ship when things go wrong. A lot of people who have built upon the concept of land ownership are going to be unhappy.
What if the experiment fails? Well, they could back out the change, but now they’ve spent a lot of money and they are back to where they’ve started. Worse, it has cost them some of their pre-existing business and it has also cost them the trust of their community. The experimental approach comes with a price.
If the direct and indirect costs of making a new pot costs $100,000, how far does it make sense to pursue a quantity approach? Once you have the start of a good platform, does it make sense to experiment and iterate our way to something even better? Probably not. Why? Because with real-world constraints, it suddenly makes a lot more sense to invest in design.
At this point, you should be able to explain why Linden Labs is pursuing Project Sansar instead of fixing Second Life. You also have a better understanding of two of the fundamental problems with metaverse implementations that we covered earlier:
“The more complex and integrated a platform gets, the slower its innovation becomes.” Technical complexity aside, with increased adoption comes increased opportunity cost.
“The real challenge isn’t technical.” You can’t make technical improvements in a vacuum. Social considerations (such as popularity, trust, reputation) and financial considerations (business contracts, revenue, legal considerations) come first.
Hopefully, you can see why people go through the trouble of building consensus, developing specifications, and publishing documents. You cannot casually change directions in a large project and innovate your way to a new approach that offers a better direction. You’ll lose much of your support in the process. The up-front work is critical to success.
These people believe that a global rationalization of a metaverse might work out better than a series of individual design choices, arrived at by experimentation, which are merged into a single architecture.
REALLY, IT IS A CHOICE — BE SMART ABOUT IT
Is design the answer? We all should be approaching a metaverse through design, right? No, you’re missing the point! Let’s not fall for the trap that we have to follow one specific approach to arrive at a metaverse. Perhaps this topic has more than a few similarities with a comparison of linear and lateral thinking. (Neither one provides a universal solution.)
Carmack’s criticism of creating specifications and building consensus is valid when that effort is divorced from real-world considerations and specific action. In a rapidly changing environment, which is where we are right now, the design process has the risk of quickly becoming irrelevant. A global design is never perfect, it is always going to have holes that need plugged. Developers shouldn’t let an elegant design lure them into dismissing more practical considerations (both technical and non-technical).
On the plus side, a good design will illuminate a more efficient path to solving a problem. A great design will reshape the problem entirely with a revolutionary change.
Experimentation and iterative improvement also has its challenges. Do they even know if they are starting with the right approach? Experimentation and iterative improvement may require a very large number of cycles to tackle more global issues. If a developer creates a successful and active platform, the opportunity cost of change will increase over time. Developers should be reasonable and guard against a string of early successes building excessive confidence in any particular direction.
On the plus side, experimentation can find answers when we really don’t understand the problem. Iterative improvement is grounded in real-world problems and is able to demonstrate incremental success. It fills in the gaps that otherwise would go missing. The right change can unlock new opportunities that were unseen up until that point.
We shouldn’t dismiss either approach.
Use the right approach, at the right time, at the right level of focus, for the right problem. Use your judgement. If you have been consistently following one path, try another. If everyone else is following another path, also try another. Balance can yield reward.
One last thing to consider. Balance is good, but do we have to arrive at balance through a string of boolean choices? No. If you can find a way to blend the two approaches together, a global design which harnesses experimentation and iterative improvement, you’re probably ahead of the game.
Now get out there and make your perfect clay pot.