The Surprising Math of Cities

Our Perceived Loss of the Frontier

How many secrets are there in the world? How hard it is to obtain the truth is a key factor to consider when thinking about secrets. Easy truths are simply accepted conventions. Pretty much everybody knows them. On the other side of the spectrum are things that are impossible to figure out. These are mysteries, not secrets. Take superstring theory in physics, for instance. You can’t really design experiments to test it. Discovery is the process of exposing secrets. The secrets are dis- covered; the cover is removed from the secret. Triangle math was a hard secret for Pythagoras to discover. There were various Pythagorean mystery cults where the initiated learned about crazy new things like irrational numbers. But then his discovered secret became convention.

Forty or fifty years ago, everyone believed that there was much more left to do. But generally speaking, we no longer believe that. There is a sense though that there are not many important secrets left. It’s a plausible view. If it’s wrong, it’s not obviously wrong. To evaluate it, we must first understand why people don’t believe in secrets anymore.

The extreme representative of the conventional view is Ted Kaczynski, more infamously known as the Unabomber. He was a child prodigy. IQ of 167. A top student at Harvard. PhD in math from Michigan. Professor of math at UC Berkeley. But then he started a solo bombing campaign after becoming disenchanted with science and technology. He killed 3 people and injured 23 more. The victims included computer store owners, technical grad students, geneticists, etc. Finally he was found and arrested in 1996.

But in late 1995 the FBI didn’t really have a clue who or where the Unabomber was. Kaczynski had written a manifesto and anonymously mailed it to the press. The government gave the go-ahead to print it, hoping for a break in the case. That ended up working, as Kaczynski’s brother recognized the writing and turned him in. But more interesting than how Kaczynski was caught was the manifesto itself. It was basically a long, crazy anti-tech diatribe. The core of the argument was that you could divide human goals into three groups:

1. Goals that can be satisfied with minimal effort
2. Goals that can be satisfied with serious effort
3. Goals that are impossible to satisfy

It was the classic easy/hard/impossible trichotomy. Kaczynski argued that people are depressed because the only things left are (1) easy things or (3) impossible things. What you can do, even kids can do. But what you can’t do, even Einstein couldn’t do. So Kaczynski’s idea was to destroy technology, get rid of all bureaucracy and technical processes, and let people start over and work on hard problems anew. That, he thought, would be much more fulfilling.

Why has our society come to believe that there are no hard secrets left? It probably starts with geography. There are no real white spaces left on the map anymore. If you grew up in 18th century, there were still lots of unexplored places. You could listen to captivating stories about explorers and foreign adventures and, if you wanted, go become a real explorer yourself. This was probably true up through the 19th and early 20th centuries, when National Geographic still published tales of exotic, underexplored places.  But now you can’t really be an explorer anymore. Or at least it’s very hard to explore the unexplored. People have done it all already. Maybe there are something like 100 uncontacted tribes somewhere deep in the Amazon. Maybe they’d have something interesting to teach us. But maybe not. Either way, most people don’t seem to care much. The oceans remain unexplored in a fairly interesting way. The planet is 72% covered by oceans. Some 90% of the inhabited ocean is deep sea. There have been only about 200 hours of human exploration there. So oceans are the last big geographic piece that people aren’t really looking at. But that may be because the default assumption is right; there’s nothing terribly interesting there. Deep sea exploration simply lacks the magic of exploring new lands and continents.

The frontier of knowledge seems to have waned along with the geographical frontier. People are increasingly pessimistic about the existence of new and interesting things. Can we go to the moon? We’ve done that already. Mars? Impossible, many people say. What about chemistry? Can we identify oxygen? That’s been trivial since the 18th century. So what about finding new elements? That’s probably a fool’s errand. The periodic table seems pretty set. It may be impossible to discover anything new there. The frontier is closed. There is nothing left to discover. 

Four primary things have been driving people’s disbelief in secrets. First is the pervasive incrementalism in our society. People seem to think that the right way to go about doing things is to proceed one very small step at a time. Any secrets that we’re incentivized to discover are microsecrets. Don’t try anything too hard in the classroom; just do what’s asked of you a bit better than the others and you’ll get an A. This dynamic exists all the way up through pre-tenure. Academics are incented by volume, not importance. The goal is to publish lots of papers, each of which is, in practice at least, new only in some small incremental way.

Second, people are becoming more risk-averse. People today tend to be scared of secrets. They are scared of being wrong. Of course, secrets are supposed to be true. But in practice, what’s true of all secrets is that there is good chance they’re wrong. If your goal is to never make mistake in your life, you should definitely never think about secrets. Thinking outside the mainstream will be dangerous for you. The prospect of dedicating your life to something that no one else believes in is hard enough. It would be unbearable if you turned out to be wrong.

Third is complacency. There’s really no need to believe in secrets today. Law school deans at Harvard and Yale give the same speech to incoming first year students every fall: “You’re set. You got into this elite school. Your worries are over.” Whether or not such complacency is justified (and we should suspect it’s not), it’s probably the kind of thing that’s true only if you don’t believe in it. If you believe in it, you’re probably in a lot of trouble.

In defense of the case against secrets, distrusting prophets has become a good heuristic.

However, that hard problems do get solved is evidence that secrets exist. It’s not always straightforward to tell whether a given problem is merely hard or actually impossible. But the people who actually solve hard problems are people who believe in secrets. If you believe something is hard, you might still think you can do it. You’ll try things, and maybe you’ll succeed. But if you think something is impossible, you won’t even try. Fermat’s last theorem is a good example. It states that no three positive integersa, b, and c can satisfy the equation an + bn = cn for any n greater than two. Mathematician Andrew Wiles started working on it in 1986. He managed to prove it in 1995. No one would ever succeed in doing these incredibly hard things if they didn’t think that it was possible. In some sense you can’t have meaningful progress if you don’t think that there are solvable secrets out there.

via

"Physibles" and the Near-Future of Pirating Physical Objects

While the subject of online piracy is certainly nothing new, the recent protests against SOPA and the federal raid on Megaupload have thrust the issue into mainstream media. More than ever, people are discussing the controversial topic while content creators scramble to find a way to try to either shut down or punish sites and individuals that take part in the practice. Despite these efforts, online piracy continues to be a thorn in Big Media’s side. With the digital media arena all but conquered by piracy, the infamous site The Pirate Bay (TPB) has begun looking to the next frontier to be explored and exploited. According to a post on its blog, TPB has declared that physical objects named “physibles” are the next area to be traded and shared across global digital smuggling routes.

TPB defines a physible as “data objects that are able (and feasible) to become physical.” Namely, items that can be created using 3D scanning and printing technologies, both of which have become much cheaper for you to actually own in your home. At CES this year, MakerBot Industries introduced its latest model which is capable of printing objects in two colors and costs under $2,000. With the price of such devices continuing to drop, 3D printing is going to be part of everyday life in the near future. Where piracy is going to come in is the exchange of the files (3D models) necessary to create these objects.

A 3D printer is essentially a “CAD-CAM” process. You use a computer-aided design (CAD) program to design a physical object that you want made, and then feed it into a computer-aided machining (CAM) device for creation. The biggest difference is that traditional CAM setups, the process is about milling an existing piece of metal, drilling holes and using water jets to carve the piece into the desired configuration. In 3D printing you use extrusion to actually create what is illustrated in the CAD file. Those CAD files are the physibles that TPB is talking about, since they are digital they are going to be as easily transferred as an MP3 or movie is right now.


It isn’t too far outside the realm of possibility that once 3D printing becomes a part of everyday life, companies will begin to sell the CAD files and the rights to be able to print proprietary items. If the technology continues to advance at the same rate, in 10 or 20 years you might be printing a new pair of Nikes for your child’s basketball game right in your home (kind of like the 3D printed sneakers pictured above). Instead of going to the mall and paying $120 for a physical pair of shoes in a retail outlet, you will pay Nike directly on the internet and receive the file necessary to direct your printer to create the sneakers. Of course, companies will do their level best to create DRM on these objects so that you can’t freely just print pair after pair of shoes, but like all digital media it will be broken be enterprising individuals.

TPB has already created a physibles category on its site, allowing you to download plans to be able to print out such things as the famous Pirate Bay Ship and a 1970 Chevy hot rod. For now it’s going to be filled with user-created content, but in the future you can count on it being stocked with plans for DRM-protected objects.

via

My Definition of Art

Art is a symbol making visible the plane of potentiality present in all aspects of life.

There is no invisible hand

One of the best-kept secrets in economics is that there is no case for the invisible hand. After more than a century trying to prove the opposite, economic theorists investigating the matter finally concluded in the 1970s that there is no reason to believe markets are led, as if by an invisible hand, to an optimal equilibrium — or any equilibrium at all. But the message never got through to their supposedly practical colleagues who so eagerly push advice about almost anything. Most never even heard what the theorists said, or else resolutely ignored it.

Of course, the dynamic but turbulent history of capitalism belies any invisible hand. The financial crisis that erupted in 2008 and the debt crises threatening Europe are just the latest evidence. Having lived in Mexico in the wake of its 1994 crisis and studied its politics, I just saw the absence of any invisible hand as a practical fact. What shocked me, when I later delved into economic theory, was to discover that, at least on this matter, theory supports practical evidence.

Adam Smith suggested the invisible hand in an otherwise obscure passage in his Inquiry Into the Nature and Causes of the Wealth of Nations in 1776. He mentioned it only once in the book, while he repeatedly noted situations where "natural liberty" does not work. Let banks charge much more than 5% interest, and they will lend to "prodigals and projectors," precipitating bubbles and crashes. Let "people of the same trade" meet, and their conversation turns to "some contrivance to raise prices." Let market competition continue to drive the division of labor, and it produces workers as "stupid and ignorant as it is possible for a human creature to become."

In the 1870s, academic economists began seriously trying to build "general equilibrium" models to prove the existence of the invisible hand. They hoped to show that market trading among individuals, pursuing self-interest, and firms, maximizing profit, would lead an economy to a stable and optimal equilibrium.

Leon Walras, of the University of Lausanne in Switzerland, thought he had succeeded in 1874 with his Elements of Pure Economics, but economists concluded that he had fallen far short. Finally, in 1954, Kenneth Arrow, at Stanford, and Gerard Debreu, at the Cowles Commission at Yale, developed the canonical "general-equilibrium" model, for which they later won the Nobel Prize. Making assumptions to characterize competitive markets, they proved that there exists some set of prices that would balance supply and demand for all goods. However, no one ever showed that some invisible hand would actually move markets toward that level. It is just a situation that might balance supply and demand if by happenstance it occurred.

In 1960 Herbert Scarf of Yale showed that an Arrow-Debreu economy can cycle unstably. The picture steadily darkened. Seminal papers in the 1970s, one authored by Debreu, eliminated "any last forlorn hope," as the MIT theorist Franklin Fisher says, of proving that markets would move an economy toward equilibrium. Frank Hahn, a prominent Cambridge University theorist, sums up the matter: "We have no good reason to suppose that there are forces which lead the economy to equilibrium."

An engineering analogy may help. The invisible hand sees market economies as passenger planes, which, for all the miseries of air travel, are aerodynamically stable. Buffeted by turbulence, they just settle back into a slightly different flight path. General-equilibrium theory, as it developed in the 1960s and 1970s, suggests that economies are more like fighter jets. Buffeted by a gust, they wouldn't just settle into a slightly different path but would spin out of control and break asunder if "fly-by-wire" computer guidance systems did not continually redirect them to avert disaster.

Economists might call the fighter-jet analogy polemic, but no knowledgeable theorist would say that the so-called "general equilibrium" model is stable. The very word "equilibrium" is deeply misleading in this context because it describes a situation that is not an equilibrium, either in plain English or in engineering. Economic equilibrium — a stable state toward which an economy would move — reveals a hope on the part of economists, not a mechanism captured in an accepted model. Speaking of "equilibrium" allowed economists to fool themselves, and others.

The failure to model the invisible hand is ironically powerful. Any given economic model might well be implausible. But if the brightest economic minds failed for a century to show how some invisible hand could move markets toward equilibrium, can any such mechanism exist? Something outside markets — social norms, economic regulation, Ben Bernanke in his happier moments — must usually avert disaster.

How can some economic models continue to assume stability? Arrow-Debreu treats each individual, firm, and good as distinct. Supposedly practical economists develop models that aggregate — homogenize. They aggregate corn, iPods, and haircuts into one uniform quantity of stuff that they call "commodities" and label "Y." And they lump all diverse individuals into one "representative agent." You can easily build stability into such a model by pure assumption. But it is pure assumption. How could decentralized trading move markets to equilibrium if there is only one good?

In a tribute to academic insularity, most supposedly practical economists are dimly aware, if at all, of theorists' instability results. They might have briefly seen them in one theory course and ignored them as geeky and inconvenient. Others dismiss them. Milton Friedman once told Franklin Fisher he saw no point in studying the stability of general equilibrium because the economy is obviously stable — and if it isn't, "we are all wasting our time." Fisher quips that the point about economists' wasting their time was perceptive. The point about economies being obviously stable was not perceptive.

Believing far too credulously in an invisible hand, the Federal Reserve failed to see the subprime crisis coming. The principal models it used literally assumed that markets are always in instantaneous equilibrium, so how could a crisis occur? But after the crisis exploded, the Fed dropped its high-tech invisible-hand models and responded with full force to support the economy.

The powerful invisible-hand metaphor refused to die. It assured German Chancellor Angela Merkel, even if she grew up in East Germany under Communism, that slashing fiscal budgets and deregulating labor markets would end the euro crisis. Based on thinking dimmed by some invisible-hand fancy, European authorities have again and again been a day late and a euro short in responding to market gales. As a result, they made the euro crisis far worse than it had to be.

Via

From Father to Son

A letter to 16-year-old Jackson Pollock from his dad. Via Brain Picker. 


Dear Son Jack,

Well it has been some time since I received your fine letter. It makes me a bit proud and swelled up to get letters from five young fellows by the names of Charles, Mart, Frank, Sande, and Jack. The letters are so full of life, interest, ambition, and good fellowship. It fills my old heart with gladness and makes me feel ‘Bully.’ Well Jack I was glad to learn how you felt about your summer’s work & your coming school year. The secret of success is concentrating interest in life, interest in sports and good times, interest in your studies, interest in your fellow students, interest in the small things of nature, insects, birds, flowers, leaves, etc. In other words to be fully awake to everything about you & the more you learn the more you can appreciate & get a full measure of joy & happiness out of life. I do not think a young fellow should be too serious, he should be full of the Dickens some times to create a balance.

I think your philosophy on religion is okay. I think every person should think, act & believe according to the dictates of his own conscience without too much pressure from the outside. I too think there is a higher power, a supreme force, a governor, a something that controls the universe. What it is & in what form I do not know. It may be that our intellect or spirit exists in space in some other form after it parts from this body. Nothing is impossible and we know that nothing is destroyed, it only changes chemically. We burn up a house and its contents, we change the form but the same elements exist; gas, vapor, ashes. They are all there just the same.

I had a couple of letters from mother the other day, one written the twelfth and one the fifteenth. Am always glad to get letters from your mother, she is a Dear isn’t she? Your mother and I have been a complete failure financially but if the boys turn out to be good and useful citizens nothing else matters and we know this is happening so why not be jubilant?

The weather up here couldn’t be beat, but I suppose it won’t last always, in fact we are looking forward to some snowstorms and an excuse to come back to the orange belt. I do not know anything about what I will do or if I will have a job when I leave here, but I am not worrying about it because it is no use to worry about what you can’t help, or what you can help, moral ‘don’t worry.’

Write and tell me all about your schoolwork and yourself in general. I will appreciate your confidence.

You no doubt had some hard days on your job at Crestline this summer. I can imagine the steep climbing, the hot weather, etc. But those hard things are what builds character and physic. Well Jack I presume by the time you have read all this you will be mentally fatigued and will need to relax. So goodnight, pleasant dreams and God bless you.

Your affectionate Dad

Bitching Monks

Via Brain Pickings, here's a wonderful collection of complaints monks scribbled in the margins of the illuminated manuscripts they were writing. Seems writing has always sucked. Pretty funny stuff. 

Interaction Design is what?

Me + Brutalism + Corpses = AIGA/NY Talk April 5th

The brave folks at the AIGA/NY have asked me to attempt a talk on April 5th at the Museum of Design. You can see the details here. If you've ever been to any of my talks before, you know that I always try to do the speaker's equivalent of spinning 20 plates with 20 sticks. So that's always fun.

Hope to see you out there and let's hope I don't break any plates.

Oh, almost forgot: if you're gonna be in Chicago in May 1- 2, I'll be speaking at the Gravity Free conference as well. Maybe I'll see you there as well.

Designing the New Business

From information management to medium of relationships

"We need to shift away from the notion of technology managing information and toward the idea of technology as a medium of relationships."

 Michael Schrage, Shared Minds

When Creating, Go Solo.

In his memoir, Steve Wozniak (co-founder of apple and inventor of its products) offers this guidance to aspiring inventors:

“Most inventors and engineers I’ve met are like me ... they live in their heads. They’re almost like artists. In fact, the very best of them are artists. And artists work best alone .... I’m going to give you some advice that might be hard to take. That advice is: Work alone... Not on a committee. Not on a team.”

Virtually all American workers now spend time on teams and some 70 percent inhabit open-plan offices, in which no one has “a room of one’s own.” During the last decades, the average amount of space allotted to each employee shrank 300 square feet, from 500 square feet in the 1970s to 200 square feet in 2010.

Studies show that open-plan offices make workers hostile, insecure and distracted. They’re also more likely to suffer from high blood pressure, stress, the flu and exhaustion. And people whose work is interrupted make 50 percent more mistakes and take twice as long to finish it.

Many introverts seem to know this instinctively, and resist being herded together. Backbone Entertainment, a video game development company in Emeryville, Calif., initially used an open-plan office, but found that its game developers, many of whom were introverts, were unhappy. “It was one big warehouse space, with just tables, no walls, and everyone could see each other,” recalled Mike Mika, the former creative director. “We switched over to cubicles and were worried about it — you’d think in a creative environment that people would hate that. But it turns out they prefer having nooks and crannies they can hide away in and just be away from everybody.”

Privacy also makes us productive. In a fascinating study known as the Coding War Games, consultants Tom DeMarco and Timothy Lister compared the work of more than 600 computer programmers at 92 companies. They found that people from the same companies performed at roughly the same level — but that there was an enormous performance gap between organizations. What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed. Sixty-two percent of the best performers said their workspace was sufficiently private compared with only 19 percent of the worst performers. Seventy-six percent of the worst programmers but only 38 percent of the best said that they were often interrupted needlessly.

Solitude can even help us learn. According to research on expert performance by the psychologist Anders Ericsson, the best way to master a field is to work on the task that’s most demanding for you personally. And often the best way to do this is alone. Only then, Mr. Ericsson told me, can you “go directly to the part that’s challenging to you. If you want to improve, you have to be the one who generates the move. Imagine a group class — you’re the one generating the move only a small percentage of the time.”

Conversely, brainstorming sessions are one of the worst possible ways to stimulate creativity. The brainchild of a charismatic advertising executive named Alex Osborn who believed that groups produced better ideas than individuals, workplace brainstorming sessions came into vogue in the 1950s. “The quantitative results of group brainstorming are beyond question,” Mr. Osborn wrote. “One group produced 45 suggestions for a home-appliance promotion, 56 ideas for a money-raising campaign, 124 ideas on how to sell more blankets.”

But decades of research show that individuals almost always perform better than groups in both quality and quantity, and group performance gets worse as group size increases. The “evidence from science suggests that business people must be insane to use brainstorming groups,” wrote the organizational psychologist Adrian Furnham. “If you have talented and motivated people, they should be encouraged to work alone when creativity or efficiency is the highest priority.”

The Emory University neuroscientist Gregory Berns found that when we take a stance different from the group’s, we activate the amygdala, a small organ in the brain associated with the fear of rejection. Professor Berns calls this “the pain of independence.”

The one important exception to this dismal record is electronic brainstorming, where large groups outperform individuals; and the larger the group the better. The protection of the screen mitigates many problems of group work. This is why the Internet has yielded such wondrous collective creations. Marcel Proust called reading a “miracle of communication in the midst of solitude,” and that’s what the Internet is, too. It’s a place where we can be alone together — and this is precisely what gives it power.

Most humans have two contradictory impulses: we love and need one another, yet we crave privacy and autonomy. To harness the energy that fuels both these drives, we need to move beyond the New Groupthink and embrace a more nuanced approach to creativity and learning. Our offices should encourage casual, cafe-style interactions, but allow people to disappear into personalized, private spaces when they want to be alone.

via

The Future of Cities

Kevin Kelly's New Rules for the New Economy

Kevin's put together a nice list of principles for networked economy. The summative idea is one I've argued that commons create but never been able to articulate so concisely: "Opportunities over efficiencies." It draws a nice distinction between an the main strategy for growth in an open/commons economy (the generation of opportunities) and a production economy (the enhancement of efficiencies). Here are some of my favorite points Kevin makes in his list:


Principle #3: Where humans are most actively engaged with their imaginations, we don't see productivity gains--and why would we? 

"Is a Hollywood movie company that produces longer movies per dollar more productive than one that produces shorter movies? Yet an increasingly greater percentage of work takes place in the information, entertainment, and communication industries where the 'volume' of output is somewhat meaningless.

"The problem with trying to measure productivity is that it measures only how well people can do the wrong jobs. Any job that can be measured for productivity probably should be eliminated from the list of jobs that people do.

"The task for each worker in the industrial age was to discover how to do his job better: that's productivity. Frederick Taylor revolutionized industry by using his scientific method to optimize mechanical work. But in the network economy, where machines do most of the inhumane work of manufacturing, the question for each worker is not 'How do I do this job right?' but 'What is the right job to do?'

"Answering this question is, of course, extremely hard to do. It's called an executive function. In the past, only the top 10% of the workforce was expected to make such decisions. Now, everyone, not just executives, must decide what is the right next thing to do."


Principle #4: Productivity, however, is exactly the wrong thing to care about in the new economy.

"To measure efficiency you need a uniform output. But uniform output is becoming rarer in an economy that emphasizes smaller production runs, total customization, personalized 'feelgoods' and creative innovation. Less and less is uniform.

"And machines have taken over the uniform. They love tedious and measurable work. Constant upgrades enable them to churn out more per hour. So the only ones who should worry about their own productivity are those made of ball bearings and rubber hoses. And, in fact, the one area of the current economy that does show a rise in productivity has been the U.S. and Japanese manufacturing sectors, which have seen an approximately 3% to 5% annual increase throughout the 1980s and into the 1990s. This is exactly where you want to find productivity. Each worker, by supervising machinery and tools, produces more rivets, more batteries, more shoes, and more items per person-hour. Efficiencies are for robots.

"Opportunities, on the other hand, are for humans. Opportunities demand flexibility, exploration, guesswork, curiosity, and many other qualities humans excel at. By its recursive nature, a network breeds opportunities, and incidentally, jobs for humans."


Principle #6: Don't solve problems; pursue opportunities.

"Seeking opportunities is no longer wisdom relevant only to the long cycles of economic progress. As the economy speeds up, so that an "internet year" seems to pass in one month, the principles of long-term growth begin to govern the day-to-day economy. The dynamics of growth become the dynamics of short-term competitive advantage.

"In both the short and long term, our ability to solve social and economic problems will be limited primarily to our lack of imagination in seizing opportunities, rather than trying to optimize solutions."


Principle #7: As the transmission of knowledge accelerates, as more possibilities are manufactured, the unabated push of incremental growth also speeds up. In the long run, creating and seizing opportunities is what drives the economy. A better benchmark than productivity would be to measure the number of possibilities generated by a company or innovation and use the total to evaluate progress.

"In the short run, though, problems must be solved. Businesses are taught that they are in the business of solving problems. Put your finger on a customer's dissatisfaction, the MBAs say, and then deliver a solution. This bit of hoary advice inspires business to seek out problems. Problems, however, are entities that don't work. They are usually situations where the goal is clear but the execution falls short. As in, 'We have a reliability problem,' or 'Customers complain about our late delivery.' In the words of Peter Drucker, 'Don't solve problems.' George Gilder distills the essence further: 'When you are solving problems, you are feeding your failures, starving your successes, and achieving costly mediocrity. In a competitive global arena, costly mediocrity goes out of business.'"


Princple #10: We can rearrange more than just bits.

"Think of the mineral iron oxide, suggests Romer. It's rust. More than 10,000 years ago our ancestors used iron oxide as a pigment to make art on cave walls. Now, by rearranging those same atoms into a precisely thin iron oxide film on plastic we get a floppy disk, which can hold a reproduction of the same cave paintings, and all the possible permutations of it wrought by Photoshop. We have amplified the possibilities a millionfold.

"The power of combinatorial explosions--which is what you get with ideas and opportunities--means, says Romer, 'There's essentially no scarcity to deal with.' Because the more you use opportunities, the less scarce they get.

"Everything we know about the structure of the network economy suggests that it will bolster this efflorescence of opportunities, for the following reasons:

"Every opportunity inhabits a connection. As we connect up more and more of the world into nodes on a network, we make available billions more components in the great combinatorial game. The number of possibilities explodes. Networks speed the transmission of opportunities seized and innovations created, which are disseminated to all parts of the network and the planet, inviting more opportunities to build upon them. Technology is no panacea. It will never solve the ills or injustices of society. Technology can do only one thing for us--but it is an astonishing thing: Technology brings us an increase in opportunities."

Twitter is Killing Its Ecosystem - The Very Thing That Makes It Valuable

Though it's a year old, this analysis of Twitter's misstep in not playing nice with partners is still an excellent and relevant read:

At O'Reilly, we've long believed that one criterion for long-term success is creating more value than you capture. When you do so, you create the possibility for an ecosystem that's larger than you are. You create a healthy environment in which you, your partners, and even your competitors can thrive.

Twitter, long one of my favorite companies, has turned the corner. After creating a very healthy ecosystem of third-party apps (standalone, web, and mobile), they've decided they need to shut down the market. In a post to Twitter's developer forum, Ryan Sarver (@rsarver) clarifies Twitter's terms of service and says new Twitter clients are unwelcome and existing clients ought to watch their backs. Further down in the thread, Raffi Krikorian (@raffi) steps in and does some damage control. While Raffi does a good job of moderating Ryan's heavy-handed statements, I don't think the position changes much. "No" becomes "it's a bad idea to create a business where you would have to bend at the whims of another organization." Exactly. When you create an ecosystem, you have a responsibility to that ecosystem. Perhaps it's an ethical responsibility rather than a business responsibility. But that ecosystem is as much responsible for your growth as you are for its, and subjecting it to your whims isn't prudent.

I'm personally saddened, at least in part because I've used Twitter as an example of how to build a successful data ecosystem. It's a system that has contributed substantially to Twitter's success, in a way that can, and should, be emulated. If you build the ecosystem successfully, the opportunities will come.

But now, Twitter has decided that it would rather not compete with its many children. Why? The reason given is that consumers are confused, and Twitter needs a consistent user experience. This reasoning strikes me as wrong-headed. Consistent user experience is a red herring. Is anyone confused by the diversity of user experience that's out there now? It's a throwback to the time when Word files only worked with Microsoft Word and WordPerfect files only worked with WordPerfect. But that was a pre-web world, and that's not how the web works — never has been. If that's what the future holds, we've got much bigger problems to worry about than net neutrality.

Are users stupid, silly, weak creatures who are easily misled? I don't think so. As I've argued elsewhere, the proliferation of bad software guarantees that we'll get some good software and occasional brilliant software. Twitter is ultimately a data publisher — and the many ways that data can be used is what makes the data publishing business interesting.

Twitter states that 90% of their users use an "official" Twitter client at least monthly. I'm sure that's true, but it's also misleading. I easily use a dozen Twitter clients, including clients built into other apps. While I use an "official" client daily, it's probably under 10% of my total use. And that's certainly part of the problem. I'm sure that Twitter is feeling pressure to find a viable business model to justify billions of dollars of valuation and hundreds of millions of investor capital. Promoted trends, tweets, and follower recommendations might be viable, though my reaction is often "You're kidding; why would you even think I'd want to follow X?" Offered a choice, I'd rather not see them — and the clients I prefer don't force me to see them. Is that the real issue? Other clients aren't displaying "promoted" topics or users? Other clients give users choices that Twitter would rather they don't have?

Are other business models for Twitter conceivable? Of course. In Twitter's early days, I suggested to Blaine Cook (@blaine) that supporting private corporate Twitter networks was a possibility — a route that was latter taken by Yammer, and where there's probably still a significant opportunity. Selling the "firehose" is also a possibility, and something that's already happening (with Gnip the preferred reseller). Analytics services, facilitating customer service — these are all legitimate opportunities for Twitter. I've been surprised that they haven't gone after them.

I am not saying that Twitter has no right to enforce some rules on the playing field it has created. Specifically:

• I have no objection to Twitter enforcing rules protecting the privacy of users, or restricting the proliferation of spam and malicious content. A client that edits tweets, changes profiles, or does anything else without the user's permission should expect to be barred from the service. 
• I also have no particular problem with Twitter competing with its ecosystem. There was some angst about Twitter "eating its children" when it rolled out the "new Twitter." Tough. For a long time, Twitter's own clients were sub-standard. Twitter has every right to build the best client they can, and to get as many users as possible to use it, as long as they maintain a level playing field. They only cross the line when they say "we won't tolerate any more competition." 
• And I have no problem with Twitter saying that there are perhaps too many clients (I wouldn't disagree), and suggesting areas in which developers who want to do something unique can experiment. Sarver offers a few such areas: publisher tools, tweet curation, realtime data analysis, etc. But here's the rub: if you're building publisher tools or selling analytics data, what's to prevent Twitter from deciding they want that business three months from now, and shutting down the competition? Given the way Twitter has behaved toward its client ecosystem, I would certainly be nervous about starting an analytics business based on Twitter data, or funding such a business. Will the rug be pulled out from under me a few months down the road? I don't know. Quora analytics, anyone?

But while Twitter can enforce rules on their playing field, they shouldn't say it's our field, it's our ball, everyone else go home. And while there are plenty of uninteresting, dull clients, there is plenty of possibility for innovation. A year or so ago, I saw a Twitter client that displayed all incoming tweets on a map. Last night, I came up with several ideas for interesting, useful clients: threaded clients, clients that display tweets in the context of the user's social graph, clients that use artificial intelligence techniques to find tweets that will be relevant and interesting. (Sort of like my my6sense, but for Twitter.) There's no shortage of ideas. As Raffi says, developers ought to think big, not just come up with dull clones that simply display timelines. But he and Twitter are missing a fundamental law of innovation: you can't tell people where (or how) to innovate, and where not to. Innovation just doesn't work that way. The best way to prevent "think big" innovation from happening is to cut off the small ideas.

Twitter's limitations on research (specifically, whitelisting and redistribution of raw data) are also troubling. Using social sites as a primary source for research is an important trend that we've been watching and publishing on, particularly in 21 Recipes for Mining Twitter and Mining the Social Web. While I understand Twitter's restrictions on redistributing data for commercial purposes, and perhaps a desire to prune the number of users who can pull a large datafeed, it seems — as Inside Higher Ed argues — that these changes hurt academic research more than anything else. Twitter has been an enormously useful channel for finding out how history unfolds. Gephi.org has a wonderful visualization of the progress of the Egyptian revolution through Twitter tweets, and I've seen much similar work. Is this kind of research still possible, without buying a datafeed that a student or an underfunded department might find prohibitively expensive?

Again, I see no problem with Twitter restricting commercial access. If you're making money from the Twitter stream, you shouldn't expect the stream to be provided as a free service. It isn't difficult to write terms of service that distinguish between research and commercial use. Pure research is symbiotic, not parasitic: that research is one reason that Twitter has its multi-billion dollar valuation. In addition to informing us, it demonstrates what you can do with the Twitter timeline, and shows opportunities that Twitter (and others) can cash in on. We're only now realizing what Twitter can teach us. Is it time to shut down further experimentation?

We have seen many companies thrive by creating more value than they capture. Twitter was one of the best. But once you've given something away, it's hard to take it back, particularly if it's something as fundamental as the right to create innovative software or to do academic research. Has Twitter said that developers can't innovate? No, but they've demonstrated that they might tell you your innovation is out of bounds somewhere down the road. It's disappointing to see Twitter undermining the ecology that made it valuable in the first place. Twitter would not be the company it is today without Tweetie, TweetDeck, TweetGrid, and many, many others. Likewise, Twitter won't become the company it could be if it cuts off scholars' ability to learn from its timeline. It remains to be seen whether a company that captures more value than it creates will inevitably stagnate. But if so, Twitter will be an unfortunate lesson, and we'll all be the poorer.