The marketplace of ideas

Since I started paying attention to the tech industry in the 1990's certain patterns have emerged.

  1. Hot new topic arrives.
  2. Lots of startup activity around hot topic.
  3. The big boys start getting into hot topic (and tend to be bad at it).
  4. The big boys start buying startups (in order to become good at it).
  5. One or more of the startups becomes a big boy themselves.
  6. The big companies prospect fresh startups for good ideas and buy the companies for their tech.
In the end, what starts as a flurry of creativity in independent directions ends up in a MUCH more consolidated market once said market matures. There are still smaller startups around, but all of the big boys have worthwhile products, and they're the ones with the marketing muscle and application-suites to bundle them in.

The same kind of marketplace seems to apply to networking ideas, though the timescale is longer. Back when spam started REALLY annoying A LOT of people, the topic was what to do about spam. The fundamental problem is that the SMTP protocol and the associated "how to send mail on the internet" RFCs don't have any kind of trust model built into them, they're purely a transport protocol. Garbage goes in, just as much garbage comes out the other side. It's efficient that way. To those trying to keep a clean mailbox this is a problem.

Many, many, many ideas were floated for how to patch SMTP, or otherwise introduce some kind of trust or costing model that would provide a meaningful disincentive to sending 1.5 kajillion emails. Most of these are still around in some form (even in the above list, when an industry gets to stage 6 there are still weeny stage-2 startups poking around with just enough of a customer base to remain viable), though not as widely deployed as they once were.

  • Stamps for e-mail! Make the US Post Office provide a $0.05 per email "stamp" to certify its from a real person! 1.5 kajillion emails become really expensive then! Win!
  • Central post-office certified mailers. Each mailer in a country has to be certified by their Postal Authority. When sending mail, pass your authority token so the receiving mailer can check it out. Mass mailers on Deutche Telecom dial-up won't be certified and can't send 1.5 kajillion messages. Win!
  • Mandate secure signatures. Email not signed by PGP/S-MIME? Toss it. Win!
  • Email Captcha. Sending an email to someone new? You have to reply to an automatic message (or click a web-link) before it'll consider you 'real' and pass your message. Sorry. Anyone sending 1.5 kajillion messages will in no way be able to reply to 1.5 kajillion of those. Win!
  • Keyword searching. Some topics aren't valid in work email. Toss 'em. Win!
  • Bayesian filtering. Dynamically figure out what's spam and what's not based on how a Real Live Person ranks incoming mail. Once it knows how you roll, all spam will go away. Win!
  • Address book registry! All people have to be certified as a real, and where they send mail from. Some skeevy Slovenian impersonating the CFO won't be able to do that. Win!
  • Sender Policy Framework! Entities list the IP addresses that send their mail. Anything not coming from there can be tossed. Win!
  • IP Reputation! The current king. Keep track of what IPs send bad mail. Then blacklist them.
What I and others have noticed is that how people access the Internet is undergoing a similar evolution. Until the wireless broadband revolution, Internet access was pretty much done from general-purpose computing devices known as 'desktop' and 'laptop' with occasional forays into special-purpose devices like 'webTV'. Now that the wires have gone away, things have shrunk again and you've got smartphones and tablets in the mix.

Unlike how the initial Internet developed, the companies pushing out the new phones and tablets have realized that providing central control of that experience can avoid some of the pitfalls of a fully open infrastructure. This is a new interaction model for people and uptake has been strong. To me this suggests that we're entering the 'consolidation' phase of Internet access, and this is where the fight gets real.

We already have entire countries attempting to regulate who can go where on the Internet, and now we have large companies doing the same thing. Unlike countries, at the moment people are opting into central control by these companies. Things Just Work, which is very attractive. To those who look at the innovation that the fully open Internet provided, this sort of curated access is very threatening.

That said, innovation won't stop, it'll just get subsumed into the big companies. Once stage 5 is hit, the creative froth starts dying down. Or rather, what generated the froth is still there, it just doesn't engender new companies at the same rate; the existing companies prospect for new ideas and glomb onto them a lot faster than they did in stage 2.

For a GREAT visual of this process, NASA has an excellent video shot on the ISS: link

Much like the consolidation list I started this post with, you'll still get a steady stream of startup ideas, but there will be some downward pressure from the central authorities. Much hay has been made over what Apple will and won't let into the App Store, which is a great case in point.

Will curated-computing replace the open-access computing we've had until now? I think it'll replace a large part of the market, but not all of it. Things Just Work, and that's a powerful driver. People don't mind closed access so long as anything they want to do is available in the box with them. This is why Apple keeps promoting the number of apps in the App Store, and why Android, Blackberry, and HP Palm do the same thing for their application storefronts.

Once access narrows down into a small group of companies, centralized control becomes a LOT easier. The Internet has been transformative because it's so decentralized it's tricky to lock down. Places like China and Iran do so with limited effect. Having a few smaller targets to aim laws at (Blackberry & India for example) makes exerting territorial control markedly easier. It is my opinion that this will provide a counter-incentive for the upkeep of these curated technologies.

Unfortunately, we'll have to wait and see how the market settles out.