With On Intelligence, I find myself in the unique position of having heavily evangelized a book before I’ve even finished it. I read half of it and started buying copies for friends. This is something I’ve never done before, so if you’re busy, you can take a quick tl;dr, and assume that if you’re interested in how intelligence works, namely how the brain functions at a high level (learning patterns, predicting the future, forming invariant representations of things) and how we might functionally simulate that with computers, do not pass go, do not collect $200, go buy a copy (Amazon, Powells) and read it.
Still here? Good, because I have a lot to say. This isn’t really a book review, it’s more of a book summary and an exhortation to activity. You’ve been warned.
A Little Backstory
Earlier this year I went to OSCON, and at OSCON the keynote that impressed me the most was by Jeff Hawkins, creator of the PalmPilot and founder of HandSpring. Here’s the video:
As appropriate for an Open Source conference, Jeff’s company, Numenta was announcing that they were open sourcing their neocortical simulator library, NuPIC, and throwing it out there for people to hack on. NuPIC was based on the work Numenta had done on neocortical simulations since he wrote the book, On Intelligence, in 2005. NuPIC is software that simulates the neocortex, the sheet of grey matter on the outside of your brain where all your experiences live. 3 years of French? It’s in the neocortex. The ability to figure out that two eyes and a nose equals a face? The neocortex. The neocortex even has the ability to directly control your body, so that muscle memory you rely on to do that thing you do so well, like riding a bike or painting or driving a car? That’s all in your neocortex. It’s the size of a large dinner napkin (the largest in humans, but every mammal has one), is about as thick as 7 business cards, and wraps around the outside of your head. It is you.
Intrigued, I went to the full length session that the Numenta team presented…
One of their main demos was an electrical consumption predictor for a gymnasium. When initialized, the NuPIC system is empty, like a baby’s brain. Then you start to feed it data, and it starts to try to predict what comes next. At first, its predictions fall a little behind the data it’s receiving, but as the days of data go by, it starts to predict future consumption an hour out (or whatever you’ve configured), and it gets pretty good at it. Nobody told NuPIC what the data was, just like our DNA doesn’t tell our brains about French verbs, the structure is there and with exposure it gets populated and begins to predict.
At the end of the talk, their recommendation for learning more about this stuff was to read On Intelligence. So, eventually, that’s what I did.
A Little Hyperbole
The simulation, in software and silicon, of the biological data handling processes, and building software off of that simulation, is the most interesting thing I’ve seen since Netscape Navigator. Everything up to your iPhone running Google Maps is progressive enhancement and miniaturization of stuff I’ve seen before. Building brains feels different.
I have a Newton Messagepad 2000 around here somewhere. It had mobile email over packet radio with handwriting recognition in 1997. In 2001 I was using a cell phone with a color screened to look up directions and browse web sites in Japan. It’s all iterating, getting better bit by bit, so that when we look back in 10 years we think that we’ve made gigantic leaps. Have we? Maybe, but software is still stubbornly software-like. If I repeat the same error 10 times in a row, it doesn’t rewrite itself. My computer doesn’t learn about things, except in the most heavy handed of ways.
Looking beyond the game space, a few weeks ago I was talking with a large networking company about some skunkworks projects they had, and one of them was a honey pot product for catching and investigating hack attempts. The connections between deep simulations like Dwarf Fortress or the AI Storyteller in Rimworld and how a fake sysadmin in a honey pot should react to an intruder are obvious. If it’s all scripted and the same, if the sysadmin reboots the server exactly 15 seconds after the attacker logs in, it’s obviously fake. For the product to work, and for the attacker to be taken, it has to feel real, and in order to fool software (which can pick up on things like that 15 second timer), it has to be different every time.
One thing that these procedural and emergent systems have in common is that they aren’t rigidly structured programs. They are open to flexibility, they are unpredictable, and they are fun because unexpected things happen. They’re more like a story told by a person, or experiencing a real lived-in world.
I believe that to do that well, to have computers that surprise and delight us as creators, is going to require a new kind of software, and I think software like Numenta’s NuPIC neocortical simulator is a huge step in that direction.
Let’s Deflate That a Bit
Ok, so NuPIC isn’t a whole brain in a box. It’s single threaded, it’s kind of slow to learn, and it can be frustratingly obtuse. One of the samples I tried did some Markov chaining style text prediction, but since they fed each letter into the system as a data point instead of whole words, the system would devolve into returning ‘the the the the the’, because ‘the’ was the most common word in the data set I trained it with.
Neocortical simulators are a new technology in the general developer world. We’ve had brute force data processing systems like Hadoop, methods developed to deal with the problems of the Google’s of the world, and now we have NuPIC. The first steps towards Hadoop were rough, the first steps towards neocortical simulators are going to be rough.
It’s also possible that we’re entering another hype phase, brought on by the rise of big data as the everywhere-buzzword. We had the decades of AI, the decade of Expert Systems, the decade of Neural Networks, but without a lot to show for it. This could be the decade of the neocortex, where in 10 years it’ll be something else, but it’s also possible that just like the Web appeared once all the pieces were in place, the age of truly intelligent machines could be dawning.
Oh, This Was a Book Review?
It’s hard to review On Intelligence as a book, because how well it’s written or how accessible the prose may be is so much less important than the content. Sandra Blakeslee co-wrote the book, and undoubtedly had a large hand in hammering Jeff’s ideas into consumable shape. It isn’t an easy read due to the ideas presented, but it’s fascinating, and well worth the effort.
In the book Jeff describes the memory-prediction framework theory of the brain. The theory essentially states that the neocortex is a big non-specialized blob that works in a standard, fairly simple way. The layers in the sheet of the neocortex (there are 7 of them), communicate up and down, receiving inputs from your sensory organs, generalizing the data they get into invariant representations, and then pushing predictions down about what they will receive data about next. For instance, the first layer may get data from the eye and say, there’s a round shape here, and a line shape next to it. It pushes ’round shapes’ and ‘line shapes’ up to the next level, and says “I’ll probably continue to see round shapes and line shapes in the future”. The bouncing around of your natural eye movements gets filtered out, and the higher levels of the brain don’t have to deal with it. The next level up says, “This kind of round shape and line shapes seem to be arranged like a nose, so I’m going to tell the layer up from me that I see a nose”. The layer up from that gets the ‘I see a nose’ and two ‘I see an eye’ reports and says, “Next layer up, this is a face”. If it gets all the way to the top and there’s no mouth, which doesn’t match the invariant representation of ‘face’, error messages get sent back down and warning flags go off and we can’t help but stare at poor Keanu…
These layers are constantly sending predictions down (and across, to areas that handle other related representations) about what they will experience next, so when we walk into a kitchen we barely notice the toaster and the microwave and the oven and the coffee maker, but put a table saw onto the counter and we’ll notice it immediately.
As we experience things, these neurons get programmed, and as we experience them more, the connections to other things strengthen. I figure this is why project based learning works so much better than rote memorization, because you’re cross connecting more parts of your brain, and making it easier for that information to pop up later. Memory palaces probably work the same way. (I’m also half way through Moonwalking with Einstein, about that very thing.)
So, Have We Mentioned God Yet?
This is where things start to get weird for me. I grew up in a very religious family, and a large part of religion is that it gives you an easy answer to the ‘what is consciousness’ question when you’re young. Well, God made you, so God made you conscious. You’re special, consciousness lets you realize you can go to heaven, the dog isn’t conscious and therefor can’t, etc.
About a third of the way into On Intelligence I started having some minor freakouts, like you might have if someone let you in on the Truman Show secret. It was like the fabric of reality was being pulled back, and I could see the strings being pulled. Data in, prediction made, prediction fulfilled. Consciousness is a by-product of having a neocortex. (Or so Jeff postulates at the end of the book.) You have awareness because your neocortex is constantly churning on predictions and input. Once you no longer have predictions, you’re unconscious or dead, and that’s that.
That’s a heavy thing to ponder, and I think if I pondered it too much, it would be a problem. One could easily be consumed by such thoughts. But it’s like worrying about the death of the solar system. There are real, immediate problems, like teaching my daughter how stuff (like a Portal Turret) works.
Let’s Wrap This Thing Up With A Bow
I’m sorry this post was so meandering, but I really do think that neocortical simulators and other bio processing simulations are going to be a huge part of the future. Systems like this don’t get fed a ruleset, they learn over time, and they can continue to learn, or be frozen in place. Your self-driving car may start with a car brain that’s driven simulated (Google Street View) roads millions of miles in fast-forward, and then thousands of miles in the real world. Just like everyone runs iOS, we could all be running a neocortex built on the same data. (I imagine that really observant people will be able to watch Google’s self driving cars and by minor variations in their movements, tell what software release they’re running.) Or we could allow ours to learn, adjust its driving patterns to be faster, or slower, or more cautious.
The power of software is that once it is written, it can be copied with nearly no cost. That’s why software destroys industries. If you write one small business tax system, you can sell it a million times. If you grow a neocortex, feed it and nurture it, you’ve created something like software. Something that can be forked and copied and sold like software, but something that can also continue to change once it’s out of your hands. Who owns it? How can you own part of a brain? Jeff writes in the book about the possibilities of re-merging divergent copies. That’s certainly plausible, and starts to sound a whole lot like what I would have considered science fiction 10 years ago.
We could throw up our hands and say we’re but lowly developers, not genius computer theorists or doctors or what have you. The future will come, but all we can do is watch. The problem with that is that Google’s problems will be everyone’s problems in 5 years, so for all the teeth gnashing about Skynet and Bigdog with a Google/Kurzweil brain, it’s much more productive to actually get to work getting smarter and more familiar with this stuff. I wouldn’t be surprised if by 2020 ‘5+ Years Experience Scaling Neocortical Learning Systems in the Cloud’ was on a lot of job postings. And for the creative, solving the problem of how the Old Brain’s emotions and fears and desires interfaces with the neocortex should be rife with experimental possibilities.
Here’s a video from the Goto conference where Jeff talks about the neocortex and the state of their work. This video is from October 1st of 2013, so it’s recent. If you have an hour, it’s really worth a watch.
Daniel Squarez‘s latest techno-thriller Kill Decision isn’t a happy book. It’s an especially unhappy book if you’re excited about quadcopters, RC planes, self-organizing swarm AI, or any of that neat, fun stuff.
Daniel’s first published book was Daemon, a novel about a programmer who, upon discovering that his time is up, creates a distributed dumb-agent network of actions and actors triggered by reports in news feeds. The thing that made Daemon so interesting wasn’t just that concept, it was that Daniel has a really good grasp on the technology, so everything that happened in the book kind of made sense. There was no magic bullet, it was all ‘oh, yea, that could work’.
Kill Decision is a book about drones, specifically autonomous drones that can kill. It was only a few years ago that I remember wondering when someone was going to strap a handgun (even a fake one) to a quadcopter and attempt a robbery by drone. Kill Decision is a book about just that, except the handgun is quadcopter optimized and the person getting robbed is the USA.
It’s been a while since I’ve read any popular techno-thrillers, but from what I remember, Kill Decision follows the arc pretty well. There’s a tough soldier type, a naive but smart audience proxy, a team of good guys for gun fodder, and a big bad. The pacing is good, the details are good, and the book keeps you guessing. I guess my only complaint is also the books point, that in the end, with a robot that can kill, it’s really hard to figure out who the bad guy is. In Kill Decision there isn’t a Snidley Whiplash twirling his mustache just off stage, at least that we get to see, and that lack of a direct villain gives the book a feeling of existential angst. The bots just keep coming, and in the end, there isn’t a clear win or loss.
Lots of thrillers are spy novels with more gadgets. They’re Jason Bourne, a lone operative outwitting the watchful, ever-present eye of big evil. It’s a big data dream, outwitting the system. Kill Decision is different. Kill Decision is a zombie novel, except the zombies are cheap, deadly, swarming technology.
Of the thousands of pictures I’ve taken since I got into photography, there are only a few on display in my house. Only one of them is what you might call professionally framed. It’s that one, to the right. It was taken in Marken, Netherlands, on the Wandelroute Rond Marken Over de Dijk. Not exactly here, but close by, on a little path at the edge of an island next to the ocean. The thing is, it isn’t a photograph. It looks like a photograph, but it’s actually a panorama, digitally spliced together from half a dozen shots. It’s a photograph, re-interpreted by software. And it could be the first step on the road to something new.
Ode to a Camera Gathering Dust
A few weeks ago I read a blog post by Kirk Tuck talking about the recent drop in camera sales, and the general decline of photography as a hobby. Kirk’s assertion was that when a lot of us got into photography, gear made a big difference. There was the high end to yearn for, but with the right skill and tricks you could make up for it. There were good sized communities online where you could share photos with other people in the same spot, and you were all getting a little better. It was something you could take pride in. Now all the gear is great. Your cell phone camera is great. It’s hard to stand-out. Everyone has read the same tutorials, everyone can do HDR and panoramas. They can even do them in-camera with one button. And as photography goes, so goes video.
For a while I thought that 3d printing and the maker movement might be a little like photography. There’s plenty of gear to collect, and it can make a big difference in the final product, but skill and technique and creativity still count for a lot. Now I’m leaning towards 3d printing and the maker movement really being a rediscovery of the physical after the birth of the age of software. Before personal computers ate the world you could still find plenty of folks who knew about gear ratios and metallurgy and who’d put together crystal radios when they were kids. I grew up in the 80s, and I don’t know anything about either of those things, but I was diagnosing IRQ conflicts before I liked girls. So the maker movement is kind of new, and photography is kind of past the curve, so what’s new-new? What’s going to eat our time and interest and energy and fill our walls and display shelves next? What are we going to collect and tinker with and obsess over?
Beautiful, New Things
It’s been said that we’re all in the attention game now. Attention is currency. In an indirectly monetized world it’s what people have to give. When you create something, you’re vying for that bit of attention. Given that, I think we’re looking at the birth of a new kind of craft, and a new kind of object.
Let’s call them magical objects: Objects that use software and computation to break or make irrelevant their inherent limitations, for the purpose of entertaining or informing. They’re objects that use software to amplify their Attention Quotient. (AQ, is that a thing? It should be.)
First, I’d like you to look at a video that hit a few days ago, Box. It’s what happens when you combine a bunch of creative folks, some big robot arms, projectors, cameras, and a whole bunch of software.
That’s pretty awesome, right? Not really practical for your house, but pretty. Let’s find something smaller, something more intimate. Maybe something more tactile. Something like… a sandbox…
Ok, now we’re getting somewhere. It’s a sandbox that reacts to your input. The software and the projectors and the cameras make the sandbox more than just a sand table with some water on it, the whole thing becomes an application platform, with sand and touch as it’s interface. The object becomes magical. When you look at a sandbox, you know what it can do. When you look at an augmented sandbox, you don’t know what it does. You have to play with it. You have to explore. It has a high attention quotient.
These kind of objects are going to proliferate like crazy in the next few years. We’re already starting to see hints of it in iOS 7’s Parallax wallpaper. The only reason that parallax wallpaper exists is to make your iDevice more magical. It serves no other purpose than to use software (head distance, accelerometer movement tracking) to overcome the limitations of hardware (2d display), for the purpose of delighting the user (magic).
Kids These Days
So as we think about the future, let’s step back for a second, and think about the children. At the Austin Personal Cloud meetup a few weeks ago I had a realization that everyone in the room was probably over the age of 30, and there were plenty over the age of 50. We have to be really careful about prognosticating and planning the future, because the world that we see isn’t the world that those in their teens and 20’s see. They have different reference points, and they’re inspired by different things. I’ve written before about Adventure Time and The Amazing World of Gumball as training for future engineers. But it occurs to me that when it comes to magical objects, we only need to look at the name to tell us where the inspiration for the next generation will spring.
Part of the thing that makes Harry Potter’s world wonderful is that things are more than they appear. A car isn’t just a car, a hat isn’t just a hat, and a map isn’t just a map. For all the plot-driving magical objects in Harry Potter like the Time Turner, there are plenty of wandering portraits, chocolate frog trading cards, and miscellaneous baubles. They amp up the attention quotient of the world. Maybe they’re the reason we don’t see Harry and Hermione checking Facebook all day, or maybe they just have awful coverage at Hogwarts.
My daughter’s about to turn 2, and her newest discovery is that if she holds a cup to her ear, it kind of sounds like the ocean. After I showed her that, she held the cup to her ear for a good 20 minutes. I hold the cup up to my ear, and I hear science. She holds the cup to her ear, and she hears magic. Her eyes are wide, and she says, “Ocean!” over and over.
We can make these magical objects now, and we have a generation that would love more meaningful interaction from physical things. We just need to start assembling the bits and deciding on a few simple standards so we can create ecosystems of art. We don’t have magic, but we have something that’s nearly as good. We have software…
That’s a documentary about Processing. You don’t need to watch the whole thing, but it’s pretty, and interesting. Processing is a programming language for visual arts. Usually those interesting visual things live on a screen, or through a projector in space or on a building. They rarely live in your house. But they could, and they could be really cool.
Wherein We Sketch Out the Future
I think that by combining the artistic software movement, emergent behavior fields like procedural game world generation, and a little bit of hardware hacker know-how, we can create a new type of thing. A magical, home object. Let’s look at one…
So this is a thing. Literally a back-of-an-envelope sketch. It’s a bowl, or a box, with an arm extending over it. In the bowl is sand, or perhaps something more pure-white but still eco-friendly and non-toxic. At the end of the arm is a little pod, it has two cameras in it, for stereoscopic 3D, and a pico projector. Maybe there’s even another projector pointing up out of it. Under the bowl is the descendant of a Raspberry Pi, or a Beaglebone Black, or something like it. It lives on a side table or end table in your house.
This magical device runs programs. The programs use the sand (or whatever you put under the arm) as an interface. It can recognize other objects, maybe little shovels or pointers or what have you. Maybe simple programs are like our virtual sandbox above. Maybe it’s like a bonsai, but instead of a virtual tree, it runs a simulation of an ecological ecosystem. Dig out your valleys and pile up your mountains, and see trees grow, animals roam the steppes, birds fly… Maybe you can even run a game on that, like Populous, but instead of looking into the screen you can walk around it and touch it. You can watch your little minions wander around the landscape. Maybe you can talk to it. Maybe it’s like the asteroid that hits Bender in Futurama’s Godfella’s episode, like Black and White but designed for the long-haul. Maybe when I’m not running my civilization on it, it plays selections from a feed of cool Processing visualizations across my ceiling.
Back to the Beginning
I’m sure there will be all kinds of form factors for these magical objects. They’ll come in pocket-sized compacts, or ceiling projectors, or robotically controlled room projectors (imagine a bunch of tiny Disney-esque mice that live in your house, but are only projected onto the walls and floorboards, not actually chewing through them). Or maybe it’s like my photo of Marken, in a frame on the wall, except that it’s based off a video clip, or some software analyzes the scene and says, “Hey, this is grass, let’s make it wave a little, and these are clouds, so they should float by, and this is a sailboat, so it should drift back and forth.” And maybe, if you lean in really close, you can hear the ocean.
Wednesday I presented a talk at the Austin Personal Cloud meetup about Building a Personal Cloud computer. Murphy was in full effect, so both of the cameras we had to record the session died, and I forgot to start my audio recorder. I’ve decided to write out the notes that I should have had, so here’s the presentation if it had been read.
In this presentation we’re talking about building a personal cloud computer. This is one approach to the personal cloud, there are certainly others, but this is the one that has been ringing true to me lately.
A lot of what people have been talking about when they speak about the personal cloud is really personal pervasive storage. These are things like Dropbox or Evernote. It’s the concept of having your files everywhere, and being able to give permission to things that want to access them. Think Google Drive, as well.
These concepts are certainly valid, but I’m more interested in software, and I think computing really comes down to running programs. For me, the personal cloud has storage, but it’s power is in the fact that it executes programs for me, just like my personal computer at home.
That computer in the slide is a Commodore +4, the first computer I ever laid fingers on.
Back then, idea of running programs for yourself still appealed to the dreamers. They made movies like TRON, and we anthropomorphized the software we were writing. These were our programs doing work for us, and if we were just smart enough and spent enough time at it, we could change our lives and change the world.
This idea isn’t new, in fact AI pioneers were talking about it back in the 50s. John McCarthy was thinking about it back then, as Alan Kay relates when he talks about his 3rd age of computing:
They had in view a system that, when given a goal, could carry out the details of the appropriate computer operations and could ask for and receive advice, offered in human terms, when it was stuck. An agent would be a ‘soft robot’ living and doing its business within the computer world.
That’s been the dream for a long time…
But that never really happened. The personal computer revolution revolutionized business, and it changed how we communicated with each other, but before the Internet things didn’t interconnect to the point where software could be a useful helper, and then we all went crazy making money with .com 1.0 and Web 2.0, and it was all about being easy and carving out a market niche. Then something else hit…
Mobile exploded. If you’ll notice, mobile applications never really had an early adopter phase. There was no early computing era for mobile. You could say that PDAs were it, but without connectivity that isn’t the same as the world we have now. Most developers couldn’t get their app onto a mobile device until the iOS app store hit, but that platform was already locked down. There was no experimentation phase with no boundaries. We still haven’t had the ability to have an always-connected device in our pocket that can run whatever we want. The Ubuntu phones may be that, but we’re 6 iterations into the post-iPhone era.
And who doesn’t love mobile? Who doesn’t love their phone? They’re great, they’re easy to use, they solve our problems. What’s wrong with them? Why do we need something else? Well, let’s compare them to what we’ve got…
With the PC we had a unique device in so far as we owned the hardware, we owned our data, and EULA issues aside, we owned the software. You could pack up your PC, take it with you to the top of a mountain in Nepal, and write your great novel or game or program, with no worries about someone deactivating it or the machine being EOLed. Unfortunately the PC is stuck at your house, unscalable, badly networked, loaded with an OS that was designed for compatibility with programs written 25 years ago. It isn’t an Internet era machine.
With the web we got Software as a Service (SaaS), and with this I’m thinking about the Picasa’s and Flickr’s and Bloggers of the world. No software to maintain, no hardware to maintain, access to some of your data (but not all of it, such as not having access to traffic metrics with Flickr unless you paid, and only export rights if you were paid up). But in this new world you can’t guarantee your continuity of experience. Flickr releases a redesign and the experience you’ve depended on goes away. The way you’ve organized and curated your content no longer makes sense. Or maybe as in the case of sites like Gowalla, the whole thing just disappears one day.
Mobile has it’s own issues. You often don’t own the hardware, you’re leasing it or it’s locked up and difficult to control. You can’t take your phone to another provider, you can’t install whatever software you want on it. Sometimes it’s difficult to get data out. How do you store the savegame files from your favorite iPhone game without a whole-device snapshot? How do you get files out of a note taking app if it doesn’t have Dropbox integration? In the end, you don’t even really own a lot of that software. Many apps only work with specific back-end services, and once your phone gets older, support starts to disappear. Upgrade or throw it in the junk pile.
Cloud offers us new options. We don’t have to own the hardware, we can just access it through standards compliant means. That’s what OpenStack is all about. OpenStack’s a platform, but OpenStack is also an API promise. If you can do it with X provider, you can also do it with Y provider. No vendor lock-in is even one of the bullet points on our homepage at HP Cloud.
Implicit in cloud is that you own your own data. You may pay to have it mutated, but you own the input and the output. A lot of the software we use in cloud systems is either free, or stuff that you own (usually by building it or tweaking it yourself). It’s a lot more like the old PC model than Mobile or SaaS.
All of these systems solve specific types of problems, and for the Personal Cloud to really take off, I think it needs to solve a problem better than the alternatives. It has to be the logical choice for some problem set. (At the meetup we spent a lot of time discussing exactly what that problem could be, and if the millennials would even have the same problems those of us over 30 do. I’m not sure anyone has a definitive answer for that yet.)
This is what I think the Personal Cloud is waiting for. This explosion of data from all our connected devices, from the metrics of everything we do, read, and say, and what everyone around us says and does. I think the Personal Cloud has a unique place, being Internet-native, as the ideal place to solve those problems. We’re generating more data from our activities than ever before, and the new wave of Quantified Self and Internet of Things devices is just going to amplify that. How many data points a day does my FitBit generate? Stephen Wolfram’s been collecting personal analytics for decades, but how many of us have the skill to create our own suite of tools to analyze it, like he does?
The other play the Personal Cloud can make is as a defense against the productization of you. Bruce Sterling was talking about The Stacks years ago, but maybe there’s an actual defensive strategy against just being a metric in some billion dollar corporations database. I worked on retail systems for a while, it wouldn’t surprise me at all if based on the order of items scanned out of your cart at Target (plus some anonymized data mining from store cameras) they could re-construct your likely path through the store. Track you over time based on your hashed credit card information, and they know a whole lot about you. You don’t know a whole lot about them, though. Maybe the Personal Cloud’s place is to alert you to when you’re being played.
In the end I think the Personal Cloud is about you. It’s about privacy, it’s about personal empowerment. It’s uniquely just about you and your needs, just like the Personal Computer was personal, but can’t keep up, so the Personal Cloud Computer will take that mantel.
The new dream, I think, is that the Personal Cloud Computer runs those programs for you, and acts like your own TRON. It’s your guardian, your watchdog, your companion in a world gone data mad. Just like airbags in your car protect you against the volume of other automobiles and your own lack of perfect focus, so your Personal Cloud protects you against malicious or inconsiderate manipulation and your own data privacy unawareness.
To do this I think the Personal Cloud Computer has to live a central role in your digital life. I think it needs to be a place that other things connect to, a central switching station for everything else.
And I think this is the promise it can fulfill. The PC was a computer that was personal. We could write diary entries, work on our novel for years, collect our photos. In the early days of the Internet, we could even be anonymous. We could play and pretend, we could take on different personas and try them out, like the freedom you have when you move to a new place or a new school or job. We had the freedom to disappear, to be forgotten. This is a freedom that kids today may not have. Everything can connect for these kids (note the links to my LinkedIn profile, Flickr Photos, Twitter account, etc in the sidebar), though they don’t. They seem to be working around this, routing around the failure, but Google and others are working against that. Facebook buys Instagram because that’s where the kids are. Eventually everything connects and is discoverable, though it may be years after the fact.
So how do I think this looks, when the code hits the circuits? I think the Personal Cloud Computer (or ‘a’ personal cloud computer) will look like this:
A Migratory – Think OpenStack APIs, and an orchestration tool optimized for provider price/security/privacy/whuffie.
Standards Compliant – Your PCC can talk to mine, and Facebook knows how to talk to both.
Remotely Accessible – Responsive HTML5 on your Phone, Tablet and Desktop. Voice and Cards for Glass.
API Nexus – Everything connects through it, so it can track what’s going on.
with Authentication – You authenticate with it, Twitter authenticates with it, you don’t have a password at Twitter.
Application Hosting – It all comes down to running Apps, just like the PC. No provider can build everything, apps have to be easy to port and easy to build.
Permission Delegation – These two apps want to talk to each other, so let them. They want to share files, so expose a cloud storage container/bucket for them to use.
Managed Updates – It has to be up to date all the time, look to Mobile for this.
Notifications – It has to be able to get ahold of you, since things are happening all the time online.
and Dynamic Scaling Capabilities – Think spinning up a hadoop cluster to process your lifelog camera data for face and word detection every night, then spinning it down when it’s done.
So how do we actually make this happen? What bits and bobs already exist that look like they’d be good foundational pieces, or good applications to sit on top?
No presentation these days would be complete without a mention of docker, and this one is no different. If you haven’t heard of docker, it’s the hot new orchestration platform that makes bundling up apps and deploying lightweight linux container images super-easy. It’s almost a PaaS in a box, and has blown up like few projects before it in the last 6 months. Docker lets you bundle up an application and run it on a laptop, a home server, in a cloud, or on a managed Platform as a Service. One image, multiple environments, multiple capacities. Looking at that Ubuntu Edge, that looks like a perfect way to sandbox applications iOS style, but still give them what they need to be functional.
Hubot is a chat bot, a descendant of the IRC bots that flourished in the 90’s. Hubot was built by Github, and was originally designed to make orchestration and system management easier. Since they connect and collaborate in text based chat rooms, Hubot sits in their waiting for someone to give it a command. Once it hears a command, it goes off and does it, whether it be to restart a server, post an image or say a joke. You can imagine that you could have a Personal Cloud Computer bot that you’d say ‘I’m on my way home, and it’s pot roast night’ to, and it would switch on the Air Conditioner, turn on the TV and queue up your favorite show, and fire up the crock pot.
The great thing about Hubot, and the thing about these Personal Cloud Bots, is that like WordPress Plugins, they’re developed largely by the community. Github being who they are, Hubot embraces the open development model, and users have developed hundreds of scripts that add functionality to Hubot. I expect we’ll see the same thing with the Personal Cloud Computer.
I’ve talked about Weavrspretty extensively here on the blog before, so I won’t go into serious depth, but I think that the Personal Cloud Computer is the perfect place for something like Weavrs to live. Weavrs are social bots that have big-data derived personalities, you can create as many of them as you like, and watch them do their thing. That’s a nice playground to play with personalities, to experiment and see what bubbles to the top from the chaos of the internet.
If you listen to game developers talk, you’ll start to hear about that initial dream that got them into game development, the dream of a system that tells stories, or tells stories collaboratively with you. The Kickstarted game Sir, You Are Being Hunted has been playing with this, specifically with their procedurally generated British Countryside Generator. I think there’s a lot of room for that closely personal kind of entertainment experience, and the Personal Cloud Computer could be a great place to do it.
Aaron Cope is someone you should be following if you aren’t. He used to be at Flickr, and is now at the Cooper-Hewett Design Museum in New York. His Time Pixels talk is fantastic. Two of the things that Aaron has worked on of interest are Parallel Flickr, (a networkable backup engine for Flickr, that lets you backup your photos and your contacts photos, but is API compatible with Flickr) and privatesquare (a foursquare checkin proxy that lets you keep your checkins private if you want, or make them public). That feels like a really great Personal Cloud app to me, because it plays to that API Nexus feature.
The Numenta guys are doing some really interesting stuff, and have open sourced their brain simulation system that does pattern learning and prediction. They want people to use it and build apps on top of it, and we’re a long way away from real use, but that could lead to some cool personal data insights that you run yourself. HP spent a bunch of money on Autonomy because extracting insights from the stream of data has a lot of value. Numenta could be a similar piece for the Personal Cloud.
That’s the Adafruit Pi Printer, Berg has their Little Printer, and they’re building a cloud platform for these kind of things. These devices bring the internet to the real world in interesting ways, and there’s a lot of room for personal innovation. People want massively personalized products, and the Personal Cloud Computer can be a good data conduit for that.
Beyond printers, we have internet connected thermostats, doorknobs, and some of those service companies will inevitably go away before people stop using their products. What happens to your wifi thermostat or wifi lightbulbs when the company behind it goes way? Personal Cloud lets you support that going forward, it lets you maintain your own service continuity.
Having an always-on personal app platform lets us utilize interesting APIs provided by other companies to process our data in ways we can’t with open source or our own apps. Mashape has a marketplace that lets you pick and switch between api providers, and lets you extend your Personal Cloud in interesting ways, like getting a sentiment analysis for your Twitter followers.
In addition to stuff we can touch over the network, there’s a growing market of providers that let you trigger meatspace actions through an API. Taskrabbit has an API, oDesk does, Shapeways does, and we haven’t even begun to scratch the possibilities that opens up.
One thing to watch is how the Enterprise market is adapting to utility computing and the cloud. The problems they have (marketplaces, managed permissions, security for apps that run premises, big data) are problems that all of us will have in a few years. We can make the technology work with enterprise and startups, but for end users, we have to make it simple. We have to iPhone it.
So where do we start? I think we have to start with a just good enough, minimum viable product that solves a real problem people have. Early adopters adopt a technology that empowers them or excites them in some way, and whatever Personal Cloud platforms appear, they have to scratch an itch. This is super-critical. I think the VRM stuff from Doc Searls is really interesting, but it doesn’t scratch an itch that I have today in a way I can comprehend. If you’ve been talking about something for years, what will likely happen is not that it’ll eventually grow up, it’s that something radical will come out of left field that uses some of those ideas, but doesn’t honor all of them. That’s my opinion, at least. I think the Personal Cloud community that’s been going for years with the Internet Identity Workshop probably won’t be where the big new thing comes from, but a lot of their ideas will be in it. That’s just my gut feeling.
The last caveat is that Apple and Microsoft and Google are perfectly positioned to make this happen with vendor lockin easily. They all already do cloud. They all have app stores. They have accounts for you, and they want to keep you in their system. Imagine an Apple App Store that goes beyond your iPhone, iPad and even Apple TV, but lets you run apps in iCloud? That’s an easy jump for them, and a huge upending of the Personal Cloud world. Google can do the exact same thing, and they’re even more likely to.
So thanks for your time, and for listening (reading). If you have comments, please share them. It’s an exciting time.
Two weeks ago I had the pleasure of speaking at my first PyTexas conference. I’d never been to PyTexas before, but I’ve been to it’s Ruby relative, Lone Star Ruby a bunch of times. In a lot of ways it was similar (the local crowd, lots of enthusiasts, two tracks of talks), but in some ways, very different…
The first and most notable thing to mention about PyTexas is that it’s held at the Memorial Student Center at Texas A&M University, which is in College Station. That means the conference is two hours from Austin and Houston, and three hours from San Antonio and Dallas/Fort Worth. This isn’t a complaint, it’s a nice facility, but it explains something about PyTexas: It’s not and will never be a large programming conference, simply due to being too far from the Texas programmer population. That being said, it’s impressive how many people they’ve pulled in, and is a testament to the Texas Python community that so many people (about 100 folks the day I was there) made the trip.
The tradeoff for the drive is that the event (being hosted by the A&M School of Architecture) is really inexpensive ($25 early bird, $50 regular). I would have thought that would have meant there would have been a big student turnout, but that didn’t seem to be the case. School hadn’t started yet, so that may be one reason. There were a lot of interested, engaged professionals there, and a lot of people doing serious day to day work with python. I saw a couple of Rackers, and though there wasn’t anyone else I knew from HP Cloud, there was some OpenStack talk in the halls.
My wife has been getting into python recently, and since I wasn’t planning on spending the night away from home (2 year old daughter + 7 months pregnant wife = at home at night), I talked her into coming with me for the day. Registration was well organized, and there were good snacks. The event had a few sponsors I wasn’t familiar with, including MapMyFitness, which tracks exercise metrics for folks, and StormPulse, which provides weather forecasts for businesses. It’s always nice to see businesses showing how they’re using a language for real. The Lone Star Ruby conference companies tend towards web startups and Rails.
The gender balance was about what you’d expect, maybe 10:1. If it was a little bigger there might be a more organized outreach, but right now it’s just word of mouth. I did hear about it on the PyLadies ATX list, and there may have been more women on the tutorial day.
I think there were some challenges on the organization side of the conference. Speakers didn’t seem to get into the registration system, and two of the speakers didn’t show up. That’d be easier to compensate for at a bigger conference, but when there are only two tracks it really shows. Unfortunately one of the no-shows was Thomas Hatch of SaltStack, whose talk I was really looking forward to. Maybe it’s online somewhere.
I’d proposed two talks, but only had time to prepare one, so I ended up spending 50 minutes talking the audience through building two simple Bottle applications. One of the apps serves as an API service, the other as a web-exposed UI. The code for both, built step by step with comments, is up on GitHub. I’ll link to the video of the talk whenever they post it.
Walker Hale from the Baylor College of Medicine down in Houston spoke before me, talking about Bottle’s sister microframework Flask. Flask and Bottle are really, really, really similar, so he stole a bit of my thunder, but I think the audience enjoyed the live coding I did (with paper diffs!), and I got some good feedback. Unfortunately the Memorial Students Center is a no-hat building (out of respect for the Aggies who’ve given their lives in defense of the country), so the audience had to endure my out of control mop.
Lunch was included in the cost of registration and provided by a nice local food truck.
There were a couple of lightning talks at the end of the day, including Barbara Shaurette of PyLadies Austin talking about her interesting new initiative to connect professional programmers with high school computer classrooms. No set of lightning talks would be complete without the next big thing, Docker.io, so of course there were two (!) of those. Docker’s going to take over the world, believe me.
PyTexas was a fun little conference, though driving down in the morning and back in the evening was really exhausting. It’s small, and isn’t as slick as some larger conferences, but it has a nice raw charm. The love the attendees and speakers have for python really shows through. If it’s easy for you to get to, and you aren’t busy, I recommend it. If they moved it to Austin or San Antonio, I’d go for the whole thing and I think the conference would be at least three times as big. (Speaking of Texas python conferences, if you haven’t signed the Austin PyCon 2016/2017 petition, please do!)
It’s summer in Texas, which means one thing: It’s time to get away. Last week I got away to OSCON, O’Reilly’s annual Open Source conference, in lovely, Portland, Oregon. Herein is the account of that trip.
OSCON is a two and a half day conference preceded by two days of related tutorial sessions. HP was a Diamond sponsor this year, so I finagled a free badge, and decided to go to the whole thing. We didn’t have extra travel budget in my team, though, so I paid hotel and airfare out of my own pocket. More on whether that was a worthwhile expense or not at the end of this post.
OSCON is a pan-technology conference. As long as the project is Open Source, it’s welcome at OSCON. Therefore you get a lot of variety, which is evidenced by the gigantic array of networking ribbons. I didn’t stick one on, but I saw a few people with displays that would have made a Texas High School homecoming corsage maker jealous.
The Distributed Sensor Network tutorial seemed really promising, but unfortunately we were missing the micro USB cables we needed to power our Arduinos. Oh, and the Adafruit XBee Adapters we got were supposed to be pre-soldered, but weren’t. Not an easy problem to solve when you have no soldering irons and only two and a half hours to do the whole tutorial.
The intent was to have an Arduino based sensor mote with temperature, humidity, IR-based movement and volume (sound pressure) sensors, which transmitted its data to a remote computer via the wireless XBee system. Unfortunately we didn’t have the XBee adapters, and until half way through the class we couldn’t even power our Arduinos. Fortunately one of the volunteers managed to run to Radio Shack and get us USB cables, but by then half the class was over. We did manage to rig up a sensor to our Arduinos and get the data appearing via serial, and we have all the parts and the book with instructions to finish the project, but it was feeling like two and a half strikes in a row before I went to the Erlang talk…
Which was awesome! Erlang is the weird friend you never knew you needed. She does all the things that your other friends are terrible at, and after a long heart to heart at the local brewery, you totally get her. Conference saved. If multi-actor, highly scalable, multi-core programming is interesting to you, there are some great resources on its page, including Francesco Cesarini’s slides.
Erlang and Go seem to be two different implementations of similar ideas, trying to solve the massive concurrency problem in a structured, production-ready, robust way. Go’s the hot new kid on the block, while Erlang has been in production for nearly 20 years. Erlang seems to be a more interesting solution to me, though if you really like writing Java, C or C++, you might prefer Go.
You might have used Erlang if you’ve used CouchDB, Couchbase, Riak, Facebook Chat, Chef, RabbitMQ, voted in any of the UK Big Brother style SMS voting events, or ever sent data over a mobile phone network. It grows across cores beautifully, and seems like it’ll be a really great solution when 64+ core processors hit the big time. So, Erlang = Awesome, Conference Tutorials = Very Risky, Arduino Sensor Motes = Someday.
Thursday’s opening party was space themed (I heard that last year it was Camp OSCON with merit badge activities and the like). They had a jumpy balloon rig, space themed arcade games, interactive art, an indoor inflatable planetarium, a make your own space helmet craft table, and laser tag. It was fun and loud, but the food options were limited for those on a diet, and as a non-social person, I soon wandered back to my hotel.
Every year OSCON has a nerd-oriented competitive activity. Beat the game, win a prize. This year the game was to collect 20 puzzle pieces (which you got from visiting booths, attending keynotes, having lunch, etc), and the prize was an OSCON 15th anniversary hoodie. As a puzzle oriented and easily obsessed person I got my hoodie Wednesday morning, a few hours after the last piece had been made available. I was somewhat disappointed to see that there were still hoodies available the last day, but I guess it’s good that those slackers were able to win, too.
Wednesday morning kicked off with keynotes, which were presented in an interesting, 10-20 minutes per speaker format. One of the opening talks was by the president of Canonical, the company that produces Ubuntu and the cloud-oriented app orchestration system Juju. He demoed Juju’s graphical cluster creation system running on top of HP Cloud, which was nice for us. Juju looks like a neat system that compliments the existing solutions well, and it’s high on my list of things to look into. There was also a great keynote about ‘My Robot Friend’ by Carin Meier, where she bravely did live hardware demos on stage, including a Clojure controlled quadcopter.
The most interesting keynote, though, was from Numenta. Numenta’s keynote was presented by Jeff Hawkins, one of their founders and the guy who started Palm and Handspring. Their technology simulates the neocortex, the part of your brain that remembers things and predicts patterns (specifically in their software, a 64,000 synapse slice of one of the layers). They call it the Cortical Learning Algorithm, and they’ve open sourced it in the form of NuPIC (Numenta Platform for Intelligent Computing). You feed data into this thing, and over time it builds up a map of the patterns in the data and can start to predict what will happen next. The science is beyond me, but the demo and keynote was great, and you can (should) watch it on YouTube. I went to their panel later, and they recommended Jeff’s book On Intelligence as a primer for those interested. There are code samples (in Python!) with the NuPIC library up on their github account.
HP covered lunch for everybody on Wednesday, but I can’t remember what it was. (I started doing a DietBet last week, so I only ate salads the entire conference.) The conversations at lunch, though, were great. On Wednesday I sat at a table with a Wisconsin lo-power FM radio and wholesale ISP guy, someone doing Hadoop at Disney (who’d previously worked at AWS), someone running a private cloud in Vancouver doing simulation-based pharmaceutical discovery, some guys from BlueHost (one of Code for America’s biggest sponsors) in Orem, and a guy who worked for an Apple accessory manufacturer in Portland.
The other panels I went to on Wednesday were one on the temporary cell phone network they setup up during Burning Man, a walkthrough of the parts and software needed to build your own cell phone with an Arduino (did you know that cell phone brains like the SIMCom SIM900 operate with an AT-command derived control setup, like your old 28.8 modem, including AT+HTTP commands to fetch web urls?), a talk on discreet math, and then one on getting kids to code (check out drtechniko.com, a robot language for kids to ‘program’ people, and Alice, a programmable machinima generator). The last panel of the day was An Overview of Open Source in East Asia, with some really interesting insights into the Open Source community in China, Korea and Japan (and they gave us all free fans!).
3 years ago at OSCON the OpenStack project made its debut, so that means it was time for a 3rd birthday bash. Fellow HP Cloud-er Rajeev Pandey and I walked over, enjoyed some gazpacho shots, picked up a t-shirt or two, and marveled at their giant paella (seriously, they were like 3 feet wide). We ran into a few other HP Cloud folks there, including Monty Taylor. There was a cute birthday cake and lots of cupcakes, but after nibbling and conversing and drinking lots of water (it was surprisingly warm in Portland), soon it was time to go. Happy Birthday, OpenStack, in software years you’ve almost hit puberty.
Thursday I attended Tim O’Reilly’s talk on Creating More Value Than You Capture (and as an aside, I felt both sorry for Tim in only getting 30-40 attendees, but also better about the 15 my talk pulled in at SXSW), and a great intro to Docker from dotCloud. If you haven’t looked at Docker, check it out. The way they bundle up app binaries on top of base machines is awesome. Then came lunch, with another great group of folks including someone managing DevOps for Disney.com (the entire thing on 60 VMs!).
After lunch was a really great talk on Kicking Impostor Syndrome in the Head by Denise Paolucci. If you ever feel insecure about your skills, dig up a video of her giving that talk, it was really great. After that was Designing the Internet of Things with the 3 Laws of Robotics, and then From Maker to China, where Brady Forrest described the challenges and pitfalls of taking a concept from prototype to small-scale manufacturing in China. One book he recommended for those interested in the product design and manufacturing process was From Concept to Consumer, which now rests on my Amazon wishlist. After that it was Hardware Hacking with Your Kids, with some funny slides and interesting anecdotes from Dave Neary, and then we were done for the day. That night I worked on my SXSW panel proposal, and went to bed early.
The trade show went on Wednesday and Thursday, and had a good mix of big companies, lots of non-profits, and some interestingly unexpected exhibitors (League of Legends maker Riot Games). There were some great shirts, including this Cloudera one: Data is the New Bacon, and its sister, Data is the New Tofu, one from the Kenyan data mapping non-profit Ushahidi, and plenty of other knicknacks and stickers for the kids back home. PyLadies was there, Wikimedia was there, Craigslist was there, FSF, EFF, and the Linux Foundation were there. Everyone was hiring. The Tizen folks are giving away $4,040,000 (that’s four, count em four… million…) dollars in app development prizes. There were more hosting and big data software companies than I have fingers and toes. It’s a good time to be in technology.
Friday was only a half day, so after a keynote exhorting us to join the ACM, one noting that everything important has already been invented, and some group singing, we settled down to business. First up was Cryptography Pitfalls with John Downey of Braintree. That was a great talk, and though I knew a lot of the gotchas he mentioned, it was still nice to hear them laid out by a professional. In short: Use a slow one-way hasher for passwords, don’t build your own crypto implementations, and always check SSL cert validity in your application code. The slides are up, you should take a look at them.
After a break we headed into Open Source Social Coding for Good, with Benetech. I’d run into the folks from Benetech in the trade show the day before, and was really excited to learn that they were doing hackathons already with HP’s Office of Global Social Innovation through their SocialCoding4Good project. I’m really hoping to connect both of them to HP Cloud and do a hackathon in Austin. The panel was great, and it was good to hear about nonprofits getting traction from corporate hackathons and volunteers. We need to do more of that. After that it was Polyglot Application Persistence, and then the conference was over.
So, back to my original question, was it worth it? Would I go again?
If you’re in Portland, or the Portland area, I think it’s a no-brainer. It’s a great conference, the attendees are sharp, it covers a ton of stuff, the keynotes are good, and I’m sure there’s something interesting every year. The trade show’s great. If you can’t snag a speaking slot or a super-discounted badge, you could get a lot of the value by getting an expo badge and watching the keynotes online. If you’re paying for it yourself, and traveling to do it, it becomes a much murkier question. So many conferences are putting everything online these days, what you’re really paying for are the networking opportunities and the experience: That conference euphoria of anything is possible. That has a lot of value, but if you’re on a budget, maybe local conferences, hackathons, or meetups are good enough. I hope I’ll be back at OSCON next year, but if I’m not, you’ll all just have to have fun without me.
Perhaps a little introduction is in order. The world that Saturn’s Children and Neptune’s Brood are set in is a hard sci-fi space opera universe. It’s thousands of years in the future, humanity has died out, but our assistants, the humanioid bots we built in our image, kept on trucking. They populated the galaxy (in the first book) and now, some thousands of years later, they have expanded by very slow means to other star systems. Of course, humanoids aren’t optimized for every environment, so the essential components of synthetic life take lots of forms, little bat creatures, mermaids, squid, worms, etc. Everything that used to be biological is now biomechanical, but still simulates multi-cell life.
Neptune’s Brood is a find-the-macguffin novel, the heroine Krina Alizond-114 is the forked prodigy of an intergalactic banker. In order to expand her reach, her mother forks 8 or 16 copies of herself into new bodies every so often. These copies are born with a debt-load (I told you this book was about money, right?), and if they manage to survive the years of indentured servitude to become real people, they may still be laboring under a giant debt load for their initial construction or housing. Our heroine is a specialist in a certain type of intergalactic banking fraud, and is trying to track down one of her fork-sisters who seems to be in trouble, and who might know the location of said macguffin.
Before Charles Stross wrote Neptune’s Brood, he read a book called Debt: The First 5,000 Years, and in order to understand how Neptune’s Brood formed, you should have at least a passing interest in money and debt. In trying to find her fork-sister, Krina is also trying to find a certain financial instrument, one that becomes clear as the story unfolds. Along the way she encounters religious zealots (spreading the flesh of humanity to the stars), pirates, Queens and cops, and more.
As I finished Neptune’s Brood, I had a real sneaking suspicion that I’d read the book before, which is either me pushing my impressions upon it, or a real reflection of Stross’s tendency to mash things up. It finally struck me that Neptune’s Brood felt a lot like Neil Gaiman’s Stardust in pacing, complete with pirates who are more than they initially appear. The pirates are almost like… well, the closest comparison I can come up with is Morpheus and his crew from The Matrix. It’s a bad comparison, but I think it relays tone.
This isn’t Stross’s first rodeo, and the book is well written, tightly paced and generally well built. The heroine is likable and relatable, and although she narrates the story largely from her perspective (so we know she gets through these scrapes), there’s still some tension. The ending is satisfying, though it leaves the reader wondering about its impact on the greater galaxy and the characters we’ve met.
If you like space operas, and especially if you like finance, Neptune’s Brood is easy to recommend. I’d probably read Saturn’s Children first (ignore the cover), because I think it’s probably a bit more ambitious and sets up the rules of the world more completely. They aren’t really connected beyond sharing the same galaxy, though, so feel free to jump in here.
One of the things those of us who don’t go through a traditional computer science program miss is a strong foundation in the hard science of computers. I don’t have a really strong algorithm, programming language design, or compiler background, but I want to learn. A few months ago I was geeking out with Rajeev Pandey, one of our Distinguished Technologists at HP Cloud (and all-around great guy), about how programming languages are like human languages and how they color our perceptions of the world. Rajeev mentioned that he could probably come up with a list of the top 5 programming language design books he’d read, and I jumped on it. I got that list from him a few weeks ago, he said it was fine for me to share it, so here it is on Amazon. I’m especially interested in reading The Recursive Universe and The New Turing Omnibus. Enjoy!
I never went to college. I wish I could say that it was entirely intentional, that I knew exactly what I was going to do after I graduated and followed that plan, but that isn’t how it happened. The real story is a lot less romantic. For those thinking about switching careers, or standing at the threshold of ‘real life’ and unsure what to do, it might hold some lessons, so let’s get started…
What really happened was that I was exhausted by school, terrible at working on things that didn’t have an immediate impact, and didn’t really get how the college application thing worked. My family has never been big on debt, and with the grades I had (from being terrible at things that didn’t have an immediate impact, like homework), I certainly wasn’t getting a free ride. I wanted a break, I wanted a chance to do real, practical things. The only problem was that I didn’t know what those real things were, and didn’t know anyone doing them.
Find an Open Door
In 1995 when I graduated high school the most exciting things were happening on the Internet. I’d learned a little HTML after getting online in 1994, but the web was still very much a “We’re trying to figure things out” space. Spaces like this are great, because even if you don’t have tons of experience, there isn’t a huge pool of best practices already to get up to speed on. I connected with some folks who were starting an Internet Service Provider in late 1995. This connection was something of a fluke, someone I knew from church. These days there are much better networking options for technology, but never turn down an opportunity.
Fortunately I had some useful knowledge about how to get MacOS machines online. It wasn’t a lot, but along with the HTML skills it got me in the door. These days the equivalent of that knowledge might be Photoshop skills from making LOLcat gifs, video editing skills from making meme mashups, some hardware skills due to school MindStorms programming, linux administration from running a Minecraft server, or social marketing skills from running a popular Twitter account, Tumblr blog or Facebook page. Anything that’s hard to master in a few days can get you in.
Don’t Expect it to Pay
When I first started doing Mac tech support for that little ISP in San Marcos I made a little over $200 a month. That isn’t much money, but it put gas in the car and put me in a position where I could play with the toys. Your job, once you have toys to play with, is to play the heck out of them.
In the first 6 months after I got my ‘job’ at the ISP, I built them a web site (you can still see it here) setup San Marcos’s first quake server, created Austin’s first streaming radio station (I registered mix947.com in February of 1996, and got the streaming working with a demo license of Real Media Server for BSDI and an old shop boombox), created a weekly user newsletter, started weekly user meetups at the shop, and even got involved with the local Internet Users Group at the library (which I ended up running).
You only do those kind of things if you’re in a space where there are no conventions or expectations. When there aren’t any streaming audio stations, setting one up with a 5 stream limit isn’t a deal-breaker. When all your users are early adopters you don’t need a marketing expert to write a user group email. You just do it. Luckily the ISP was run by Chad Neff, a great artist and stalwart defender of the user. He encouraged me to try things, and was my first great mentor in technology.
Hold on Passionately, but Loosely
An early, hard lesson to learn is when to let go. I didn’t let go of that job well, and though there were extenuating circumstances, and more people than just me were caught up in it, it made my life really messy for a few years. When you’re in the middle of the job, fight for the users as hard and as passionately as you can. If you aren’t creating things for someone, it’s a waste. Whether you’re knitting hats or writing tweets, you’re doing it for someone. Strive to make them as happy as possible.
Conversely, you have to know when it’s time to go. All things come to an end, and being able to sense that end and depart gracefully is a skill. Learn it. If you’re going into tech, read founder stories, especially the stories from founders who get kicked out. There’s a shift at each phase of a project or company life-cycle, from startup to growth and growth to long-term maturity. Finding out which phase you fit into best is important, as is being able to sense when that shift is coming.
Aside: Do you like to experiment, throw things together and see what sticks, with little heed for long term consequences? You’re probably startup minded. Do you like some stability, but enjoy seeing success build, working long nights to land the next client? Maybe growth is your bag. Are you risk-averse? Do you like long-term stability, dependable processes and maybe even enjoy corporate politics and intrigue? Then maybe you want a project in its mature phase.
Also, strive to recognize when things are heading for the toilet. There’s some honor in being the last one to turn off the lights and lock the door, and I’ve done it more than once, but it’s rarely the best thing for a career. Try and step back once in a while and assess things from the outside. Get some opinions from people you trust. Do right by your users, but recognize that not every situation is salvageable.
It’ll Be Embarrassing
For a long time I had a vision for starting a web design firm like Vivid Studios, a bay area web design shop that had the mid-90’s Wired techno-punk aesthetic nailed. It was a techno rebellious company producing amazingly creative, cutting edge work for great clients, and I wanted to be just like that. Unfortunately I was in San Marcos, Texas, not San Francisco, California, and I didn’t know anything about running a business, much less a hip design business. I didn’t know Bauhaus from an outhouse, if you know what I mean.
I carried that dream around for a lot of years, wanting to belong in a group of smart, forward thinking creatives. The dream took a lot of different shapes, and matured as I did. The first attempts were… laughable. In 1997 I started doing business as 57th Street Productions (yes, we apparently offered ‘innovative thinking’ as a service), which in 1999 became 57th Street, Inc. 57th Street lasted a year and a half before ceasing to be.
Aside: A while ago I’d read something that said you can find a lot out about a person by how they view their youthful mistakes. People who think ‘look at me, I was so stupid’ versus people who think ‘look at me, I was so cute’. People who realize that youth and inexperience is a perfectly valid excuse for shortcomings are more likely to grow and be happy than people who judge themselves harshly. Don’t be down on your past. Everyone has been the fool. Don’t settle for that being the whole story, though.
When you read stories about Bill Gates or other tech luminaries starting companies in their 20’s and being wildly successful, what you don’t read is about the support networks they had that made it possible. You don’t hear about the people they knew who had business experience, the years they’d had access to computers in their teens, the contracts they’d gotten due to flukes. When you don’t know how to get from point A to point B in business, don’t assume you can just muddle through. Go out and read some business books. Realize that if you don’t know people who need your services/product/etc, you can’t make money. Realize that if you only have one of these clients and don’t have any way to find a second, your business isn’t really a business, it’s just a relationship. Go find people who run real businesses, and get them to teach you the ropes. Ask them how they find clients, especially if they’re in a business similar to yours (say, physical engineering services to technology consulting). If you can’t sell your product for more than it costs to make, again, no business. You don’t need an MBA, but you need to know how to balance a checkbook, forecast earnings, pitch a client, close a deal, and make a profit.
I ended up doing some things I’m not really proud of at 57th Street in the hope of forcing that Vivid Studios dream into reality. I made some bad decisions (hiring people for personal reasons, not diversifying the client base, not making enough connections), and the only reason we made it as long as we did was that it was hard not to make money in technology in the late 90’s. If you’d like to get a taste for my embarrassing phase, you can check out these two tours, one from my ISP days at itouch.net, and one from my days as 57th Street, Inc.
You’ll Get a Break
What seems to happen is that eventually, if you keep plugging along, you’ll get a break. It will almost always be a result of some risk you’ve taken, or avenue you’ve explored. If you’re well connected, I guess it could be a connection your parents or buddies have, but that wasn’t my experience. In 1997, after joining The WELL, I met Jon Lebkowsky. We got into an online discussion about FreeNets, something I was interested in from my connection with the San Marcos Internet Users Group, and ended up having lunch at the Waterloo Ice House. We ate at the Ice House because it was next door to Jon’s gig at the time, Internet Guy at Whole Foods Market.
Nearly everything that has happened since, I can trace back to meeting Jon. Jon was having some trouble with Whole Foods in-store kiosk system. They were Windows NT Workstation based PCs with touch screens that browsed an internal web site in a locked-down browser. They were always breaking, stores shipped them back to WFM Central, and they had to be fixed. Jon needed someone to do the fixing, and I took the job. Your break may not be glorious. For me it was a windowless room fixing and re-imaging Windows NT machines, but it was a foot in the door at a company that had real enterprise-level problems, and even better, I got in at a very unique time.
Don’t Be Afraid to Go Up
My time at Whole Foods, in retrospect, was very strange. I’m sure some people have had similar experiences in other places, but now that I look back on it, it was kind of crazy. I think, though my memory is a little foggy, when I started working contract for Jon on the kiosk project I was making about $15/hour doing run-of-the-mill PC maintenance. Over the next 3 years my rate ended up peaking at something like $150/hour, and I was on a 40 hour a week retainer. Somewhere near the end the CIO of Whole Foods Market asked me into her office and offered me the chance to rebuild their programming team, hiring whoever I wanted. I was… 22. So, it’s a weird story.
I think my experience at Whole Foods comes down to two things. One, the luck to be in the right place at the right time, and two, never saying no to a problem. When I started on the kiosk project I was just re-imaging systems, fixing ones that were broken, and shipping out the replacements. That’s $15/hour work. Eventually the vendor that was supplying our keyblock software (so you couldn’t get out of the browser and break the machine) disappeared, so I offered to write a new one. I’d never written Windows NT device drivers before (or really any C code), but you don’t know you can’t till you try. Once you’re maintaining source code you’ve suddenly become more than an IT tech, and I think my rate bumped to $35/hour.
Now comes the right place/right time side of the story. This was in 1998. The Internet was hot, E-Commerce was boiling hot, and all the sharp programmers who’d toiled away for years on awk scripts and maintenance software wanted to go do the hot new thing. Whole Foods Market started WholeFoods.com, and nearly all the programmers from inside of Whole Foods left to join it. This left a gaping hole in the company that was being filled by one person.
Simultaneous to this exodus I, too, was exploring the job opportunities at WholeFoods.com. They made me an offer for $35k a year, and after verbally accepting it, I drove over to Whole Foods to get some dinner. In the parking lot I ran into Mark Mills, that one guy holding closed the gaping hole in internal development. While swinging around a pole in the parking lot, Mark gave me my next big break. Come to work for me, he said, and I’ll pay you $85/hour contract full time. You don’t have to be great at math to know that’s a lot better than $35k/year, so I declined WholeFoods.com’s offer, and went to work for Mark. Sometimes the opportunities are obvious.
You Have Potential in Others Eyes
When I joined Mark on the programming team, I was not a great programmer. I wasn’t even an ok programmer, but Mark, like Jon and Chad, must have seen potential, so he gave me problems to solve, and let me solve them. He gave me advice, showed me some tricks, and let me do things how I needed to do them. Mentors like this are great, seek them out, cleave to them, and strive to be like that when you’re in a position of authority.
After a few months of building data exports, Mark left Whole Foods as well. And then there was one.
Again, this is a right place, right time story. I had web skills, sys-admin skills, network skills and programming skills, and was in a large company with no internal programmers. Over the next few years I was able to build a suite of web applications (job posting, CMS, inventory management, document management, etc), working directly with the teams who would be using them, without any real technical oversight. I like to think that I did a good job, but I suppose that isn’t for me to judge. I just know that they were still using some of those applications years later, and the people I worked with always seemed happy to see me.
Once you get an opportunity to work on projects, it’s a chance to prove yourself and get experience shipping real product. During this phase I never had a project cancelled, I always delivered them on time, and I supported them myself. Strive to be the best, work professionally, and treat your users and customers how you’d want to be treated. You’ll make mistakes (always compress uploaded documents in a document storage system, your network storage admins will thank you later), but you learn from them.
Surprisingly, during this phase I even got called back in to WholeFoods.com (and later WholePeople.com) by Jon to build some integration software with Yahoo! Store, and managed to deliver in a few weeks what another consulting firm said wasn’t possible. This is your ‘don’t know it isn’t possible’ phase, enjoy it. Work your butt off, learn as much as you can, try new things. Responsibility comes next, and it’s a bear.
Out of the Garden
Eventually the gravy train ends. Whole Foods Market’s CIO offered me the job as lead developer, and the opportunity to hire anyone I wanted to rebuild the programming team. My life would have been completely different if I’d accepted, but I couldn’t in good conscience. I was 22. I didn’t really know what I was doing, but I knew I didn’t know what I was doing. I didn’t really want an employee gig. I turned her down.
The next few years heralded the popping of the dot-com bubble. I drifted away from Whole Foods Market as they hired programmers internally, though I kept maintaining the systems that ran the WholeFoodsMarket.com web site until they replaced the entire thing in the late 2008. From the time we launched it (on time) in 2000 to 2008, it was powered by the same Apache Server-Side Include based architecture, running on a single Sun machine.
After WholePeople.com imploded with the dot-com bubble, Jon Lebkowsky and I started talking about starting a web consulting company. Visions of Vivid Studios started dancing in my head. I even managed to rope my buddy Matt Sanders into joining us. Together we founded Polycot Consulting, and started learning all those business lessons the hard way.
Remember how I said that a business with only one customer and no way to find more isn’t a business? That was us at Polycot. We spent a lot of time in the wilderness, trying to find work in the post-dot-com rubble. It wasn’t easy. We learned a lot of lessons the hard way. A few of them:
You can’t pay your rent with leads, you can only pay your rent with paid invoices.
There’s a difference between the things you want to get the job done and the things you need to get the job done.
Doing cheap jobs for ‘exposure’ is a trap. You will end up just doing cheap jobs, and your customers will expect the world. Our CPA once told us that Pro Bono work was a great way to get other work, but the other work will always be Pro Bono.
Corporations view the world differently than non-profits and mom-and-pops. Don’t ask a Fortune 500 if they want to pay $100 extra for a life-time license of some software you’re using, that isn’t real money to them.
You need someone who knows how to sell. You can evangelize a product, but you have to sell consulting.
Make a product, and make sure that everyone’s willing to put the time into it. Better yet, make a bunch of products. When you’re scraping by on hourly work it’s easy to say ‘this doesn’t pay, I’m not going to do it’, but look at it this way: Each of those products is a learning opportunity, and in consulting, if you don’t learn you die. One might even make some money.
Evangelize your successes. Write up each project that you do. Publicize the heck out of it. If you did something awesome and no one knows, it doesn’t matter.
Recurring income is what keeps consulting businesses afloat. Just because you, as a scrappy developer, think that support contracts are a ripoff doesn’t mean they are, and if they didn’t exist, most of the things you like wouldn’t, either.
Running a business is crazy hard, most of them fail, if yours doesn’t, good for you, but be open to the possibility that it should have.
Realize that you could very well be doing work in technology for the rest of your life. Take every opportunity to learn a new thing. The more you know, the more valuable you are, and in the end…
You are your product.
The Soft, Cozy Womb of Corporate Life
One upside to doing a bunch of projects for a bunch of people is that we met a bunch of other technology people. I did a few projects for Mitch Kapor (of Lotus fame), we had Matt Mullenweg in our office before WordPress got huge, I worked with the guy who designed Google+ on a project, and some guys we worked with are behind SB Nation, The Verge and Polygon. Once you meet smart people and show them you’re a decent sort of person, other doors start to open. These doors are sometimes soft, inviting, and open onto worlds of bureaucracy and 401k plans.
One of the last projects we did at Polycot before the founders went their separate ways was MindBites. MindBites is a video commerce platform, and after we built the prototype, we migrated the customer to a company called Squeejee for ongoing development work. A few years later, some folks from Squeejee would end up at Hewlett-Packard, brought in to spearhead HP’s push into the public cloud space. They would bring on Matt Sanders, and thanks to the good impression I apparently made, me.
I’ve been at HP for two and a half years, and I’m finding that a lot of the lessons I learned earlier still stand. Namely:
Look for a space where people believe in forgiveness over permission. Then do what you feel needs to be done.
Look for smart people and learn from them. Communicate. Converse. Network, even if it’s hard.
The people above you want solutions, so when you’re presented with a problem, come up with one, and do it.
People in the corporate world are used to passing the buck and bureaucracy They are impressed by responsibility and rapidly delivered solutions.
Take credit when it’s due, share it when it should be, make sure contributions aren’t overlooked.
Don’t let yourself get stuck. Corporate life can be a trap. A slow moving, slow progressing trap. Always be on the lookout for the next spot, the way to gracefully exit, the new problem. This counts double if you’re a startup phase person.
Many (most?) corporate projects get the axe eventually, some before they even ship. Don’t take it personally. Try to lead the inevitable downturn. Play from a position of innovation. If they want to kill it, be the person proposing the exciting new possibility.
Take every opportunity the company offers to learn, present, meet, train, etc. Just because you got a job doesn’t mean you get to stop hustling. Again, the cardinal rule is…
You are still your product.
Fill in the Gaps
If you’ve focused on the frontend, do some backend tutorials. If you’ve done HTML and CSS, try Drupal or Django or Rails. If you’ve done databases and integration projects, do some front end stuff. Look at jQuery. If you’ve done just web stuff try loading up a server, setting up backups, and installing software. If you’ve done server stuff, try creating some HTML5 Twitter mashups. If you’ve only done sites for a small set of users, go big, pull down some giant Twitter datasets and start playing with R and Hadoop. If you’ve used imperative scripting languages, try functional ones. If you’ve mainly done P-languages or Ruby, try Lua or Go or TCL or LISP. Write a compiler. Do some computer vision projects. Hack on Arduino or the Raspberry Pi. Write an Android app. Go outside your comfort zone.
As to how deep you get with these things, here’s an arbitrary rule of thumb I just made up: Learn enough that you could give a 45 minute talk about it. If you’re single, learn a major new thing every quarter. If you’re married, every 6 months. If you have kids, especially little ones, every year. Adjust as you see fit.
Above all, don’t beat yourself up if you find yourself behind some imaginary curve. If you’re 35 and only know Java, that’s fine. That’s great! There’s tons to learn, and it’s going to be crazy and exciting and you’re going to look at technology in an entirely different way. If you’re 50 and think you’d really enjoy this, there’s never been a better time to learn, and it’s never been easier to get from nothing to a working product. Try one thing. Pick one of these things that interests you, and spend a weekend on it, if you can. Commit to just getting one thing working. If you can relate it to your job and do it on company time, all the better. If you enjoy it, keep going.
Life Without College
So, I may have strayed from my original point about college. There’s a maxim I’ve heard that goes like this: Your degree gets you your first job, and after that it’s all about the work you’ve done. For people who don’t go to college, the trick is getting that first job, and filling in anything you may have missed by not going to school.
Look for under-saturated specialties. It isn’t a great time to get into small business web design. That ship has sailed to custom WordPress themes. If the web really floats your boat, get into Drupal, but don’t stay there forever. Technology moves, albeit sometimes slowly. Mobile development was ripe a few years ago, but making a profit in it is really hard. It’s a good skill to have, but a hard market to compete in. Look at things like RubyMotion, to get your feet wet. There are opportunities in DevOps (a fancy word for programmers who deploy their own code into production), big data, personal and business clouds, personal analytics, and integrated Internet enabled devices. There are always jobs to be had in enterprise software. Tech companies introduce software to solve new problems, so look at the announcements that are getting a lot of buzz. CloudFoundry had a lot of buzz, and now Docker is really hot.
Getting your first job:
Find something you feel excited about (programming, networks, server administration, HTML, design) and do a bunch of it. If you’re a lecture-learner, watch videos. If you need practical applications, ask people for ideas of projects. If you have collaboration skills, pair program.
Meet people. Go to Meetups. Join online groups. Listen a lot. Don’t be afraid to ask questions. Follow the rabbit hole down. Don’t be afraid of not understanding. The pieces will fit with time. You have to practice, though. You have to actually write code, create graphics, code web pages.
Share what you’ve learned. If you can teach it, maybe you’ve learned it.
If you’re programming, share your code on GitHub. If you’re creating videos, post them to YouTube and Vimeo (I partied with those guys once, they’re cool, but New York-trendy).
Ask for feedback, don’t expect it to be glowing. Don’t try and change everything, but internalize what you get. You’ve created something, don’t doubt yourself. The goal is to get better, not to be perfect.
Talk at meetups with people from companies you respect. Look for open doors, even if they aren’t exactly what you want. Expect to do a lot of hard work that isn’t glamorous and isn’t fun. It’s better to do less fun work at a company you love than cutting edge work at a place you hate.
Get some experience doing contract jobs, say, on oDesk. It will suck. You will hate it, but it will teach you about shipping code, supporting code, and dealing with clients.
Technology managers rarely care about the jobs you’ve had, and almost never care about what school you went to. They care about the work you’ve done. When I’m hiring now, education is nearly irrelevant. How you spent 4 years as an immature post-20-something is nothing compared to how you spent the next 5 or 10. Google seems to agree. Google has teams where 14 percent of the folks never went to college… Google!
Hierarchical academic environments still exist (HP Labs is really oriented that way, I’ve heard), and are probably places you want to avoid. Most places like this have a reputation for being so. If you ask around, you can probably get the skinny.
If the opportunity appears, jump on it.
Once You’re In:
Never turn down an opportunity to do something that excites you.
Find a mentor, someone who shares your interests and has experience. Don’t go crazy with their time, but don’t underutilize them. People who’ve been around for a while want to share what they’ve learned, but they want you to show initiative.
Find excited, cool people. If you’re in a corporate environment it can be easy to get depressed. Don’t be an antagonist. Be the person you want to hang out with. The future is wide-open and unknown. The present is temporary. Always be dreaming.
Take advantage of learning resources and your newfound credibility.
Watch for the phase changes. Be sure you’re where you’re most productive. Seek out managers who understand that personality fit, and strive to keep you there.
If you get hired with no prior tech experience, you probably aren’t going to make much money. Work on your skill set, network, and realize that you may need to join a different company to work your way up the salary ladder quickly. Learn to negotiate salary. Google it. It’s important.
Once You’re an Old Hand:
Share your knowledge.
Protect those below you. You’re experienced and have tough skin, sometimes they don’t. They need to know the realities, but they may not need to know how the sausage gets made.
Look for people who need mentors. Encourage them. Connect them with things you think will help them.
Take the time to learn about the people you work with. Everyone has a story. Maybe they didn’t get a CS degree. Maybe they’ve had similar challenges. Maybe they have an amazing background or skill you knew nothing about.
People come into technology with different skill sets There is no such thing as the complete programmer. Look for your own gaps and those in others, and figure out ways to fill them.
Lead by example. Do good work, don’t be a jerk, and treat everyone with respect.
A Few Last Notes
If I’ve learned anything in the last 15 years of being in technology, it’s that patterns repeat. I’m sure there will be changes in the future. Once you have kids, your desire to really jump on those transitions may start to slow down, but in the end they’re what a career is about. I’ve been fortunate to meet some very smart people inside HP who’ve been there for 30 years or more. They started out on calculators and are now in cloud. Maybe I’ll start in the web and end up in synaptic AI. Maybe that’ll be at HP, maybe it’ll be somewhere else. There’s always something new to learn, and there’s always that product of ‘you’ to work on.
If anyone reading this is looking for specific advice, needs a mentor, or would like some feedback, let me know. A lot of very gracious people have given me a lot over the years, and I want to pay it forward.
Last night after driving home from the Austin PyLadies meetup, my wife sat in our driveway for 20 minutes listening to the end of an episode of WNYC’s Radiolab. Later, after we’d headed to bed, she spent another 20 minutes retelling the story to me, minus Radiolab’s flourish and production. The story was still interesting second hand, and comes down to this (I’ll wait if you’d like to go listen to the episode of Radiolab, I’m sure it’s excellent):
Two people discover hundreds of letters from WWII on the side of Route 101. They’re from soldiers replying to a woman on the homefront. The soldiers call her mom, but she isn’t their mother. The two ask around, no one knows anything about them. One of them, a creative writing professor, ends up using the letters as projects for his students. He gives them a letter, and their task is to create a story around it. A soldier, a woman stateside, an unlikely connection. The other discoverer wants to track down relatives, she wants to uncover the truth. She ends up discovering it, but he’d rather not know. He wants the possibilities.
Even told second hand, the story stuck with me on a meta-level. There aren’t a lot of things that would make my wife sit in the car in the driveway for 20 minutes listening to the radio, but a good story is one. We love stories, we love it when they’re well crafted and well told. But we also love the possibilities of them. Sometimes we don’t want the truth, we want magic, we want to dream the dream of what could be. Sometimes the truth can’t exist, and the closest we can get is a dim outline of it. Sometimes the dream is better.
The Promise: Stories that Tell Themselves
A few days ago I ran across a blog post by Tynan Sylvester, a designer on the game Bioshock Infinite. It’s all about the dream of simulations for game designers, how we think that by creating more and more complex systems, we might eventually build a system that is complex enough to manifest stories. Austin Grossman’s latest novel, YOU, is about that, in a way. The protagonist is a game designer and the antagonist is just a manifestation of some long-running game rules. As game designers, we want to design games that surprise us. That’s the ultimate payoff, to build a game that entertains you, and not just a twitch game that is enjoyable for its mechanics, but a game with stories compelling enough to sit in the car in the driveway for 20 minutes at 9 o’clock at night.
Lots of game designers have tried to do this. Tynan talks specifically about systems in early versions of Bioshock where the player would have to play autonomous bots (splicers, gatherers and protectors) off each other to progress. They hoped that amazing, emergent gameplay would be the result. In the end it didn’t work, and the game moments that they’d hoped would happen spontaneously ended up being heavily scripted. Players crave story, but that story can’t be left up to their persistence and chance, especially when creating a commercial title. In that environment, a great story has to be guaranteed.
Dwarf Fortress: Madness in Text Mode
There are a few notable exceptions to this principle, and they’re mainly smaller games driven by singular minded creators. The best example of this is Dwarf Fortress, a massive and inscrutable simulation game where the the player takes on the role of an overseer, and the titular dwarves are simulated autonomous entities inhabiting the world. Dwarves have names and hair colors, what Tynan calls Hair Complexity, things that add perceived simulation depth without effecting anything else. (When was the last time you played an RPG where a plot point hinged on your hair style?) They also have more integrated systems like hunger and social needs. They have personalities, they get sad, and sometimes they go crazy. The dwarves live in a randomly generated world, so your game isn’t like my game, and even my second game won’t be like my first.
Dwarf Fortress has a very dedicated core following, and one of the reasons is that it really lives at the edge of apophneia, the experience of seeing meaningful patterns emerge from random data. At the core of Dwarf Fortress is a collection of rules governing behavior. A dwarf without food will eventually starve. A dwarf without personal interaction may eventually go crazy. Dwarves are scared of wolves. Dwarves exist in a world generated fractally, a world that feels real because it mirrors patterns in nature. Therefor, as more and more rules get layered on, and more and more people play more and more games and get better and better at creating experimental mazes for these digital rats to play in, stories begin to appear, or so we perceive.
Dwarf Fortress didn’t generate these stories, though. People played the game, sometimes hundreds or thousands of times, and while gazing into the mandala of the game, they nudged and pulled the threads of the world and created stories based on the events that occurred there. Dwarf Fortress isn’t a windup toy, it’s a god-game, and the players impact on the game world is more than negligible. The stories generated there are as much created by the players as by the game.
I Fight For the Users
While my wife was out at PyLadies last night, I coincidentally watched TRON: Legacy. It occurred to me as I was thinking about writing this post, that it’s a movie about this possibility: The dream of a world inside a computer, a world created by a brilliant programmer, a world that once set in motion can create stories, unexpected events and enthralling narrative. The creator steps aside, and no longer controls the game from the top-down. The creator becomes a god among men, watching things unfold from their level.
In TRON: Legacy, the magic of digital life comes in the form of Quorra, the last of the ISOs, Isometric entities that appear spontaneously from the wasteland of the computer. Digital DNA, digital life. Enough rules, enough circuitry, enough care and magic happens. That premise is exciting, and to programmers it’s intoxicating. For those of us in the digital generation, that’s the dream we live with. That’s what we keep trying to make happen wherever we go and whatever project we work on, be it big data or software bots.
But the lone programmer, no matter how brilliant, and working for no matter how long, can only produce so much code. Stories from one person only grow so far, only change so much, and rarely surprise and enthrall. Dwarf Fortress as a dwarf isn’t a game most people would play. It’s hard to see the overall story, and the game isn’t good at presenting it. But if there were more players…
EVE Online: More Interesting to Read About Than to Play
If it’s possible (albeit insanely difficult) to have stories appear in a single player game, it must be easier for stories to manifest in a multi-player game, right? Games like World of Warcraft have largely fixed, planned out stories. It comes back to the challenge that Bioshock had, complex systems are exciting to designers, but players want immediate story gratification. Complex systems take dedication to understand, dedication most players don’t have. When new multiplayer games are announced they sometimes hint at players making a real impact on the world, but those systems usually fail to live up to the hype. The latest game to promise this is The Elder Scrolls Online. We’ll see if they can do it.
One game that does this and thrives is EVE Online. EVE is a massively multiplayer online space combat simulation, one that spans an entire universe. It’s possible to play EVE as a loner, but it’s also possible to align yourself with a faction, and have your small efforts merge with hundreds or even thousands of others to build armadas and giant dreadnaught ships, to control entire solar systems and even galaxies. The designers and administrators of EVE take a largely hands-off approach. They don’t want to kill the golden goose, so they design the game for balanced conflict, and let the players sort it out.
You could say that EVE is a computer program for generating stories, and in fact the’ve even made a deal to do a TV show based on player stories from the EVE universe. Except again we find that that EVE isn’t the thing generating the stories, EVE is just a place where the stories happen. To a player only experiencing the events inside the game it may seem mysterious and amazing, and it certainly is to those of us who read about the events afterwards, but it’s really just a sandbox. People play pretend with enforceable rules, but you can’t separate a story that happens inside of EVE with the real life stories that happen outside of it: The scheming that happens on IRC or in forums, the personal vendettas, the flexible allegiances and the real-world money that flows through the system. There’s no way to watch something occur inside of EVE, and even if you had perfect clarity on everything that happened inside, have any way of knowing for sure what really caused it. If you take away the players, the legions of dedicated fans scheming and plotting, you just have an empty universe.
Facebook and the Timeline of Truth
I think a lot of web developers secretly wanted to be game designers. Becoming a game designer is difficult, there aren’t as many jobs and the hours are terrible. Instead we build web sites, but we’re building systems too, and we want to tell stories.
I joined Facebook back in April of 2006. I had a @swt.edu address from Southwest Texas State (now Texas State University) from an extremely brief stint (sub 1 day) as an IT staff-member, so I got in a few months before they opened it for everyone. Getting into a new, exclusive social network is a bit like finding a new simulation. We hope the software can tell us new stories, that it can make some sense of the data it has. With Facebook the promise was that if it collected enough information about us, it could tell us that magical story. That’s what Timeline was supposed to do. Give Facebook enough photos, enough checkins, enough friend connections, enough tagged posts and it would be able to tell the story of our lives.
In the end, though, Timeline doesn’t tell you a real story. It reminds you of stories you’ve heard and experienced, but Facebook is only a dumb algorithm working with imperfect data. It’s smart enough to target ads, but it can’t understand the meaning, and it can’t remix the data in really compelling ways. It can’t be Radiolab. Most of the time the prioritization it comes up with I just want to turn off. Its attempts at story are so bad I’d rather use my own organic cognitive story filters.
With every new Facebook feature announcement, with Google+ or the next thing that processes all your activity, the promise is that the system can get better at telling those stories. We want to believe it will happen. We want to believe that a couple thousand web developers and a couple billion dollars could create a story machine, but I’m not sure it can. I was reading an article about HP’s R&D budget the other day that said Facebook invests 27.5% of revenue in R&D, a larger percentage than any other company they tracked. You can bet a good chunk of that is going towards the search for story, in some form or another.
Weaving a Web
I’d be remiss if I didn’t mention Weavrs at this point, since they are essentially digital actors that derive stories from the mess of social media. Weavrs are designed specifically for apophneia, they produce content one step up from random, and rely on our desire for patterns to throw away the things that don’t fit. We project stories on to them, and for a project with the limited resources that it had, it’s exceedingly good at it.
My weavr twin is posting about HP Moonshot servers. That’s almost eerie, but it’s also posting about hockey tickets. The story makes sense if I’m picky about the things I include, but it isn’t an internally consistent narrative. The narrative is impressed on it by the people who see it, like reading digital tea leaves. Your story of my weavr is different than mine.
With enough resources and time, weavrs might become a real story machine. That’s a moonshot program, though, and I don’t know who’s going to step forward and make that happen. Investment follows money, and right now the money is racing towards big data.
Autonomy’s main product is called the Intelligent Data Operating Layer, or IDOL (symbology, ahoy!). They call the processing of information with it Meaning-Based Computing. From what I’ve heard it’s certainly good at what it does, but while it promises Meaning from Data, and that promise separated HP from 9 Instagrams or 2,500 Flickrs, there has to be some apophenia at work here. Just like watching solar system battles inside of EVE gives you a piece of the story and playing hundreds of games of Dwarf Fortress will result in games worth telling stories about, the system data is never the entire picture.
I really like Stephen Wolfram. Stephen believes in the fundamental computability of everything. While I love reading his blog posts, and I am interested in and admire his idea, I have to wonder how far the hyperbole is from actual execution. Given enough computable facts and enough understanding about the structure of narrative, a perfect Wolfram|Alpha should be able to tell me stories about the real world. But it can’t. They aren’t even trying to approach that. Wolfram|Alpha isn’t creating Radiolab. They want answers, not stories. You know what tells stories? Dirty, messy, all-too-human Wikipedia.
A Different Kind of Magic
My friend Matt Sanders works for a bay area company called Librato. Librato is a big data startup, having pivoted from some other work to running a service that collects vast amounts of metrics and provides dashboards on top of it. With Librato Metrics you can feed data points, set alert triggers, create graphs, and watch activity. It’s big data without the prediction. It promises no magic, but relies on our own. It optimizes data for processing by human eyeballs.
The 3 pounds of grey matter between your ears is still the best computer we have, running the best software for deriving stories and making sense of data. Librato works because it doesn’t try to be what it can’t. Google Analytics tries to offer Intelligence Events, but more often than not, it can’t offer anything more helpful than that visits are up from Germany 34%. You would think that by combining traffic source analysis with content changes and deep data understanding Google would be able to tell you why visits are up from Germany, but most of the time that basic percentage is the best it can offer. It still takes that 3 pounds of meat to pull together the data and interpret it into a story. While computers may be generating articles on company reports or sports games, they’re not creating Radiolab.
I think there’s still a lot of room for innovation here. The Archive Project I dreamed of long ago is essentially a system for telling stories and discovering meta-stories. Maybe someone will finally build it. Maybe the next Dwarf Fortress will be a world that runs persistently in the cloud, a world where our games interact with other people’s games, where crowdsourced Hair Complexity snowballs until you can get lost in the story if you want to. A game where if you want to turn off a random path and follow it down to the river you’ll find a fisherman who will tell you a tale interesting enough to make you sit in your car for 20 minutes, enthralled by a narrative.
Maybe the framing of a story is what big data needs to become personally relevant. Maybe that’s its magic trick. Maybe narrative is the next great big data frontier.
I sometimes wonder about the generation of kids growing up today, in this big data, analytic-driven, always-on world. I wonder how they will embrace it, like we embraced computers and connectivity. I wonder if they’ll have the ability to hear the prognostications of the computer, to listen to the story from the machine, and consider it a kind of truth. To internalize it, but also keep it separate. To know the machine knows a truth, but not necessarily the absolute truth. Maybe that will be their power, the thing they can do that those of us from the generation before can’t. Maybe that is where the dream finally comes true.