Fail (too) Fast - Startup Week One

Fail faster, that’s one principle of the lean movement. You can read The Lean Startup and you can feel great about how much more efficient you are. Just follow the Build Measure Learn cycle and you’ll never waste time building something that people don’t want.

It’s simple right? A week ago I left my job to work on a startup full time with a friend. We’ve talked about trying it for years and the timing was finally just right. We’ve had a few ideas floating around and we started with what we thought was the strongest one. One week later and here’s where I am:

Form a hypothesis

Existing server configuration management tools like Puppet are too difficult and expensive to implement. Developers and sysadmins will buy a solution that removes that pain and expense. We can build a simple, sustainable business serving these people.

Create an experiment

MightyOps let’s you control your server configuration, automatically as you work.

Learn?

Initial feedback ranges from “awesome” to “useless” and is generally lukewarm. There are valid concerns, misunderstandings, suggestions for small and large pivots and a smaller than expected conversion.

Trough of Despair

I’m confused, where’s my black or white, fail or pass answer? This is supposed to be science right? This is worse than concurrent programming. I psyched myself up and threw myself out there so where’s the payback?

I want to say the idea was dumb and I had no idea what I was thinking, I want to kill it and pretend that I’m failing fast. I want to pretend that I’m being rational and objective and I’ve learned from this mistake. I fear that there’s no product, no market, and definitely no fit. I’ve committed myself to this path, my savings account will only last for so long and I can’t afford to spend another minute on this lame idea. I have to throw it all away and start over, learn from my horrible mistakes.

But really I’m just afraid. Afraid that the world outside is a gray fog of both encouragement and setbacks. I’m thinking of starting over not because it’s the right thing to do, but because I know how to start and I don’t know how, exactly, to continue.

So I promise myself that I will stay determined, I will not kill ideas out of fear, and I will fail fast but not too fast.

Discuss on Hacker News.

Now or Never

Long before I start looking for a new job, I get that inkling that things are coming to an end. The shine wears off, it’s difficult to get out of bed, tasks that should be interesting seem pointless, and lunches turn into bitch sessions. Sometimes my discontent is my fault and other times it’s out of my control but either way it still makes me feel like shit.

It sucks leaving smart people from whom I still have so much to learn. It sucks leaving a company that I know will succeed. It sucks leaving all the cool technology I’ve built and will no longer have the chance to build. But I’m unhappy and that sucks too.

Most of all, I’ve hated that nagging feeling in the back of my mind that tells me I may not be able to find another place that makes me as happy as this one once did. That feeling that I keep seeing the same mistakes being made and believing I’d avoid them. That feeling that I need to do something different, experience something new, learn hard lessons, and fail or succeed by my own volition.

I can’t ignore that fucking feeling any more.

Bootstrap an Audience

Louis CK has been all over the internet this past week. First he wrote a very personal missive urging people not to pirate his new show, he was on Reddit, and now he’s shown that this experiment payed off.

When Kevin Smith decided to independently distribute and market his movie Red State, many people said he would fail miserably. But he did the math and he knew he’d be able to make back the money for his investors because he’d amassed such a large, devoted, core audience.

Trent Reznor and Radiohead did it before them. And the response on the internet to all of these guys has been overwhelmingly “well of course it works for them, they already have an audience”.

But what no one seems to be mentioning is that an audience can be built, without too much work.

You don’t have to be famous

These guys in show business had to grind away for years to get their thousands or millions of fans. But now you can look at youtube or twitter and see thousands of people you’ve never heard of with millions of viewers or thousands of followers. In a fraction of the time it took Louis CK to get his audience, people all over the internet are gathering fans. According to Kevin Kelly it only takes 1000 true fans to reach critical mass, so:

  • Engage with a community
  • Do so on a consistent basis
  • Have fun

Look at Patrick McKenzie, Zach Holman, Sahil Lavinga, Jason Roberts, and Notch. These are pretty normal guys in our community who have decided to share their knowledge, build neat tools, engage us with weekly podcasts, or share awesome code with everyone. In exchange we follow what they do, listen to what they say, and buy what they sell.

It’s hard work, and it’s easy to let slip, but I truly believe that these days anyone can get tens of thousands of people interested in what they are doing.

Oh, and follow me on Twitter and tell me why I’m wrong.

Instance Variable Shelf

Too often I see code that does the following:

declare('Foo', {
  constructor: function(input) {
     this._input = input;
     return new Bar(this._input)
  }
});

Chances are that Bar also copies input into an instance variable. Soon there’s a chain of three or four classes all maintaining input’s state. If Bar later updates input with a new struct, all of the other copies are stale and invalid. The only solution at that point is to bubble a change event throughout the tree so that each instance gets updated.

But it’s totally unnecessary. Foo likely doesn’t need to keep a reference to input. If it does need it for some operation later on, Foo should ask Bar (and Bar should ask whomever is holding the canonical copy).

This is why the functional programming people say that mutable state is the source of most bugs. This may seem obvious now, but it’s incredibly easy to make this mistake if you aren’t paying attention. Think really hard before creating a new instance variable. Think extra hard about creating a second reference to that variable anywhere else in your code. Instance variables are longer lived than you expect, don’t use them as a shelf.

Simple Ain’t Easy

We watched another video during our Lunch and Learn Fridays. In this one, Rich Hickey masterfully draws the line between simple and easy. It’s been over a month since I first saw it but his words are still burning in my mind.

Hickey’s talk focuses on two common words; simple and easy. Think about easy as near at hand or familiar and simple as uncomplicated. And while we rarely think about the differences between the two, it turns out that in software thinking about simplicity as separate from ease is quite important.

The question that I keep asking myself is why is it so hard to make something simple?

Easy: near at hand

In software, easy is king. There’s a sense that if your language, tool, or project can’t demonstrate its superiority in the first paragraph of its website, then it will languish and die.

So the next time we start a project we ask ourselves, what’s easiest? We remember the latest hyped thing that promises to get started in five steps which is one less step than the previous hyped thing.

I’ve seen many arguments for a tool based on how easy it is to use. I’ve also seen those arguments tumble months later when the tool has had to be monkey patched to hell in order to fit requirements. This happens so often that it feels inevitable, but it could be that ease is just a really poor justification for using something. If anything, thanks to Hickey, I can now articulate why it’s such a poor choice.

Simple: one braid; uncomplicated

So, Hickey’s argument is that if you confuse easy with simple, your project will become complicated, complected and intertwined without hope. On the other hand, if you understand the difference and focus on simple, you will make better decisions that will create a healthier project. The solution to maintainable and sustainable software is simplicity not ease.

Just try to do this. People have been preaching for simplicity forever. We have tools, practices, and laws that promise to decouple concerns. Yet when Hickey asks his audience if they’ve ever worked on a simple project, properly modularized and abstracted, he needs to raise his hands twice because no one else had seen such a thing. Are most of us too stupid to write a simple system? Are we too lazy to use the tools at our disposal?

It’s actually not difficult to spot complicated code. In the past, if the code was also written in a consistent, familiar way I’d let it go and move onto the next thing. Recently I’ve tried to tackle my complex code and I find that I quickly run into the framework’s complexity. There are ways around the problems but they’re ugly or hard. Seeing code with this new perspective is painful.

Yet, when I’ve taken the time and have been able to simplify a problem, I find that I’m rewarded with that wonderful, fleeting feeling that what I’ve created is correct.

No Pain, No Gain

I feel that the problem is that the tools we currently have to tackle simplicity are not easy. It takes time and effort to grok design patterns. It is hard to follow the law of demeter if everyone around you is building things the easy way. It is far easier to create mockeries if you’ve never used dependency injection. Hell, it’s even easier to complicate unit testing with maintaining a test database and fixtures.

There are tools that provide simplicity, but they rarely reach that tipping point of mass adoption.

So please, watch the video, simplify your code, call out libraries and frameworks that are easy but complicated, and help build new tools that focus on simplicity. Make it simple then make it easy.

Product Kata

A friend told me that he would never build a new product from scratch again. It’s painful, annoying, and you never know what you missed until it hits you at 4am. Day. After. Day. Week. After. Week.

The whole process of releasing something is so full of complicated, little, nit picky details that I can’t help but feel like I’m spinning my wheels. This is really difficult because I’m also working a full time job. It feels like every hour I can spend on a side project needs to be productive.

Building Frustration

Two years ago, I had what I thought was a great idea. I talked to potential customers and there seemed to be consensus that there was a business there. Then I tried to build out my idea as a minimum viable product (mvp). I figured I could have something running in a month. Eighteen months later I had only spent maybe 80 hours on the project. It was something I thought about all the time, but it was incredibly difficult for me to work on it during evenings and weekends. I felt as though every time I touched the program I’d run into a problem or limitation, not with the concept, but with some third party service or library.

As I kept eating up my time reading documentation, digging through source code on github, weeding through stackoverflow, and fighting installation dependencies I started to doubt the fundamentals of my idea. What was my engine of growth? How could I attract initial customers if it wasn’t a problem people searched google to solve? Instead of being able to test any of this, I became more and more frustrated until I stopped working on it altogether.

People say that just releasing something is one of the biggest hurdles. I believe that releasing something is difficult because most coders only ever work on existing products. The foundation of a program is usually only ever built once, usually by a single person, and rarely documented.

There is very little novel code in an mvp; it’s all wiring and dependencies. This is almost (or should be) the exact opposite experience of working on existing products. At work, I’ve always coded on or around existing products, it’s fairly rare that I run into a situation that something completely new has to be built. There are libraries and tools and conventions you can fall back on when working around something someone else has built. I suspect this is likely a reason that many bootstrapped businesses come from the world of consulting and contracting, since they are more likely to be involved with projects during their inception.

Build Everything

So my new plan is to become adept at building. I want to take my simple, stupid ideas and dedicate a weekend or two to getting as far as I can to a released product as possible. Then I will leave it where it is and move on to the next idea before I get frustrated and never want to work on it again.

In the same way that there are code katas, I am practicing product katas.

Right now, I’m nowhere close to releasing something in a weekend, I’m not experienced enough in taking these raw libraries and services and building a customer ready application. But every time I build out a new idea I get much further and learn to solve new problems.

The first Rails app I created took days for me to really figure it all out, now I can prototype something in minutes. I ran into problems deploying to Heroku initially, now with a few apps, there’s no problem. It took me weeks to learn the different components of AWS, but if I want a new instance I can have it without thinking. I could name dozens of problems that I’ve run into with each idea I’ve tried to implement, but with practice I’m learning to solve them more generally and quickly.

Eric Reis talks about the build, measure, learn cycle. The faster you can loop through the stages, the more likely you’ll build something that people want. The more ideas that you can build, the more likely you will find one that succeeds.

They say practice makes perfect. I’m just trying to clear that first hurdle.

Why we are Still Deploying Overnight

(and what we’re doing about it)

So a co-worker sent me this blog post by Brian Crescimanno today that asks: why am I still deploying overnight?

Well, it’s a question that’s been bothering me for months. Sometimes when I bring up this question I find good reasons, other times I find excuses, and other times I find something no one has really thought of. Looking at the comments on that blog post there seems to be a lack of concrete examples about how people can deploy during the day or why they are stuck deploying at night.

I’ll run over our deployment process, try to explain the historical reasons for what we do, and then talk about what we plan to do about it.

Now, a lot of the advice surrounding the aforementioned post is that how you get to a point where you can deploy your application during the day is very specific to your application. People are saying that the problems your company will face have to be taken case-by-case because they’ll be so much different than another company. I think that’s a bit of a cop out, let me know if our situation is at all similar to yours.

Our application is a single page web-app with long lived (12+hour) user sessions. Our clients run offices in North America so typically we see user activity between 6am and 8pm, Monday to Friday. Since we are a single page application, our compiled Javascript file is quite large and we do our best to make sure it is cached on clients’ browsers (we use a url like app.js?version=1.01 to ensure that the caches are invalidated after each release). Because these offices depend on us completely we cannot take the site down during that 14 hour window, so if we are going to do a deploy during the day, it must be done without downtime.

The Deployment

  1. On the Thursday before the release we test our data migrations on a snapshot of client data. This is necessary because too often we were running into situations where a migration would work fine from development to staging but a particular client’s data would cause the migration to fail.
  2. On Friday a script is run to prep the release branch. The script
    • checks out the next release
    • minifies the Javascript
    • creates a tarball and copies it to our app servers
    • extracts the tarball beside the running code and then left alone
  3. On Sunday night a script is run that
    • logs out any users
    • swaps the site in Apache with a “we’re down for maintenance” site
    • turns off db replication
    • creates a snapshot of our database
    • runs the data migration
    • restores db replication
    • changes the symlink for our app from the old release directory to the one created on Friday
    • restores the Apache site
    • does a quick smoke test

The Friday step could be done on Sunday, but the Javascript build and propagating the tarball takes about 40 minutes that no one wants to spend Sunday night doing.

What is stopping us.

So what issues do we face if we were to try to deploy during the day?

  • We can’t sign users out during the day. Right now this is a cheap way for us to force them to redownload any new client side files.
  • We can’t use a “we’re down for maintenance” site.
  • Currently, we turn off replication because it would allow us to promote the slave if the data migration fails. We can’t lose a minute of customer data so replacing the master db in this manner wouldn’t work during the day.

So, in general we have two core issues and two tangential issues:

  1. A client side browser cache that references our old version.
  2. Database migrations that could potentially fail, lock up the application, or cause inconsistent data to be created while the migration was taking place.
  3. Our QA department currently does a full sweep test for regressions before each release, this takes at least two weeks but most people here feel it is a necessary evil.
  4. Our current development process is also focused around releasing groups of features, a single patch release is currently cumbersome.

As I said, I and others at work have been thinking over these issues for awhile. Here’s what we’re going to do.

First Steps

  • Separate some of our clients to point to a different app bundle so we can release to a subset first. We’ve actually had this for a long time but never use it.
  • Start with trivial changes. We’ve defined this as: ** Nothing that affects the REST interface, so as long as the json structures going back and forth remain the same, we avoid issue #1 ** No database migrations - This way we completely avoid core issue #2 ** The trivial change can not do more than one thing. This should alleviate #3, QA can do a branch test and we as developers have to be extra careful and guard against regressions with functional tests. ** We’ll just have to deal with issue #4 as necessary

Down the Road

Both issues #1 and #2 are handled by making sure the application is both backwards and forwards compatible.

In regards to the Javascript cache, this means that the data consumers need to be flexible in regards to added or missing fields. I think we are pretty close here. Using a setAttributes on a model is one side of the equation, checking if fields exist before they are used on the client side is the other. I imagine we’ll run into other issues here but if we remain responsive I’m sure we’ll be ok.

For the database, we need confidence in our migrations, already we test before deployment so that helps. In general, our migrations are adding new tables or columns, creating new foreign keys and other constraints, or optimizing stored procedures. I think each of these cases are generally compatible but we will have to watch for table locking. Our segmented users will alleviate a large part of the risk once we choose to run these types of migrations during the day.

Renaming columns and changing their data types or underlying meaning is a more difficult problem that I don’t quite know how to tackle yet. I’ve seen some vague strategies using db views or staged migrations where data exists in parallel for a few releases. If anyone has information about handling these types of migrations with no downtime, I’d be very interested.

Issue #3 is clearly an automation problem, we’ve started writing Sikuli tests but we’ve got a long way to go. The more we automate, the more we can expand the scope of our daily releases. Until we have confidence that regressions are being caught, we likely have to stick with the simplest changes.

As far as the development process is concerned, we have two intertwined issues, continuous deployment and during the day deployment. I believe that by adapting to releasing more features during the day, we will naturally evolve a process that will allow us to do releases more often. To do continual deployments of features will require reworking a large part of the development process. We use a git-flow model, while nice for more traditional releases, is a bit complicated when doing hotfixes, maybe we want to switch to a GitHub-flow…

Baby Steps

Over the last year and a bit we’ve gone from deployments that took four to five hours with numerous manual steps to well defined and scripted bi-weekly release. It’s difficult to take an existing process and an existing culture that’s always done releases a certain way and move them towards something new.

Being aware of your deployment process and why you’ve built it up in such a way will help you find the way forward.

Java’s JPA for Rails Fans

I’ve been thinking about ORMs lately.

I like Rails, it’s super quick to get up and running, the convention over configuration idea was brilliantly executed and has been endlessly copied. I believe that Rails forever changed the way that we write software.

I also like Java’s JPA (Java Persistence API)[1]. I hope to show those of you familiar with Rails and ActiveRecord why the JPA way of doing things is worth a look.

I’ve never seen a direct comparison between ActiveRecord and Java’s JPA. So I feel it would be interesting to start there.

In rails we have our models:

class User < ActiveRecord::Base
  has_and_belongs_to_many :todos

  def markAllTodosComplete
    @todos.each { |todo| todo.complete }
  end
end
class Todo < ActiveRecord::Base
  def share(user)
    user.todos << user
  end

  def complete
    self.is_done = true
  end
end

In Java things look pretty similar [2]:

@Entity
@Data public class User implements Serializable {
  @Id
  private int id;
  private String email;

  @ManyToMany
  private Set<Todo> todos = new HashSet<Todo>();

  public markAllTodosComplete() {
    for (Todo todo : this.getTodos()) {
      todo.complete();
    }
  }
}

@Entity
@Data public class Todo implements Serializable {
  @Id
  private int id;
  private String description;
  private boolean isDone;

  public void complete() {
    this.isDone = true;
  }

  public void share(User user) {
    user.getTodos().add(user);
  }
}

They are actually very similar, if you were to swap out ActiveRecord for DataMapper, they’d be even closer. For anyone who stopped using Java in the days of EJB2.0 and insidious xml, this might be quite refreshing.

However, I feel that JPA has a clear advantage over the Rails way. JPA defines annotations to define their persistence units to be managed whereas Rails uses inheritance.

This is a really powerful feature. The models do not know they can be persisted, it’s not their responsibility. They do not know about the join table for a relation, they just have a field of type Set. The models can create an inheritance structure that makes sense for the domain instead of being forced onto an enormous superclass. This is what Uncle Bob is talking about in this short video.

The models are bootstrapped into the persistence layer with annotations instead of chained to it with inheritance. With ActiveRecord it takes experience, discipline, and a realization that things can be better. Corey Haines gave a thirty minute talk complaining about how slow Rails unit tests are and how he uses a simple strategy to make them nigh instantaneous. The tl;dr of the talk is to separate your business logic into a mixin you then include in your models. The model takes care of persistence, you leave that to ActiveRecord, and your unit tests worry about the business logic.

When you use JPA this is the default. Unit tests run quickly because they do not need to spin up a container, you deal with plain old Java objects. You can swap out the JPA implementation without changing a single model because the different frameworks have agreed on this API and because the annotations are merely seams on top of the code where the containers can place their hooks.

JPA enables all of this but similar designs in the dynamic language frameworks are rare. I am not aware of any other than PHP’s Doctrine. I wrote about Rails here, but Django and all of the other frameworks that force your models to inherit from a god class are asking you to tightly couple your code to their implementation.

So what am I missing here, why have so many communities ignored the JPA pattern? Why shouldn’t we use something like annotations to bootstrap our models?

[1]: I’m talking about the JPA (the standard) as if it is an ORM, but feel free to use Hibernate (an implementation of the standard) in its place.

[2]: Ignoring imports and using the great project Lombok for @Data. Also (since I really didn’t feel like downloading Eclipse and Glassfish and creating a new project and downloading jars and fighting with classpaths and because I haven’t used Java in a few years) I wrote the code in vim and can’t guarantee correctness.

Lunch and Learn: Articulate Coding

We get lunch brought in to work on Fridays. I took the opportunity to show great videos to the team while we all ate. It’s a lunch and learn with zero effort.

Today we watched Kent Beck’s QCon talk on Responsive Design. It’s certainly worth checking out if you’re bored, although it is a bit slow in parts and the non-mic’d audience questions do get annoying.

There were two really great ideas in the talk that will probably keep me from sleeping tonight. The first was the concept of design principles. Beck says that a design principle is what you fall back to when patterns fail you. Principles help cull the decision space and narrow your design to something that is less likely to bite you in the ass down the road. He suspects there are a number of universal principles like “don’t repeat yourself” but also personal principles like “make safe changes” and business principles like “our software exists to assist people to make decisions”. The subject of Design Principles is easily the topic for another post.

The second idea that really struck with me is the anecdote near the beginning of the talk. Beck talks about wanting to deconstruct his own design philosophy, wondering if it was possible to actually break down every decision he makes when writing a program into a set of personal design patterns or strategies. So he set out an experiment.

Beck’s experiment is simple.

He would type a few characters and then stop. He would think about those characters and determine what pattern they were expressing. Then he thought about why he chose that particular pattern. Next, he wrote down his findings and typed a few more characters. He would only continue if he had documented the new pattern or if he had already encountered the pattern previously.

If you think about this, it’s a pretty crazy experiment. When we program, we face multiple decisions on every line. For instance, imagine this method declaration in a typical oo language

public function doSomething($id, $job, $newname = 'Anonymous') {
  • Why did I explicitly set the scope to public?
  • Why did I name the function like this?
  • Why did I use camel case for the function name?
  • Why didn’t I define types for the parameters?
  • Why did I put the parameters in this order?
  • Why did I choose these parameter names?
  • Why didn’t I use camel case for the last parameter?
  • Why did I put the open brace on the same line?
  • Why did I use single quotes instead of double?
  • Why did I indent?
  • Why did I use spaces instead of tabs for indent?
  • Why did I use four spaces for the indent?

Here are twelve minor decisions[*] for a line containing only seven words, more significant design principles like dependency inversion or single responsibility haven’t even been touched. You quickly realize that the number of instincts and habits that we all have as coders must be enormous. We arrive at these patterns deliberately, through convention, and by accident. For example, I deliberately put the $id parameter first because I know I’m building an api that uses that same parameter everywhere and it will maintain consistency for those calls. I use camel case and leave off the typedefs because that’s the convention in the team’s code base. And when I look through the rest of my code I find that my use of single quotes and double quotes is completely random, I suspect that my tendency to switch languages often has made it difficult to build up a consistent pattern in my mind.

Now, Beck continued to document these decisions as he built his application. He found that the same decisions began to crop up more and more so that by the end he was able to code without documenting anything new. At the end of the project he found that his personal design patterns were now mostly written down. More importantly, he found that he was better able to speak about, defend, and consciously review those decisions.

He became more articulate about his craft.

Remember, this is a leader in our field with decades of practical experience who’s written many easily digestible books and papers about deeply complex problems. Despite all of that, he still found that this exercise helped him communicate on a whole new level.

This really resonated with me because I feel that I am not always as articulate as I should be. I find it very easy to wave my hands and say this code smells, that’s a good idea, or I’d rather do it this way without really understanding why I say it. The best developers that I’ve worked with always find a way to explain not only why their design decision will work in a particular case but why at a fundamental level it will be more robust, maintainable, and readable. I sometimes find that magic thread of reasoning and am able to thread the argument with the correct words to explain myself exactly. But those moments are fleeting. Hopefully, through deliberate practice I will be able to improve my signal to noise ratio, stop the hand waving, and articulate my positions with more reasoned arguments.

Personally, I can’t wait to try this experiment. Maybe not for an entire project, but a new class file could provide me with some good material for introspection.

As I kick off this blog, with many of the same goals as I hope to gain from the experiment, I must remember there are no silver bullets. If I want to be able to speak about my craft more eloquently I have to speak about my craft more often. Although I may think about development every day, I hope that putting my thoughts down in writing will help me to articulate my code.

Follow the discussion on Hacker News

[*] I can think of at least four more decisions I made when I wrote that line, how many more can you point out?