Author: nancyhua

It’s been almost 2 years since we started Apptimize. I think part of how we got here is by not knowing how hard it was going to be, like Columbus blithely sailing to India without knowing what the heck he was doing. When I heard the story about Columbus as a kid, I thought, “What an idiot.” Who starts sailing to somewhere with a bunch of ships without knowing the way? But every day we all launch towards a new world and rewrite the maps as we go.

When I started Apptimize I knew what our core strength would be: technological superiority. After working with a top team at GETCO where we were inventing technologies decades ahead of what anyone had, I knew how big a difference the team made and started recruiting the best people I knew from MIT and YC. I asked everyone who the best engineer they knew was and why. The first people we worked with were people whose references considered to be “un-hirable because they’re so good they just want to do their own thing.” Nevertheless I convinced some of them to join Apptimize. Everybody we hired, I made sure they were on board with the vision of inventing technology that would transform mobile innovation. Improving other people’s ability to innovate was what I had concluded was the highest leverage thing I could work on.

I’ve been lucky to have always worked on teams that are really good at engineering, and when you have a hammer, everything looks like a nail. For ages I viewed every problem as a technological issue. Didn’t matter what it was, technology will solve it. (Abortion? Duh, just invent a machine that sucks the fetus out and incubates it somewhere so you don’t have to choose between the rights of the mother and the rights of the child. Middle East crisis? Simply invent a machine, etc, etc.)

One of the first users who deployed Apptimize was a company I’d never heard of. I looked them up was surprised they simply put us into their very popular app, especially since this was a huge company that had just IPO-ed. Awesome! Basically they realized we had the best technology, way better than they could build in house. We didn’t need to do any marketing- they somehow found us despite us not being in the top search results for anything including our own name. They signed up despite our terrible website. They managed to integrate our SDK without any documentation and they deployed without ever talking with us. This was working.

Then our dashboard went down just as they were logging in to check out some test results. They were disenchanted and didn’t try to log back in for months.
“Sorry our dashboard was down for a few hours. This will never happen again!”
Silence.
I reasoned the problem was that they lost confidence in our technology. A crack in one area signaled fissures in other spots. The solution was to attack the technology even more and make every single part amazing.

We worked on the product. I kept the prospective customer updated on all the new stuff we were doing, “We just added this feature that could be useful to you guys that no one else can ever build!” Eventually they were sufficiently impressed by our new features that they tried to install our update to use us again. I was surprised by how willing they were to try again considering the activation energy required to move their giant organization towards this idea again. “They must really want our product,” I thought. “They can’t build this in-house so they’re willing to endure a lot to try to use us.”

Then they couldn’t figure out how to install the new SDK because it was different from the old SDK. We explained, “The new SDK is fundamentally different. It’s better and does not require any programming at all to install. Can you start with a clean app and just follow these instructions? This SDK is easier and will give you a lot more power.” Awesome, right? Everyone else who’d tried the new SDK loved it. The number of support emails about how to install went to almost zero.

This company was different. They weren’t getting it and kept trying to install it the way they installed the first time.
“I’m not seeing the place to add the code snippet.”
“…There is no code snippet. Can you take out any code snippets you still have in there?”
After some back and forth, they gave up again.

After all the energy we put into making everything super fast and easy, after all our talk of saving our users time, they said, “We don’t have time but maybe in a few quarters.”

I lost 10 pounds of delusions right there. I learned a lot from that experience about a lot of different areas and don’t think we really made that many bad decisions, but I do think my perspective had been wrong in many ways. After all the resources we devoted into simplifying our deployments and making our product better and better, we discovered that the one customer it failed for was the one customer for which it mattered most. I had believed that if we had a product people wanted and the best technology, then we’d be good to go. I’d believed that the quality of our core features would be tested after the user set up the SDK. But that’s not what happened at all. The gap between reality and my expectations was oceanic. When you’re building a startup, a million things like this happen every day that teach you things you’d never learn in school.

One gap between school and reality is the way you’re judged on your work. In school there usually aren’t instances where everything relies on you getting the right things right at a particular moment that could happen at any time. Teachers are like, “Well, you got this one problem wrong but your work was ok so you get partial credit, and your other answers imply you understand the material, and I didn’t put anything on the test that I know we haven’t covered, so your total score is a B.” That’s not life.

In real life you can have done everything well but if you fail at this one thing then it’s all a fail. You also have to handle things that you couldn’t possibly know and that no normal person would be good at. Customers are like, “Your onboarding is annoying so I judge all the rest of your product as terrible, my browser is wonky with your site right now so I’m not going to assess this other stuff so it’s an auto-fail, why aren’t you amazing at graphic design in addition to everything else, you didn’t respond to my email in the middle of the night and now I’m never going to read it, so your score in mobile technology is an F.”

There are these clutch moments all around us where everything you’ve been working on for years comes down to one thing happening correctly and you simply have to nail it. If the customer can’t install your product, it doesn’t matter how amazing it is because they’ll never get to use it- you might as well have spent that whole time watching TV instead of building an awesome framework or whatever.

People have started messaging me about the YC interviews. I was debating whether YC tries to bridge that gap between real life. YC interviews are not real life; they’re like school because you know when and how you’ll get tested on certain areas. It’s not like you’ll suddenly have to demonstrate you know how to field a PR disaster if your product isn’t even launched. You’re at a certain stage and they only ask you things relevant at that stage and they only test that knowledge in a particular way. In real life, no one cares what stage you’re in- you better be ready when the test hits.

When we had our YC interview, everything we’d been working on came together for 10 minutes where we convinced some of the smartest people alive that we somewhat knew what we were doing. It was one of the first times we’ve ever had to pitch to skeptics, which sounds hard but is simple if you practice because you already know what they’re going to ask. No one has ever asked me a question I haven’t thought of already because all I do is think about the mobile space and the probability of someone coming up with a new question in a few minutes is low. Thus just work on an answer that people will 1) understand and 2) believe. In this type of situation, anything is possible if you prepare.

If you have an interview, please message me because I’m happy to help practice. I emailed many people when we were prepping for our YC interview and every single one of them met with us. It’s possible I’m a genius wrt cold emails, but for me the lesson is to pay it forward. Please give me a chance to do that!

Jeremy and I incorporated Apptimize in February 2013 and applied to Y Combinator 2 months later. Below is our Y Combinator application because a lot of people ask to see it. If you’re in the Bay area and would like feedback on your application, shoot me a note and I’ll try to help because my Paul Graham simulator is pretty good (he’s always making fun of my Peter Thiel simulator though). You can read about my Y Combinator experience here. My next post will be about the many ways reality has revealed itself as different from what we saw when we wrote this application, because the competitive landscape has changed, we are now a 15 person company, and our users include the top apps in the world. I hope you find this useful and upvote/share this because I’m even including our embarrassing video. Good luck!

http://apptimize.com

http://youtu.be/M5toaOjqsyc

What is your company going to make?

Apptimize lets you AB test mobile applications. You keep the native experience without needing to push changes blindly or rely on users to update. There’s a web interface to manage experiments, and a WYSIWYG interface for non-programmers. Apptimize removes the pain of designing a controlled experiment, serving variations, collecting results, and calculating statistical significance. Right now you have to be a developer and statistician to AB test a mobile app, but we make it so that non-programmers can AB test too. Apptimize makes optimization as easy for mobile as it is for web. Apptimize technology could transform the process of testing and pushing changes and be integrated into 100% of apps.

nancyhua; Nancy Hua; 27; 2007, MIT, Bachelors of Science Mathematics for Computer Science, Bachelors of Science Writing; nancyhua.com, @huanancy; GETCO algorithmic trader, Quantitative Strategies Team Leader

jorlow; Jeremy Orlow; 28; 2007, Purdue, Bachelors of Science Computer Science; @jeremyorlow; Software Engineer at Google, Software Engineer at Three Laws of Mobility (startup acquired by Motorola that was acquired by Google), DrawChat

Please tell us in one or two sentences about the most impressive thing other than this startup that each founder has built or achieved.

Nancy: trader who ran the Fixed Income Quantitative Strategies team at GETCO (GETCO grew from 100 to 500 people to become the premiere algorithmic trading company); world class expert in Fixed Income trading and exchanges.

Jeremy: owned IndexedDB (the emerging w3c standard for storing data in a browser) within Chrome; edited the spec, worked closely with Mozilla and Microsoft on the design, and wrote most of the initial implementation in Chrome/WebKit; simultaneously started the London Chrome team.

Please tell us about the time you, nancyhua, most successfully hacked some (non-computer) system to your advantage.

Nancy wanted to work in the Middle East but there wasn’t a culture of internships. Nancy discovered if she didn’t mention she was just a sophomore she could interview as a consultant (and get a company car and phone). She was the first student ever hired for Mercury’s R&D office in Israel (a load testing company acquired by HP).

At Google, Jeremy became an expert in free travel. After getting on shortlists for university recruiting, he positioned himself as a datacenter expert and visited many across America. After targeting developer relations, Jeremy got on the shortlist for places like Moscow, Berlin, Manila, Singapore, Sydney, and Tokyo, giving talks, meeting partners, and exploring- all for free.

Please tell us about an interesting project, preferably outside of class or work, that two or more of you created together. Include urls if possible.

We prototyped an app called Firesale that helps people sell unwanted stuff. To create a market of buyers, we brought on full-time Craigslist market makers. The Craigslist expert users complained about the process of being first to email a poster, so we optimized the messaging to make transacting as fast for them as possible. They also complained about Craigslist lacking a reputation/identity system, so we implemented one. We put Firesale on hold to work on Apptimize.

How long have the founders known one another and how did you meet? Have any of the founders not met in person?

We met a couple years ago through mutual friends and started working together when Jeremy convinced Nancy to leave NYC for the Bay.

Why did you pick this idea to work on? Do you have domain expertise in this area? How do you know people need what you’re making?

We picked this idea because Jeremy had looked for a mobile AB testing solution when working on Drawchat, but couldn’t find one. Three 50+ people companies, 3 YC companies, and 10+ indie developers have signed up to beta test our product. All the programmers/contractors we’ve interviewed have also asked to sign up for our private beta. This is an immediate need for most mobile companies.

Nancy is an expert in experiment design and data analysis. Jeremy is an expert in mobile and has built many efficient, scalable backends. We both love being data driven and view life as an experiment.

What’s new about what you’re making? What substitutes do people resort to because it doesn’t exist yet (or they don’t know about it)?

Most wait for app store approval and push many changes simultaneously. They eyeball the results and haphazardly rollback suspect changes.

Desperate people resort to basic, home-grown solutions. Because of other projects, Switchboard and Clutch.io evolved incomplete solutions (we noticed errors: randomization mistakes that mess up the experiments, poor error handling, malformed responses that’d crash your app!).

There hasn’t been much focused effort towards creating a seamless AB testing experience for native apps. AB testing for mobile is a technologically harder problem than for websites due to challenges particular to mobile devices (ie. intermittent internet, lack of cookies/iframes, users running different versions). Existing solutions ignore complexity whereas we view handling it as our core business.

Who are your competitors, and who might become competitors? Who do you fear most?

Several companies very recently entered the game. Swrve has so far focused on games. Pathmapp is focusing on overall analytics (pretty different from our approach). Abstate is unlaunched. Artisan and Arise.io have buggy, immature products. A risk is that Visual Website Optimizer or Optimizely will decide to focus on expanding from websites into native apps. Native might be a natural next step for them since they offer web app support in premium plans, so we’ll grow aggressively.

We think there’s no dominant player because nobody has made anything good yet. Our goal is to be the best.

What do you understand about your business that other companies in it just don’t get?

Our competitors are developers building for other developers, so most only offer programmatic interfaces. We understand often the goal setters and decision makers aren’t programmers. Apptimize makes it simple for non-technical owners, product managers, designers, and marketers via a WYSIWYG interface and a website to control and create experiments.

Our experimental setup, results, and analysis will be superior. Stanford PhD’s helped with our statistics by pointing out problems with competitors’ setups (ie. fixed sample sizes, small data set handling).

We’ll target companies who don’t monetize through app sales, instead using apps for branding, coupons, other off-app conversions. Although our first users are indie developers, most profitable apps make <$2K per month, so we’ll grow to targeting corporations like United, Starbucks.

How do or will you make money? How much could you make? (We realize you can’t know precisely, but give your best estimate.)

The plan is a monthly subscription. We’ll offer customers help with experiment design. If we charge premium customers $1K per month and get 200 customers (less than 2 sales a week) over 2 years we’d make ~$2.4MM per year 2 years in. Artisan (launched this month) claims to charge $1K-$10K per month, so that’s possibly a better price.

Ultimately we want to be the default way people change their apps. Everyone would use Apptimize to test each idea, and then use Apptimize to deliver the change to users. 100% of apps would use our library to reduce time to propagate changes and tighten the app development cycle. We’d help erase the line between apps and the web.

If you’ve already started working on it, how long have you been working and how many lines of code (if applicable) have you written?

We started in January, and Apptimize is currently ~8K lines of code (not including libraries, html, or css) and works end-to-end. The frontend is JS, CSS, and Angular. We’re on EC2 mainly using PostgreSQL, nginx, and Netty/Java.

How far along are you? Do you have a beta yet? If not, when will you? Are you launched? If so, how many users do you have? Do you have revenue? If so, how much? If you’re launched, what is your monthly growth rate (in users or revenue or both)?

Apptimize works and we just launched our private beta this week! We have 100+ signups but we only accepted 2 friends this week because we are working closely with our first customers to shape the future of our product.

The beta has the Android library, a website dashboard to manage experiments, and a results page showing statistics and conclusions. The WYSIWYG interface will be ready in a few weeks. Our research suggested starting with Android because Android developers rely on freemium (compared to iOS who make a lot off premium) and want to AB test to optimize in-app purchases, etc. Our iOS version is coming in a few weeks.

If you have an online demo, what’s the url?

yc.apptimize.com/admin

How will you get users? If your idea is the type that faces a chicken-and-egg problem in the sense that it won’t be attractive to users till it has a lot of users (e.g. a marketplace, a dating site, an ad network), how will you overcome that?

Our first customers are our friends’ startups. To target our next customers, we downloaded their apps and their competitors’ apps and are designing experiments for them. If they find the pre-designed experiments useful, they can easily start testing with those the instant they sign up.

We’ll offer customer referral rewards such as temporary premium memberships. We also want to make it easy to see and implement case study results by suggesting experiments to potential users. For marketing, we will ask and answer stackoverflow and Quora questions regarding how people AB test on mobile.

We could partner with companies in related fields like App Annie or Parse.

If we fund you, which of the founders will commit to working exclusively (no school, no other jobs) on this project for the next year?

Nancy and Jeremy are committed to exclusively working on Apptimize for the next few years.

If you had any other ideas you considered applying with, please list them. One may be something we’ve been waiting for. Often when we fund people it’s to do something they list here and not in the main application.

EEG machine to read babies’ minds. We like playing with our Emotiv machine, know prominent MIT/Stanford researchers, and see parallels between EEG analysis and high frequency market data for financial instruments (both systems produce massive amounts of data that seem random but aren’t).

A page-less browser using crowdsourcing. It’d show logical dependencies, assumptions, relationships between ideas, and best arguments for and against each belief.

Please tell us something surprising or amusing that one of you has discovered. (The answer need not be related to your project.)

People think it’s red, but no one knows the best button color.