Blog

I'm Open-Sourcing My Node.js App

February 06, 2016

After over one year of working on my side project and trying to monetize it, I’ve decided to call it quits and share my journey and the source code with the community (Github). In this post, I’m going to explain the whole development process, describe my failed attempts, and get into more technical details in case someone wants to use the app as a boilerplate to create their own.

The Idea

I was itching to create my own software-as-a-service web application while working full time. After all, the idea of having a few sources of passive income is frequently sold to software developers, and I wanted to take part in the “micropreneurship” revolution. I also wanted to practice time management and prove to myself that I can build something from scratch, and possibly have an extra source of income. There’s no need to mention the rewarding sense of building things on top of all that.

The idea of an automated virtual phone system is neither original nor sexy. In fact, there are many companies out there that are doing the exact same thing (e.g. Grasshopper, eVoice, MightyCall, etc.). This meant I might be able to take over a very small part of an established market. I knew Twilio provided an easy-to-use API, so I just had to do the math and see how high the barrier to entry is.

Research and Design

I researched the feature set of top players. I gathered a list of features that I thought I could implement in a few months and prioritized them. I then used KickoffLabs to create a fake landing page advertising all those features. Next thing I needed to do was to buy some paid traffic and measure what percentage of the visitors are willing to press the “Get Started” button. Even though pressing that button doesn’t mean the intention to pay, it indicates the pitch is compelling enough for them to spare a few seconds of their time.

When I got a good conversion rate on the fake landing page (and collected a few email addresses of prospects), and compared the cost of acquiring new customers to their potential lifetime value, I started the design phase. At this stage, I carefully listed all possible features from competitors, and added some of what I thought a potential customer would need based on pure speculation. Then, I sorted them by how important they are and how much time each one of them would take me. The outcome of this stage was a high-level list of features that my minimum viable product (MVP) needed.

The next stage was technical design. I decided to use Express.js framework on Node.js, and use Twilio and Stripe as external service providers for telecom and payment processing respectively. The free tier of Amazon Web Services provided enough computing power to get a staging server up and running. I tried not to use anything too close to the cutting edge, and compromised with more established libraries like jQuery, but I leave the details for the “Under The Hood” section of this post. The technical design was based on an MVC architecture, and I only needed pen and paper to document the database design and screen mockups.

The Interesting Part: Coding!

As many other fellow software engineers can agree, coding is the most interesting part of the process. It’s certainly not the most important part though. Having been coding for 15 years, I found this part well inside my comfort zone. The real challenge was time management and staying focused. Working full time and the human need to be offline leave too little time for extra screen time. However, what really helped me was the great power of habits. I spent an hour or two every single day. No exceptions. If I was very tired or feeling down, I would work on the parts that didn’t need much attention, but the trick was to keep up the streak.

I used Trello to break down the tasks and keep track of my progress. As expected, I had to deal with numerous unexpected issues. Any line of code that you create or comes with external libraries, and any API call to service providers is a potential headache. I took pleasure in solving them one at a time, and I think the source code at its current state contains a wealth of solved problems that I’ll certainly use in my future projects.

I asked a few friends to beta-test the app for me. I applied their feedback and fixed a few bugs. The development took 3-4 months, and I was ready to launch the product after all. I used a free Bootstrap theme to design the landing page. The only money spent so far was about $40 to get the logo designed and register the domain.

Going to The Market

To an engineering-minded person without any serious business experience, this part is the most daunting. To a business-savvy person, this is probably the most important part. Even though engineers like to make fun of all those “I have a great idea and will give 5% equity to an engineer to implement it” Craigslist ads, a person that can execute well on the idea is rarer than a 10X unicorn engineer.

So I got started by creating a couple of social media accounts, and buying some paid trafic from Google search network. The way my app was designed required users to enter their credit card number before buying a virtual phone number. I was too afraid of abuse, and didn’t want to give away too much freebies to acquire new customers. The first campaign showed that I need to be more open. A good number of people signed up, but all of them stopped at the payment screen.

I added a couple of free trial options. I also commissioned a friend to create a low-budget explainer video. I also changed the on-boarding workflow so users can see the control panel before entering their credit card number. These changes allowed more people to get into later stages of the sales funnel. However, none of the users that bought a phone number started paying for it after the trial period.

I guess I didn’t have enough motivation to try inbound marketing, or cold-calling potential customers. They were certainly outside my comfort zone. Attempts at selling the website on Flippa or finding someone to help me with sales and marketing didn’t go anywhere either. The amount of effort that it takes to get any traction for this web app seemed too high, and it eventually led me to open-source it and move on.

Under The Hood: Source Code

I used Express.js on Node.js for back-end, and Bootstrap and Knockout.js for front-end. Twilio was the obvious choice for telecom API, and I hooked up my app to Stripe for payment processing. I deferred writing tests for when I had paying customers, but the software was designed robust enough to evolve into a solid application once it was proven in the market. I used exact versioning in my NPM manifest file, and included all the packages in the Git repository to remove any incompatibility or NPM infrastructure risks.

The architecture on the back-end is MVC, and the front-end uses AJAX calls to populate the page. The server merely sends a Knockout-enabled page to the client, which in turn calls an API command or two to get the required data in JSON format. By this design, I avoided the headache that managing a single-page application will give you, and provided a blank page for users as soon as possible. I think users like to see “somehintg” (the loading indicator) very soon, but show more patience for loading data after that.

There are many opportunities to refactor the code (especially in front-end views), and there are certainly bugs to be found. However, here’s a list of different libraries and components that I used or wrote:

  • Express.js: Express is a great web framework, and the rich ecosystem of middleware allows you to shop around for best solutions to common needs.
  • Passport: A useful library for user authentication. I had to write my own session store that uses MySql (session_store.js), and use bcrypt to encrypt passwords, but Passport pretty much provided the rest.
  • Sequelize: It’s a good ORM that you can use for data modelling. Express doesn’t come with an ORM, so I had to pick between this or Bookshelf. I finally went with Sequelize, and it worked just fine for me. Getting entity relationships right was a bit tricky though.
  • Q: This promise/async library helped a lot with avoiding the infamous Node.js callback hell.
  • BigNumber.js: When it comes to money calculations, you shouldn’t trust JavaScript at all. To avoid using their float numbers, I used BigNumber. It’s a bit slow to write simple math, but the reliability makes it worthwhile.
  • Connect-assets: This is an older asset packaging library. It minifies and aggregates Javascript files, and compiles stylesheet files. That’s good enough for smaller web apps.
  • Moment: The absolute best when it comes to manipulating dates. Its timezone library made working with timezones a breeze.
  • Jade: I used this for my views. It removes the HTML clutter.
  • NodeMailer: I wrote utility files to compile Jade files, and send emails via Amazon’s mail service.
  • Twilio: If you’re using Twilio, their Node.js library is a no-brainer.
  • Stripe: Not having to deal with storing credit cards is great. They provide an easy API, and all you need to store and work with is tokens instead of sensitive user information.
  • secret_sauce.js: This is a “secret” API call that I thought might be useful for invoking certain scheduled tasks, for instance cleaning up expired sessions, or charging customers. I never got to the stage that required automating them though.
  • Bootstrap: Some people may be allergic to Bootstrap at this time, but it gives you a generic well-designed-looking page for cheap.
  • Knockout.js: This is great for front-end data binding. If all you have to do is deal with a few forms on your page as opposed to a full-on single-page app, Knockout.js is your friend!

Conclusion

Creating Phonjour was a great learning experience for me. I don’t regret putting effort into it, and I hope the journey and the code scraps can be useful to others as well. In my future attempts at a side project, I’ll try to find a business-savvy person to take care of the business side.


Keeping It Simple and Clean (Part 2)

June 28, 2015

For part 1 click here!

Do NOT Repeat Yourself

Another way to ensure things will remain simple is the DRY Principle (“Don’t Repeat Yourself”), which states “every piece of knowledge must have a single, unambiguous, authoritative representation within a system”. Violating the DRY principle can hurt on many different levels; if a state is stored in multiple places, a piece of code is copied and pasted, or a logic is handled by multiple components. As a result the information stored by the program is likely to become incorrect, your code is prone to contagious bugs, or miscommunication between team members are made easier.

Every time you find yourself copy/pasting a piece of code only to change it slightly, think again about how a helper function can be extracted. If a small piece of algorithm or logic has to be repeated in more than one place, that’s another candidate for refactoring. Except for special cases in distributed computing, no information should come from more than one source either. Even if the logic around it is obvious to you at this time, modifying the logic later will prove tricky to other team members or your future self.

Fundamentals: Data Structures and Algorithms

If you’re a hobbyist or only getting your feet wet, you may be excused, but there’s absolutely no excuse for any career programmer at any level to not learn computer science basics. I skip telling horror stories about huge mistakes made by people who hated math at school and assumed “theoretical stuff” are only for academics. One of the most important areas is data structures and algorithms.

One significant benefit of learning the basics is being aware of time and memory complexity. How long a piece of code might take, or how much space would it need? What if it’s fast for my test data, but it becomes prohibitively expensive in production? Your textbooks provide simple tools to answer those questions, and even if you haven’t read them, you can at least get familiar with a few common data structures and algorithms and know how they scale as the data grows. However, there will never be a right answer to out-of-context design questions involving time and memory complexity, because design decisions will almost always involve some trade-off between time, memory, development time, maintenance costs, etc.

Another benefit of getting familiar with the basics is being able to follow the flow of information. Where the information is stored, how it is stored or retrieved, and where and why it is needed are usually answered by being aware of the underlying data structures and their algorithms. Besides the theoretical aspects, there are also best practices that come from experience and other software engineering practices. For instance, it’s established that DOM is mostly used to represent data, and should not be used to store data (HTML5 data attributes aside).

YAGNI

“You Aren’t Going To Need It”. Very well said! If a feature is not required by business, or there’s no certain near-future plans to add it, you wouldn’t implement it. The same logic can be applied to lower-level design decisions. More often than not, people would like to look way ahead and over-design their systems for a rainy day when the dreaded future feature will come and they just kill it with their (over-)preparedness. The idea behind the YAGNI principle is that even if future is foreseeable, you shouldn’t sacrifice the current code base because of that.

Like many other similar principles, it doesn’t give us a certain recipe, but it tells us that, because of human nature, we’re more likely to over-design than under-design.

Idioms and Design Patterns

Standing on the shoulder of giants should feel good. Walking on a beaten path will tell you other people have gone down the same path. The same goes for time-tested patterns in programming. Design patterns and programming idioms specific to a tech stack are the gift of past engineers to us to deal with the same issues. The only downside is that it may not be obvious which patterns will fit our current needs. A pattern that perfectly answers a specifc situation, may be too complicated for another. Therefore, design patterns should always be used carefully.

Intermediate engineers, after learning a few design patterns, tend to over-use them. Obtaining the hammer of design patterns shouldn’t deceive you into seeing every programming issue as a nail. Take caution and be fine with rolling your own, but certainly add them to your toolbox as they will often come in handy.

Keeping It Clean

Writing clean code is more than an obsession with aesthetics or mindless principles. There are numerous papers on how having a consistent coding style will make programmers more productive. Just like a well-written novel, a clean code doesn’t require excessive cognitive power to read and understand, and it saves more of your human RAM for important coding decisions.

A clean and consistent code base will save other people and your future self time in reading, debugging and modifying. I’m not going to expand on how to write clean code, but I felt it was important enough to deserve a mention. Here are a few bullet points on keeping the code clean and consistent:

  • Code should be self-documented. The modules, the fields and methods, and even the variable names should make it obvious what they’re storing or doing.

  • Documentation. Not everything can be conveyed by writing self-documented code. Sometimes, adding a concise comment next to an inevitably complicated piece of code can help others understand it faster. In my opinion, external documentation should be saved for very specific cases like space shuttle or robotic surgery software. It’s often unnecessary for the rest.

  • Coding Style. This one tends to be ignored as a nuisance to sloppy programmers. The value of familiar style is mostly psychological, and it’s shown to result in more productivity. If your team doesn’t have a styling guide, just take one of the common ones and pitch it.

  • Consistency. Humans are more efficient on auto-pilot, and the most important factor in achieving that is consistency. Consistency can be maintained on many levels from styling to naming to larger design patterns.

  • Simplicity? Sometimes making a code more readable may go against keeping it simple. Again, it’s an art to find the right balance. Adding an interim variable to put a name on a value in the middle of calculation is justifiable. Adding a whole class to store that value is not.


Keeping It Simple and Clean (Part 1)

June 28, 2015

When I started programming circa 2000 as a hobby, all I cared about was how cool my programs and games were. It was mostly about using whatever tools I had at my disposal to make them happen. Participating in programming contests later turned me into a sloppy programmer that would take clever shortcuts, and see every small problem as an opportunity to apply a complicated cocktail of data structures and algorithms that I’d learned at school. That was sometimes the only way to solve some of those problems, but nothing at school or those contests prepared me for the great lesson that I would learn in real-world programming.

“I didn’t have time to write a short letter, so I wrote a long one instead.” - Mark Twain

Writing simple letters is not easy. Writing simple programs is not easy either. I learned that taking pride in the ability to make complicated contraptions that accomplish a simple task is a sign of professional immaturity. I learned an engineer who can keep things as simple as possible is the real programming artist. No expression could articulate it any better than Antoine de Saint Exupéry’s if I’m allowed another quote from another great author: “It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away”.

This principle can almost perfectly translate to engineering. The KISS principle (“Keep It Simple, Stupid!”) is a great example of putting simplicity first. As a more software-engineering-centric example, Agile principle #10 states that “simplicity, the art of maximizing the amount of work not done, is essential”. One would ask what simplicity means in software, and how it could be achieved. Let’s see if we can find the answer.

Simplicity in software

There’s been extensive academic research, and many metrics were invented by academics and good-old general line managers to measure software productivity, complexity, and value. The general consensus is that software is too complex to be accurately quantified, but there are oversimplified metrics that can still tame some of that complexity. Lines of code is probably one of the first metrics that was used in the industry. It’s very easy to measure and it correlates well with complexity, but it’s not too difficult to make some condensed piece of software that no one can understand and cannot be legitimately considered simple.

Other metrics usually try to quantify software complexity in terms of branches, components, number of interfaces, or moving parts in general. As this is not an academic post, and I don’t have enough authority to analyze them, let’s just leave the nitty-gritty details to academics and admit that simplicity to the average developer Joe is more of an art than science. Maybe the hipster artisanal JavaScript programmer at your local Starbucks can get it right easier than the Ivy school PhD student after all.

How to achieve simplicity

You must have realized by now that there’s no silver bullet to achieve simplicity. There are however tools and patterns that make it easier for us to eradicate complexity in software, one wrong design decision at a time. I’ll enumerate some of the tricks, but it won’t be a comprehensive list by any means.

Understand the damned code first

Chances are you’re not creating everything from scratch. You’re working on something that a few other poor souls put together, and if you don’t want to be sticking more incongruous mud on the pile, you’ll need to understand a thing or two about the code base. This will first and foremost keep you from re-inventing the wheel; someone before you probably encountered a similar problem and spent time to fix it properly, so why not make use of that?

Another important measure, especially when fixing bugs, is to understand the root cause of the unwanted behaviour. If you don’t kill the bug where it lives, and just try to stick a band-aid on the symptoms, you’re picking an uphill battle with a bug that you don’t even fully know. In a well-designed code base, the root cause should ideally be in one place, so you’d need to address it in a single place instead of chasing it around and playing whack-a-mole with every symptom of the bug.

It always pays off to at least have a basic understanding of what every major component in code does. This helps you make proper use of them. Misusing or abusing other components will make the software more fragile, since other developers cannot always guess unorthodox application of their components and every change can potentially pull the rug from under your code.

Encapsulation

The integrity of systems is better protected if their boundaries are well defined. Logical components of software should also have their own boundaries. Encapsulation is facilitated and enforced in some object-oriented languages, but in general, some discipline will make it possible in every language. Classes, private fields and subroutines are examples of encapsulation tools that can be implemented in languages as wildly loose as JavaScript.

The best way to achieve encapsulation is to define modules, clearly define the boundaries, and separate the concerns of those modules. This is a very high-level recipe, and in practice, things can get sticky and impossible to separate. However, it should be obvious to software engineers, when deciding where to put anything, whether defining the boundaries will add any value or it’s turning into another anally-retentive obsession with how to store your possessions in one of the 100 drawers in the bedroom.

For part 2 click here!


How A Software Engineer Invests His Savings in Canada

May 01, 2015

Obligatory legalese: this post is about my own personal finance in Canada, not general financial advice. There’s always risk in investments, and they’re done at your own risk.

Backstory

As I graduated from grad school and my paycheques started exceeding my expenses, I realized I have a new grown-up issue to deal with; not too bad for an issue! I wasn’t very comfortable with buying invisible assets that constantly fluctuated in price. When I cut through common human irrationality and realized saving my surplus income in a savings account is a sure-fire way to decay my assets, I started looking into buying shares in funds.

As a rather young software developer without much responsibility, I should be more risk seeking. As much as gambling with individual stocks is tempting, I decided against it; there’s a high risk there and high upkeep for staying current with financial news. Composite funds, with their broad range of risk profile from aggressive leveraged funds to those tied to non-Greece government bonds, provide a low-maintenance, low-risk investment vehicle.

What

Mutual funds are typically very convenient to buy. They’re usually offered in employees’ benefits for RRSP matching or automated withdrawals. The downside is that they take away a big chunk of your money every year. It’s common for them to rake in 2-3% of your savings, even in years when they lose 20-30% of it investing in volatile securities. Exchange-traded funds (ETFs), on the other hand, provide a much cheaper option (usually less than 1% management fee), have higher liquidity, and have been proven to be as effective as actively-managed funds. You’re basically betting on the whole civilization to stay in place, and slowly move forward as it has so far.

Canadian Couch Potato had a good model portfolio back then. I started putting my money in 3 ETFs: 40% trusting governments and companies to pay off their bonds (XBB.TO), 40% trusting the world is moving forward (XWD.TO), and 20% in betting that conservatives in Canada will make their social backwardness worth it (ZCN.TO). This portfolio has a very low management fee, and is traded in Toronto Stock Exchange, so no money is lost in exchange rates.

Where

I opened 3 accounts with Questrade: TFSA, RRSP, and a margin account that holds the rest. They don’t have well-lubricated processes, have subpar customer support, and won’t just stop spamming you, but they have something unique; it’s free to buy ETFs with them. The spread of buy and sell prices is very low, and they don’t have any hidden fees, so it makes dealing with them worth it.

As a sidenote, if you’re going to start saving with them, you can use their referral program to earn $25-250 in cash when opening a new account. They also gives the referrer a $25-50 bonus, so if you don’t know anyone with a Questrade account, feel free to use my QPass key: 486304026379159

How

The first thing I accomplished was to max out my TFSA account. TFSA is a great Canadian program that allows you to earn investment income tax-free. At this time, assuming annual 10% gain from the ETFs, Canadians can save up to ~$1000 a year in taxes. The growing limit of TFSAs and the magic of compound interest will make them a bigger deal in the future. After maxing out the TFSA, I started contributing to my margin and retirement accounts. You can use up to $25K of your RRSP in buying your first home, but anything beyond that is about how much you value your future self.

It’s important to keep your portfolio as balanced as possible (40-40-20 in my case). Selling ETFs on Questrade costs money ($5-10 per trade), so I avoid re-balancing the portfolio unless the gap is too large and new contributions can’t close it. I also try to move funds to TFSA as soon as the limit grows (usually January 2nd), and to RRSP once I decide how much I’m saving this year in my retirement account. This helps keep the taxes low, and sometimes helps with rebalancing. A spreadsheet is all I need to determine how to do the re-balancing.

What Else?

The housing market in Vancouver is overvalued according to unbiased experts. I don’t want to lock all of my past and future savings in a single asset that can dip in value at anytime. Buying a house also eliminates some of your options, and makes your unexpected expenses much more volatile. Real estate is therefore out of question, and that doesn’t leave anything else that can meet my criteria of low-effort, low-risk investment.

Good luck with your investments!


Faith-Free Mindfulness Meditation

January 17, 2015

For a long time, I was dismissing suggestions to try out mindfulness meditation. It sounded like newly discovered eastern religion nonsense, advocated by the new age crowd or people desperately trying to fill the spiritual void in their lives. I didn’t care that they advertised relaxation as a side effect. Cutting through the mumbo jumbo to mine practical recipes for peace of mind didn’t seem worth it. Mindfulness remained an untouched realm of good-feeling insanity until frequent remarks on Hacker News made me discover Sam Harris’s new book.

Sam Harris has a reputation for fighting blind faith and irrationality. He’s also a renowned neuroscientist, which gives him all the authority to talk about faith-free mindfulness meditation. In his new book, Waking Up, he brings the attention of scientifically-minded folks to the baby of mindfulness in the bathwater of eastern religions. He’s extra careful to not make any irrational assumptions, and keeps apologizing for using conventionally-unscientific words like spirituality. Sam doesn’t make any claims about reducing our consciousness and sense of self to the physical world. He just invites us to experience meditation for ourselves.

Beside enjoying the read on human brain wonders such as split brain experiment, I was able to identify what Sam calls “the illusion of self” with my psychedelic-induced experiences back in the day. While on magic mushrooms for a few times, I witnessed the illusion of self; my ego, my sense of self, faded away and I was able to look at myself as a separate person. My psychedelic journey was -luckily- very positive, and I always suggest friends to carefully try it out at least once. I didn’t know psychedelics are a shortcut to the state of mindfulness. Wouldn’t it be great to get the same feeling of relaxation, general satisfaction in life and compassion toward others without taking the biological and legal risks?

The point of this post is not to describe what mindfulness is and how one should meditate to reach that state. There are a variety of resources on the subject, and if someone wants to start looking into them, it should be rather easy to get started. The marginal benefit that people get, even at the beginning of their quest, makes it easier to keep motivated and explore more. If you are as skeptical as I used to be, I hope this will convince you to give mindfulness the benefit of the doubt. I did and have been enjoying it so far.