As the self-appointed IT director of my new family, I was tasked with finding the best solution to easily upload, back up, and share pictures and videos of our newborn daughter. While there are a myriad of cloud services that a normal person would go for, I didn’t want to rely on them to safeguard our most precious moments. To be honest, I was also itching to create a serverless app on AWS, without committing to much cost or maintenance overhead.

AWS has a host of relatively cheap services that makes creating small serverless apps easy. The JavaScript API allows you to keep most of the logic in your browser, where you can access AWS directly or through a proxy (e.g. API Gateway). I’ll explain how I used them:

• Cognito for authentication: The main users of this app were my family and friends, so I only had to worry about unintended mistakes as opposed to malicious abuse. This allowed me to create pre-defined users for different roles (e.g. admin and visitor), and use Cognito to authenticate them. The JavaScript API lets you safely hard-code the public codes in the web page, and log in using a password only.

• IAM for authorization: Once users are authenticated, IAM should give them minimal privileges to do their tasks. For example, I gave file upload access to admin users, but only read access to visitors. The Principle of Least Privilege prevents users from wreaking havoc. IAM didn’t have the finest-grained access levels, but for a trusted set of users, that should be good enough. For more flexibility in authorization, of course, it has to be done on a proxy or web server.

• S3 for storage: Amazon’s simple storage is truly simple to use. I used it to store media files, thumbnails, and the static website assets. You may make the static site public, but put media files behind Cognito. The nice thing about S3+Cognito is that you can use the Cognito token in the S3 URL and access it in your website as you normally would with hosted images.

• DynamoDB for database: The list of galleries and files, timestamps, captions, user access, and comments have to be stored somewhere. While S3 provides limited ability to store metadata for each file, the permanently-free tier of DynamoDB has enough capacity to store them in a few tables. The NoSQL (or rather schemaless) nature of the database makes it easy to quickly add new features.

• Lambda for processing: A serverless architecture will not be complete without a function-as-a-service component! In this case, I used a S3-triggered function to create an entry in the database and process newly-uploaded images and videos. It could do anything as simple as generating thumbnails (Lambda is pre-packed with ImageMagick), or dealing with EXIF idiosyncrasies.

As for front-end, there wasn’t much needed for this app. Besides the good ol’ AWS JavaScript API, I used Bootstrap for easy styling, Knockout for two-way data binding, and Unite Gallery for media viewer. Unite Gallery is an open-source library with a few available themes and easy setup. However, getting videos to play on mobile and JPEG EXIF orientation proved to be challenging.

If I found time to improve the app, these areas would come up next:

• CloudFormation: As of now, most of the configuration has been manually done and its persistence is at the mercy of AWS. I can use CloudFormation to formulate how everything was set up, so it can be retrieved if anything goes wrong. Amazon provides CloudFormer to create a template from an existing architecture, but it didn’t cover the bulk of my configuration around Cognito and security policies.

• Automatic rotation: not all browsers can show the right orientation of cellphone images based on their metadata. I can use a Lambda function to automatically rotate images based on their EXIF orientation field.

• API Gateway: Using the combination of API Gateway and Lambda, there would be no need to give users direct access to AWS resources. This will improve the security and may make the app useful for other people who can have a more serious use case for it.

• Backup: A scheduled task that backs up media files and database records and writes them onto cheaper storage (e.g. AWS Glacier). For a more paranoid approach, I can also store them on another cloud service provider such as Google Cloud.

I’d be happy to get feedback on the design, and ways to improve the architecture.

Chilkoot Trail, a 53-km trail from Dyea, Alaska to Bennett, BC, was the main way for gold rush prospectors in late 1890s to get to Klondike River in Yukon. This slideshow contains photos of my 5-day hike through this historic trail.

Slideshow plugin by Pixedelic.

If you liked this, you may also like my Iceland Trip post.

A friend of mine was trying to get hired as a software developer, so he asked me about resources to hone his skills and practice for programming interviews. I came across a few websites where you could solve programming puzzles online. Your code would be sent to a server to run in a sandboxed container, and you would shortly get the result. My friend was specifically learning JavaScript. That made me wonder how cheaply and effortlessly I can create and maintain a similar site for JavaScript only.

The most expensive parts of hosting are usually tied to the computing power of servers. No server can beat the increasingly cheap option of not having a server at all. If your website can work with static resources, you can store everything you need on a storage service, configure your domain, spend a few cents a month, and never have to break a sweat about how your website is running. But how much can you accomplish with a static website?

In the case of an online JavaScript practice service, you can get away with not having a server for many things:

• Core: The core of your service, running users’ code, can be delegated to their browsers. For this, web workers are a great feature that cleanly isolate potentially harmful code, and are supported in modern browsers.
• Assets: All assets on landing pages, blog posts, and other informational pages are static.
• Security: Users can see how you are running their code, see your test data for programming problems, and reverse-engineer them. In this case, let’s agree, it is actually serving the purpose of teaching JavaScript.
• Personalization: Local Storage can be used to store the history of their problem-solving. This doesn’t survive a browser data deletion or moving to another device, but oh well!
• Community and Engagement: I haven’t added any feature of such nature yet, but there are free or cheap services like Disqus or SumoMe that can add comments or email collection widgets to your static page. I’m not aware of any service for a leaderboard, but I’m sure if the site becomes popular enough, I can roll my own AWS Lambda script to take care of that.

In order to create the site, I’m using a Node.js script. Jade templating engine, Markdown language, Uglify-js, and Express.js for a local test server have come in very handy. I’ve written an automated build script, where I only need to add a single JSON file for every problem, and it creates the whole page, adds it to the index and sitemap, and deploys new or updated files to Google Cloud Storage. Google Cloud Storage makes it easy to host a static website, but it doesn’t support HTTPS yet. I’m using Cloudflare’s free plan to add HTTPS, server-side analytics, a caching layer, denial-of-service protection, and even a widget to show warnings if a browser is not supported.

I might open-source this in the future, but for now, feel free to practice JavaScript online for fun or technical interviews!

The price of real estate in Vancouver has been rising for years, and if you’re not the gambler type, renting has never been a more financially sensible option. However, according to a report by Canada Mortgage and Housing Corporation in October 2015 (PDF), the vacancy rate in City of Vancouver is at 0.6% with some neighbourhoods at as low as 0.3% (English Bay).

If you are a renter like myself, and you’ve decided to stay put for a while, it doesn’t affect you in the short term (British Columbia has a rent increase cap at less than 3% per year). But tough luck if you want to move for any reason! I’ve heared anecdotes of reasonable Craigslist listings being snatched up in the matter of hours, and what’s left is flawed units or extortionately expensive ones.

Another rumour is about people who buy properties in popular locations, and instead of putting in on the rental market, they effectively run a hotel business on Airbnb without all the pesky licenses and regulations. As much as I love using Airbnb for private rooms in cities I visit abroad, the NIMBY in me is not very happy about this trend. That’s why I decided to crunch the numbers and have a better idea of just how bad the Airbnb effect is here.

Luckily, there’s a website that continuously scrapes Airbnb listings and publishes the data. I grabbed the Vancouver data from December 2015. It contains all the listings that were available in 2015. For the purpose of this analysis, I only looked at private homes (no shared or rooms), and only those with at least 90 days of availability to filter out occasional travellers that rent out their principal residence.

The Inside Airbnb data comes with the neighbourhood of each unit, but it doesn’t exactly match the official neighbourhoods posted on the City website (e.g. Gastown is in Strathcona in one and in Downtown Eastside in the other). While the city has over 20 neighbourhoods officially, the CMHC rental report is broken down into 10 zones only. For the sake of visualization, I had to roughly map neighbourhoods to zones, but it shouldn’t affect the results much.

The following map shows Vancouver neighbourhoods with information about their rental situation (click on each to see). I calculated the estimated number of vacant units by multiplying the total number of rental units in each zone by the reported vacancy rate. The map is colour-coded by how high the number of Airbnb listings are relative to the number of vacant units (darker red for a higher ratio of Airbnb listings over vacant rentals).

It seems that central areas and those closer to transit have it worse. Downtown has more than 10.2 times Airbnb listings than it has vacant units for long-term rental, but it’s as much as 15.8 times in Mount Pleasant and Renfrew Heights. It’s not easy to draw a conclusion about how releasing all those Airbnb listings into the rental market would do. Maybe more people would move to Vancouver and the rents wouldn’t budge much. Maybe more people would move in and rents would go lower too, but at the cost of lowering the overall income tax. I leave that analysis to people with more expertise, but it’s obvious that better regulations for those underground hotel businesses would make life slightly easier for locals who can’t or don’t want to buy into the real estate insanity.

After over one year of working on my side project and trying to monetize it, I’ve decided to call it quits and share my journey and the source code with the community (Github). In this post, I’m going to explain the whole development process, describe my failed attempts, and get into more technical details in case someone wants to use the app as a boilerplate to create their own.

The Idea

I was itching to create my own software-as-a-service web application while working full time. After all, the idea of having a few sources of passive income is frequently sold to software developers, and I wanted to take part in the “micropreneurship” revolution. I also wanted to practice time management and prove to myself that I can build something from scratch, and possibly have an extra source of income. There’s no need to mention the rewarding sense of building things on top of all that.

The idea of an automated virtual phone system is neither original nor sexy. In fact, there are many companies out there that are doing the exact same thing (e.g. Grasshopper, eVoice, MightyCall, etc.). This meant I might be able to take over a very small part of an established market. I knew Twilio provided an easy-to-use API, so I just had to do the math and see how high the barrier to entry is.

Research and Design

I researched the feature set of top players. I gathered a list of features that I thought I could implement in a few months and prioritized them. I then used KickoffLabs to create a fake landing page advertising all those features. Next thing I needed to do was to buy some paid traffic and measure what percentage of the visitors are willing to press the “Get Started” button. Even though pressing that button doesn’t mean the intention to pay, it indicates the pitch is compelling enough for them to spare a few seconds of their time.

When I got a good conversion rate on the fake landing page (and collected a few email addresses of prospects), and compared the cost of acquiring new customers to their potential lifetime value, I started the design phase. At this stage, I carefully listed all possible features from competitors, and added some of what I thought a potential customer would need based on pure speculation. Then, I sorted them by how important they are and how much time each one of them would take me. The outcome of this stage was a high-level list of features that my minimum viable product (MVP) needed.

The next stage was technical design. I decided to use Express.js framework on Node.js, and use Twilio and Stripe as external service providers for telecom and payment processing respectively. The free tier of Amazon Web Services provided enough computing power to get a staging server up and running. I tried not to use anything too close to the cutting edge, and compromised with more established libraries like jQuery, but I leave the details for the “Under The Hood” section of this post. The technical design was based on an MVC architecture, and I only needed pen and paper to document the database design and screen mockups.

The Interesting Part: Coding!

As many other fellow software engineers can agree, coding is the most interesting part of the process. It’s certainly not the most important part though. Having been coding for 15 years, I found this part well inside my comfort zone. The real challenge was time management and staying focused. Working full time and the human need to be offline leave too little time for extra screen time. However, what really helped me was the great power of habits. I spent an hour or two every single day. No exceptions. If I was very tired or feeling down, I would work on the parts that didn’t need much attention, but the trick was to keep up the streak.

I used Trello to break down the tasks and keep track of my progress. As expected, I had to deal with numerous unexpected issues. Any line of code that you create or comes with external libraries, and any API call to service providers is a potential headache. I took pleasure in solving them one at a time, and I think the source code at its current state contains a wealth of solved problems that I’ll certainly use in my future projects.

I asked a few friends to beta-test the app for me. I applied their feedback and fixed a few bugs. The development took 3-4 months, and I was ready to launch the product after all. I used a free Bootstrap theme to design the landing page. The only money spent so far was about $40 to get the logo designed and register the domain.

Going to The Market

To an engineering-minded person without any serious business experience, this part is the most daunting. To a business-savvy person, this is probably the most important part. Even though engineers like to make fun of all those “I have a great idea and will give 5% equity to an engineer to implement it” Craigslist ads, a person that can execute well on the idea is rarer than a 10X unicorn engineer.

So I got started by creating a couple of social media accounts, and buying some paid trafic from Google search network. The way my app was designed required users to enter their credit card number before buying a virtual phone number. I was too afraid of abuse, and didn’t want to give away too much freebies to acquire new customers. The first campaign showed that I need to be more open. A good number of people signed up, but all of them stopped at the payment screen.

I added a couple of free trial options. I also commissioned a friend to create a low-budget explainer video. I also changed the on-boarding workflow so users can see the control panel before entering their credit card number. These changes allowed more people to get into later stages of the sales funnel. However, none of the users that bought a phone number started paying for it after the trial period.

I guess I didn’t have enough motivation to try inbound marketing, or cold-calling potential customers. They were certainly outside my comfort zone. Attempts at selling the website on Flippa or finding someone to help me with sales and marketing didn’t go anywhere either. The amount of effort that it takes to get any traction for this web app seemed too high, and it eventually led me to open-source it and move on.

Under The Hood: Source Code

I used Express.js on Node.js for back-end, and Bootstrap and Knockout.js for front-end. Twilio was the obvious choice for telecom API, and I hooked up my app to Stripe for payment processing. I deferred writing tests for when I had paying customers, but the software was designed robust enough to evolve into a solid application once it was proven in the market. I used exact versioning in my NPM manifest file, and included all the packages in the Git repository to remove any incompatibility or NPM infrastructure risks.

The architecture on the back-end is MVC, and the front-end uses AJAX calls to populate the page. The server merely sends a Knockout-enabled page to the client, which in turn calls an API command or two to get the required data in JSON format. By this design, I avoided the headache that managing a single-page application will give you, and provided a blank page for users as soon as possible. I think users like to see “somehintg” (the loading indicator) very soon, but show more patience for loading data after that.

There are many opportunities to refactor the code (especially in front-end views), and there are certainly bugs to be found. However, here’s a list of different libraries and components that I used or wrote:

• Express.js: Express is a great web framework, and the rich ecosystem of middleware allows you to shop around for best solutions to common needs.
• Passport: A useful library for user authentication. I had to write my own session store that uses MySql (session_store.js), and use bcrypt to encrypt passwords, but Passport pretty much provided the rest.
• Sequelize: It’s a good ORM that you can use for data modelling. Express doesn’t come with an ORM, so I had to pick between this or Bookshelf. I finally went with Sequelize, and it worked just fine for me. Getting entity relationships right was a bit tricky though.
• Q: This promise/async library helped a lot with avoiding the infamous Node.js callback hell.
• BigNumber.js: When it comes to money calculations, you shouldn’t trust JavaScript at all. To avoid using their float numbers, I used BigNumber. It’s a bit slow to write simple math, but the reliability makes it worthwhile.
• Connect-assets: This is an older asset packaging library. It minifies and aggregates Javascript files, and compiles stylesheet files. That’s good enough for smaller web apps.
• Moment: The absolute best when it comes to manipulating dates. Its timezone library made working with timezones a breeze.
• Jade: I used this for my views. It removes the HTML clutter.
• NodeMailer: I wrote utility files to compile Jade files, and send emails via Amazon’s mail service.
• Twilio: If you’re using Twilio, their Node.js library is a no-brainer.
• Stripe: Not having to deal with storing credit cards is great. They provide an easy API, and all you need to store and work with is tokens instead of sensitive user information.
• secret_sauce.js: This is a “secret” API call that I thought might be useful for invoking certain scheduled tasks, for instance cleaning up expired sessions, or charging customers. I never got to the stage that required automating them though.
• Bootstrap: Some people may be allergic to Bootstrap at this time, but it gives you a generic well-designed-looking page for cheap.
• Knockout.js: This is great for front-end data binding. If all you have to do is deal with a few forms on your page as opposed to a full-on single-page app, Knockout.js is your friend!

Conclusion

Creating Phonjour was a great learning experience for me. I don’t regret putting effort into it, and I hope the journey and the code scraps can be useful to others as well. In my future attempts at a side project, I’ll try to find a business-savvy person to take care of the business side.