Writing a Book as a Programmer

Comment

Writing a Book as a Programmer

Writing a Book as a Programmer

Note: This Post was originally published on Hackernoon and Practical Dev

I recently did something that was rather surprising to me. I ended up writing a book without even realizing.

In this post, I will discuss how my book/interactive course, Coding for Visual Learners, came to be, and what writing a book looks like as a programmer from a technical point of view. I will also talk a bit about creating online courses as it was the starting point of my writing journey. If you are short on time, you can skip to the third part of this article where I talk about what kind of tooling I ended up using for writing and publishing a book or check out the tl;dr below. But remember, sometimes the journey is more rewarding than the destination.

tl;dr:

  • Begin with the tools that you are most comfortable with. It is important to keep the friction low when you are just starting out.
  • Your emerging needs will inform your specific tooling decisions so just start creating instead of getting stuck at planning.
  • I tried Workflowy and Notion.so for writing. Settled on StackEdit.io for its Markdown support, cloud syncing and offline editing capabilities. It was a good enough choice but can fall short when gathering feedback on your material.
  • Leveraged YAML front matter to include metadata inside the text and Nunjucks templating language to be able to use conditionals and variables.
  • Created node.js scripts to move documents around, and to pre&post process them according to the desired output format (offline vs online).
  • My process was automated but not overly engineered. Remember, do things that don’t scale.
  • I worked with a proofreader on Fiverr
  • Used Leanpub to self-publish and to generate an unbranded ebook in multiple formats to be sold on the Amazon Kindle store.
  • This linked post by Zapier is a must-read if you are planning on self-publishing.
  • Used Educative.io to create an interactive and animated course out of my already existing material.

Part 1: How did I write a book without even realizing?

Let’s start with a proper introduction. My name is Engin and I am a software developer, an educator and now an author. I have been teaching an Introduction to Python course at a college for more than two years now, and a couple of months ago I started to create online courses/video tutorials on the side as well. I usually find myself scripting the entire course before doing the recording to be able to have a more predictable and a repeatable outcome. In other words, I am not great at improvisation. These scripts that I end up writing are usually very comprehensive and contain virtually every single word that I plan to say during recording. I published my first online course last year on Pluralsight, Automating the Web using PhantomJS and CasperJS. You should check it out if you haven’t done so yet. It’s okay, I will wait for you to come back.

This time I wanted to create another online course, but I wanted to do everything by myself to have a more end-to-end experience in course creation. I have been working for someone else for my entire life and never really handled any transaction all by myself. I figured creating a course could be a good exercise in experimenting with the business side of the things as it would need me to handle things like marketing or distribution as well. In my preliminary research, I identified a market gap in visual ways of learning to code and choose to use the excellent JavaScript library, p5.js, to teach programming to beginners. As a self-taught programmer, I remember clearly the struggles that new programmers face. I set out to create the kind of course that would have helped me when I was starting out.

To write the scripts I would be reading during recording, I have been using a service called Workflowy. It is essentially a list creator where you can create deeply nested collapsible lists. I found that this kind of an interface maps well to how I structure my thinking. It allows me to create a new list item for every couple of sentences. This makes reorganizing your text really easy just by dragging and dropping. And collapsing lists helps with hiding and managing the complexity.

Workflowy mapped really well to how I did my initial recording as well. I was recording only a couple of sentences at a time since it felt like recording continuously for a long time would increase the chance of errors in my speech. #ESL

Learning One: Later I found out that this is not the greatest way of going about recording. This level of granularity increases your editing and recording time a lot. An audio engineer recommended me that I should try to record in batch (like even trying to do the entire thing in one go) and try to edit/patch it at a later time.

Workflowy is a great tool and I have been using it for many years now for other productivity related purposes as well. It makes writing and note-taking easy and friction-free however it is not great at showing code snippets, images, or anything other than plain text. A couple of months ago, I came across Notion and was super impressed by its functionality. It is basically like Workflowy as it can also create infinitely nested lists but also pretty much anything else as well (checklists, formatted code snippets, Markdown documents, etc…) as far as organization/productivity tools go. I decided to migrate my work to Notion and continue writing over there. Its polished interface made me a bit more proud of my work in progress and seeded the idea that this body of text that I am developing could be more than what it’s intended to be. With features like code highlighting, image importing, etc. Notion made me see what my work could look like if it was on paper (that is digital paper). Aesthetics do matter.

At this point, I noticed I had written more than 150 pages. Using list item was keeping my approach very iterative and their easily collapsible nature was hiding my own progress from myself. I noticed I wrote a book without even realizing. So that’s how you write a book. One sentence at a time, my friend. One sentence at a time.

Learning Two: Little things add up. I was only writing a couple of talking points every day, but over time, they gave me enough material to create a book out of them. Having systems that allowed me to work in a consistent manner really helped with achieving a remarkable end result. The choice of the medium was also helpful as well. Lists were keeping things atomic and collapsible lists were hiding the increasing body of work from me, preventing me from self-sabotaging by triggering anxiety.

This is where things started to get a bit technical since I had a problem at hand. A book requires a continuous text whereas what I had was a huge bullet list of sentences. I needed to keep both formats for different reasons. A list was helping me with my approach to recording online courses at the time, whereas it was problematic if I were to create a body of text from this document. Enter coding.


Part Two: Choosing a Platform for Writing

I never really understood why programming might be needed for self-publishing a book. But it was starting to dawn on me.

Notion had an option to export my script as a Markdown file. I needed to convert this Markdown file which was essentially a huge nested list into a flat body of text. For this purpose, I used a node.js Markdown library. That worked well but the remaining problem was that it was now an HTML document with a huge unsorted list. Still a long haul from being usable.

At this point, I started to use another node.js library called Cheerio to be able to manipulate the resulting HTML into the desired form. Cheerio is a library that offers a JQuery-like syntax on the server-side for the manipulation of hypertext data. Even though Cheerio was helpful to a certain level, it still didn’t get me all the way there.

As a result, I decided to add type hints to my text. Basically, providing a set of tags to guide Cheerio to do the correct manipulations. I would be adding a {{ h1 }} to indicate the list header needs to be an h1 title, or add a {{ + }} symbol in front of some of the sentences to indicate that an item needs to be merged with the previous one to create a paragraph.

Around this time, something terrible happened, though. Notion decided to roll a set of poorly communicated updates that blocked me from working on my text for several consecutive days #firstworldproblems. At the time, Notion was a relatively new service so their desktop application was not great in caching the online data as well. I found myself unable to work on my productive hours because a cloud service was unavailable. This was a hard lesson in not putting too much faith in cloud services. Notion failed me several times over a couple of consecutive days making me realize that I needed to find a new home for my writing.

Now, this is not to say Notion is a bad service. It is actually an amazing product. But I made the mistake of relying on it too much at an early stage in their product development where things were still very much in flux and their offline application wasn’t great at syncing with the online platform.

Learning Three: You need to have a way of working offline if things are to become inaccessible. You can’t assume cloud will always be there.

I started to look for alternatives that would replace my workflow and quickly settled with StackEdit.io. It allows you to write Markdown in the browser using the local cache. It can sync with Google Drive in an easy manner as well. This is around the time that I also found out that my previous way of doing recordings wasn’t very inefficient. Recording (and writing) a script only a few sentences at a time is a bit too granular, and slows you down when it comes to editing. This prompted me to transfer all my writing into StackEdit.io as a flat markdown text, completely getting rid of the list structure.

Learning Four: There are no right or wrong tools when you are just starting out. The concept of right, wrong, best exists in the realm of clickbaity titles of the internet. If I were to get obsessed with my tooling from the get-go, I wouldn’t be able to do any meaningful work. You will get things wrong, and that’s alright. It is part of the journey and the learning process.


Part Three: Coding for Publishing

At this point I had all my writing in StackEdit, I also got rid of all my previously added style hints as I could just be leveraging the Markdown syntax moving on. StackEdit can automatically sync with Google Drive so I had a way of fetching the files I was working on for offline processing as well. I finally had a reliable way of building a pipeline around things. This is where coding came into play in full force.

I wanted to be able to create multiple outputs from my Markdown source file. I was already working on sections in the book that were more in-depth which I started to consider having as a premium offering. I wanted to create an ebook out of the material but I also wanted to be able to display the same material on my Jekyll website as well. This already implied four different variations to the source text, like online-free, online-paid, offline-free, offline-paid. Another difference between the online and offline output was going to be that I could be using video (gif) files in the online version but these same visuals needed to be static images in the offline version.

Now I was starting to understand how programming can help with my process. I had a single authoritative source and multiple output formats that needed to be managed and this started to look more and more like an automation problem.

I started to build node.js scripts to establish my workflow. I was already syncing my Google Drive to a folder on my disk. So I wrote a script that would fetch the desired files from Google Drive to my working folder and place it in a staging area.

'use strict';

    const fs = require('fs');

    const SOURCE_DIR = '/Users/username/Google Drive';
    const TARGET_DIR = './staging';

    // remove the target content synchroniously
    let targetContent = fs.readdirSync(TARGET_DIR);
    targetContent.forEach((file, index) => {
        let targetFile = `${TARGET_DIR}/${file}`
        let isFile = fs.lstatSync(targetFile).isFile();
        if (isFile) {
            fs.unlinkSync(targetFile);
        }
    });

    fs.readdir(SOURCE_DIR, (err, files) => {
        files.forEach((file) => {
            if (file.startsWith('p5js-') && !file.endsWith('.gdoc')) {
                fs.createReadStream(`${SOURCE_DIR}/${file}`)
                    .pipe(fs.createWriteStream(`${TARGET_DIR}/${file}.md`));
            }
        });
    });

I was hosting the images that were referenced inside the Markdown file on imgur. I built yet another script that would download these images into the staging folder. I needed images to be on my local for compiling an ebook.

Then I created another script that would pre-process the files in the staging area and move them to their corresponding folder that is determined based on the target destination. To be able to render different content in between online and offline versions, as well as the paid and free versions, I decided to use Nunjucks templating language. This solved two main use cases for me:

Using Nunjucks, I was able to conditionally render content based on the targeted output format. For example, a gif in my document might be represented like this:

{% if online %} 
![01–01](http://i.imgur.com/<id>) 
{% else %} 
![01–01](images/01–01.jpg) 
{% endif %}

Having an if-else statement, I can set a variable inside my preprocessing script to decide which conditional to execute and use this data to render the Nunjucks template.

const data = {online: true}; 
let renderedContent = nj.renderString(content, data);

Nunjucks also allowed me to create variables. I could use a variable like {{ item }} in my Markdown text which can render to a value of book or course depending on the destination I am targeting. (I ended up creating an interactive course using the book material where I needed to refer to the material as a ‘course’, more on that a bit later).

Using the pre-processing script I was also able to manipulate the front matter I had in the original Markdown file using the node library called front-matter. This needed to be done since one of my target output format was a .md file for Jekyll and I wanted to automatically add some additional front matter attributes for Jekyll to parse.

This all might sound like terribly over-engineered and unnecessary. And there is a chance that it might very well be. But one thing that I was happy about this process was that even though I yielded to the developer inclination of tooling up in the face of the slightest problem, I didn’t get overly obsessed about building generic, scalable solutions. All this code I wrote is actually pretty ugly and embarrassing to be shared here but the point is to move fast and build automated solutions in a way that’s not going to waste your time. The primary objective is not building systems to deliver your content, it is delivering the content, however, possible. You should be doing things that don’t scale.

Learning Five: Definitely read Paul Graham’s essay on doing things that don’t scale. Things that are not efficient can get you massive leverage in the short run. But if you are to get bogged down by the concerns of scalability when you are only starting out then you might miss out on opportunities of growth and sources of motivation like the sense of delivering value to people which might impede and even ruin your progress.

One technical decision I regret is using Promises for my file system operations. I think I was trying to prove myself that I was comfortable with using them but they were completely an overkill for my circumstances as I didn’t have any performance concerns. The excessive usage of Promises started to have a mental toll, where I just wanted to move fast, as they are not as straightforward as synchronous operations would have been. Game developer Jonathan Blow has a great talk on optimizing for the cognitive load when developing personal projects. Granted this is not at the scale of anything he is working for but if you are just working on something that needs to work regardless of how you need to make sure that it is usable as possible. Don’t try to be smart, because most days you will dumber than your smartest self.

I also created a post-processing file that I would run for some specific purposes. For example sending the document to a copy editor ( I worked with a freelancer on Fiverr), I didn’t need to have the code snippets in there as they were ballooning up my word count and hence affecting the pricing. Having an automated workflow allowed me to remove them easily. There was an instance where I was faced with an opposite problem where I just needed the code snippets. This was again solved quickly by using the post processing file to selectively remove the target elements (anything that is not a code snippet) from the files.

I ended up publishing my work on three different platforms. I first published the book on Leanpub. Having already established a pipeline, it was very easy to integrate my work with Leanpub and use their Github integration. After pushing my work on Leanpub, I exported an unbranded version of the book using their tools and placed it on Amazon Kindle Store. Zapier has an amazing blog post on the self-publishing platforms. It is a must-read for anyone that is interested in this space.

The most amazing discovery for me was coming across Educative.io through a post on Hacker News. Educative.io is an online course creation site where you can build interactive courses using blocks that allow you to add executable code snippets (among many other things) inside a document in an easy manner. This allows you to create documents that are similar to Jupyter notebooks. Transferring my source text to their platform was a breeze as it uses the Markdown format as well.

I am not claiming that my workflow was perfect. One big shortcoming was that gathering online feedback on my text from other people was pretty hard. For that use case, working in Google Docs would have been much more useful. But unfortunately, Google Docs doesn’t offer a great way of working with Markdown files.

Also using Nunjucks templating in your source text introduces a little bit of an overhead as you can’t just copy-paste text, you need to process — compile — it first. But considering the efficiencies that were gained, I find this to be a reasonable tradeoff.

This summarizes my journey. If there is a final lesson to be derived here I think it is not to be overly obsessed with tools and best-practices from the get-go and just start with creating things. It is the content and the product that really matters and all the other concerns are secondary, at least initially. Don’t let yourself get slowed down by choices, you probably don’t know enough at the beginning to inform your decisions so it is important to get started and iteratively adjust according to your emerging needs.


congrats

Thank you for making it this far! Feel free to check out Coding for Visual Learners and feel free to reach out to me on Twitter or through my website with your comments, suggestions and questions.

Also thanks to Hoi-En and Leigh at Myplanet for reading drafts of this.

Comment

Comment

Do Things That Don't Scale

In his essay titled "Do Things That Don't Scale" Paul Graham talks how in the initial stages of a startup, it can be really hard to gain traction. That’s why you shouldn’t wait for users to come to you but try aggressively to recruit them. This idea might not appeal to the engineer mindset as it might appear to be not scalable or sustainable. But even small gains at a very early stage can add up to significant numbers in the long run. Besides you don’t have anything to lose at an early stage but have a tremendous payoff. Also the fact that it is not scalable means that it is to your advantage to do this since big companies can’t compete with this approach. Paying strong attention to your existing users can gain you customers for life.

To find users to recruit manually, you might want to create something that solves your own problem. Then you can recruit your peers as users. Or you can do a comparatively untargeted launch and try to see who is more enthusiastic about your product. This happened with Pinterest when they noticed there was a strong design interest for their product. This is not only about finding and acquiring users though, but also making them really happy. The feedback you will get from engaging with your early users will be the best you will ever get.

Couple of things to keep in mind that can be derived from this rule. If there is a subset of people in your target group that you can target to get a critical mass of users quickly, then you might want to do so. This will help keep things a bit more focused to begin with (like if was for Facebook when it was first launched) and also the subgroup will feel special because your product will appear targeted to them.

You can run a business that is based on automation manually, until you can’t. This can allow you to launch a product early and acquire users that way.

That’s what’s meant by ‘do things that don’t scale’. These things fly on the face of efficiency but will get you leverage in the short run. But if you are to get bogged down by the concerns of scalability when you are only starting out then you might miss out on opportunities of growth and sources of motivation like sense of delivering value to people which might impede and even ruin your progress.

You should really check out the original essay though, it is full of amazing advice for startups at their early stage.

Comment

Comment

Unity - First Impressions

For the last few months I have been finding it harder to ignore the hype around Virtual Reality. It seems to have reached a critical mass since something related to it pops up wherever I take a look. I won't be recapping all the positive indicators out there, but the magnitude of the hype indicates an expectation for a considerable financial payoff from the stakeholders invested in this technology, which are numerous and includes tech giants such as Facebook and Google.

This is all very exciting since a decent implementation of Virtual Reality have been long overdue and the recent reviews that are coming in from the people that had a chance to try it out are very promising. What is also great is that initiatives like Google Cardboard is making it very affordable for developers to experiment in this front without a potentially forbidding hardware investment.

In a recent Hack Day at our company, we had one of our developers get the Google Cardboard VR sample scene working on his Android. His experience and enthusiasm encouraged me to take a closer look to try out this technology for myself. But being more experienced in using animation software than writing graphics code I decided to take the path of least resistance - or code -, and use the Unity SDK for Cardboard to bring projects from over Unity.

One primary comment that I have heard that warranted this decision for me was that the necessity to use OpenGL without any higher level abstractions when working with the Android Cardboard SDK. My impression of OpenGL is that it is rather low level to be used for any complex scene work, at least at my skill level, so that was not really an option.

So I went ahead and installed Unity. I haven't installed the Cardboard SDK yet, as a matter of fact I don't have the cardboard itself as well. I am taking some time to learn Unity and so far it has been a rather pleasant experience. The application has numerous similarities with other animation software out there which makes it very approachable. I started off with their official 'Roll a Ball Game' tutorial which in my opinion offers an excellent introduction to the application. I am impressed by the seeming ease of building and deploying a simple game. The steps involved didn't feel redundant or convoluted at all. It seems to be a well thought-out and designed tool that I am happy to have finally tried out.

I am planning to continue with their official tutorials to get a better hang of the tool before jumping into the Cardboard/VR world. I am actually not very interested in game development and looking at VR as a possible avenue for visualization, interaction or presentation. But even then, Unity has a ton to offer to for all those domains and so far I am liking what I see.

ttps://developers.google.com/cardboard/

https://unity3d.com/

Comment

Comment

Observing that maximum number of people who can productively, simultaneously work on CSS is one.
— @threedaymonk

Comment

Comment

Review for the talk "Build Scalable, Automated CSS"

tl;dr

You spend most of your time reading code, so optimize your CSS for readability. Use descriptive class names and don't restrict yourself in terms of the character length. Don't use element selectors except for reset or sitewide base styling purposes, as you would be trying yourself to a specific element and markup. Semantic markup using elements might be unnecessary given the rise of semantic attributes.

Summary

Below are my impressions and notes from the talk "Build Scalable, Automated CSS" at CSSConf Asia from speaker Christian Lilley.

($speaker: "Christian Lilley")

In his talk $speaker argues that just like your Javascript, your CSS should adhere to some basic programming principles such as the following:

  • Be dry (Don't Repeat Yourself)
  • Be maintainable (Write for updates and debugging )
  • Don't optimize prematurely.

There are some general rules that you can follow to write more maintainable CSS. According to the $speaker; one thing that can be done is to stop yourself from using hacks such as 'float'. Float was pretty much only invented to have text flowing around an image. Using it beyond it's intended purpose ends up creating ugly - incomprehensible code.

I found myself semi-agreeing with this point. I too think that a layout that is based on some convoluted usage of float's can make be pretty incomprehensible but having a float based layout that is abstracted from you through convenient class names (like what Bootstrap would provide) should not be an issue. Given that it's an abstraction, it should also mean that the underlying implementation could be changed by whatever that is most convenient when necessary.

The reason I am not quite on board with inline block is I couldn't find a reliable way of solving the white space bug yet. And the word is that flexbox might not be the best choice in terms of compatibility but is definitely getting there. But it is worth mentioning that flexbox seems to be the most convenient alternative to build layouts easily and efficiently.

During a development process you use most of your time trying to understand your own code, so optimizing for readability is a huge gain in terms of efficiency and your own performance. $speaker claims the usage of element names as selectors is the worst idea ever, since you would be tying yourself to a specific component in your HTML architecture but also be writing a pretty generic looking CSS. You should instead be using descriptive, intelligible class names. Don't worry about having too many classes around or having long names for them. They don't bite and there is no issue whatsoever in leveraging classes as extensively as you can. They are as efficient as it gets in terms of performance. The $speaker goes on to say that even 'the cool selectors' such as n-th-child should generally be avoided if possible due to sub-optimal perfomance and ambiguity problems.

Namespacing is a good start but is not a substitute for a more comprehensive naming strategy. It is like saying 'Statistically speaking I am somewhat sure that a collision won't happen'. Well it just might.

The only good usage of element names are for 'normalize' or 'reset' files - which you should be using by the way even though they are not enforced -, or a style change that you need everywhere now and moving forward (like removing underlines from anchor elements)

At this point in time it goes without saying that CSS preprocessors are good. in fact they might be the best thing since sliced bread for a bread loving CSS user. Make use of those variables (like I am doing in this review) and also the mixins, import statements, etc... Though one thing to beware of could be the use of descendent selectors. The problem with them can be that you would be tying yourself to a specific markup increasing coupling in between your HTML and CSS but also in case you were also using element selectors you would find that you can't put new, external, components in your markup in a sane manner. Your selectors would have unintended consequences.

Some preprocessors can give you a nesting syntax for you to create name prefixes when flattened such as: .main-content : { &__left-nav {} compiling to: .main-content__left-nav {}

His last point was the most interesting for me, the $speaker makes a compelling argument about not sweating it too much about using some of the new semantic html elements such as <section> or <article>. To quote him directly:

"Don't feel guilty (if you can't use semantic elements). The era of semantic elements for machine readability is over in favor of semantic 'attributes' (see: wai-aria , microformats)"

Even though I found the entire talk to be too long given the amount of information it contains it was still an enjoyable watch.

Comment

Comment

Reflecting on Pycon 2015 - Catherine Bracy Keynote

It is going to be hard to summarize the last few days in terms of the amount of information that was flowing around me, considering that at certain times I was absolutely overwhelmed by this flux. There were talks I attended where I didn't even attempt to understand what was being talked about and other times where I had a brief idea but still didn't put any mental effort in capturing the information as I saw no immediate applicability. I have this unrooted assumption that captured information can pose an overhead to your brain if you don't have any short-mid term plans of retaining it. But there were also moments of inspiration and knowledge acquisition and my intention here is to summarize what those moments were.

“Good governance and good policy are no inextricably linked to the digital.”

The conference kicked-off with an inspiring keynote by Catherine Bracy, which was essentially a call to arms for programmers to take on more responsibility in their communities by putting their skills into good use. She is the community organizer for 'Code for America' and founder for 'Code for All' which are non-partisan organizations that engages in digital projects alongside with provincial governments to improve the state of bureaucracy. Their hope is the by improving the day-to-day interactions of citizens with the government they can improve the relationships between the people and their government which helps democracy and can level up to a deeper civic engagement. She started with giving examples from the disastrous government tech projects that failed to deliver and went on to talk about those that had positive impacts on the lives of people who benefit from the improvement in social services the most. Examples that come to mind are cell phone services that inform people about presence of a nearby internet connected public computer, or an easy crowd sourcing website for other people to help with hydro bills of people who can't afford them. One interesting project she mentioned was 'Own a Drain' where residents can own a drain hole in their neighbourhood and assure it is not clogged in case it is to rain. The whole service they provide is gamified so it makes it easy to participate. Catherine emphasized that good governance now is intimately linked with digital and we need to participate in this system of governance at least at the local level whenever we can if we want to see an improvement in our services. It was a great talk, well delivered and had a strong activist undertone which was empowering. It was an unfortunate but a truthful declaration she made during the Q&A that "it is now the rational choice to choose not to vote".


Find the talk here:
https://www.youtube.com/watch?v=LOZk5ttyC9Y

Comment

Comment

Decision Making

“a decision is the act of committing to a specific plan of action. the word ‘decide’ comes from the latin ‘decidere’ which means to ‘cut off’. When you make a decision, you are cutting off the other possible avenues that you could explore, leaving only the path you are committing to. If you are not cutting off viable options, you are not really making a decision.”
— Josh Kaufman

Comment

Comment

McDonald's Theory

I use a trick with co-workers when we’re trying to decide where to eat for lunch and no one has any ideas. I recommend McDonald’s. An interesting thing happens. Everyone unanimously agrees that we can’t possibly go to McDonald’s, and better lunch suggestions emerge. Magic!

This is a good read on how to get yourself started. 

https://medium.com/what-i-learned-building/9216e1c9da7d

Comment

Comment

A Case For Starting From Scratch

A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.
— John Gall's Systemantics

Comment