A Complete Rewrite: Stenography Lesson Generator in NodeJS

My wrists were hurting. So 8 years ago the CIO of the company I worked at told me about stenography: the process of writing shorthand to write over 200 words per minute on a fancy keyboard by pressing multiple keys together at the same time. An ergonomic alternative to the painful contortions of keyboard shortcuts on a regular qwerty keyboard. But how would I learn it?

This is a story about independent development and the adventure of undertaking a complete rewrite.

The result is the Typey Type CLI, which took an hour’s work down to 10 seconds. I now use this to build steno lessons for Typey Type for Stenographers to help students drill and study steno.

Humble beginnings

Initially I extended an open-source Flash tool for basic lessons built in Haxe but Flash was dying and I quickly reached limits. So I began building a web app in React called Typey Type for Stenographers.

I needed to create:

• fundamental lessons to teach the basic structure and theory of how to string sounds together to make words
• full sentences to drill the realistic process of typing prose
• lessons for coding syntax and symbols.

I used a lot of material from the public domain like Aesop’s Fables and stories by Virginia Woolf via project Gutenberg.

The project started out as a little app for myself but eventually became a popular choice in the open-source hobbyist steno community for a time.

A Ruby and bash backend to generate lessons

Initially I hacked together a gnarly static lesson generator. To produce the material, I strung together some Ruby and shell commands to produce a list of words that met some criteria like “one syllable words with multiple consonants” or to split a passage of text into words. Each word or bit of punctuation needed a hint to show you which keys to press. Together, words and hints made up the static lesson files.

Learning to sling command line tools

Throughout the process I built up knowledge of various Unix tools. comm to compare sorted lines of text in different files, sed to stream text and replace text in place, tr to transpose characters, and more. I developed an appreciation for the darker arts of regex. I also learned how to use them altogether in a pipeline.

I credit much of my knowledge of the command line to this project.

A React front end

The React app grew alongside the static lesson generator, providing an interface to practice lessons, get fast feedback, and track progress. I added new features like the ability to use your own steno dictionary, which meant the front end also needed to look up steno hints for how to type words but using a custom dictionary. This was something more dynamic than the static lesson files were designed to handle.

I was halfway through the custom dictionary feature before I realised I was effectively rebuilding functionality in the static lesson generator. (We’ll come back to that.)

Open source

The code started privately but once the app got traction I eventually open sourced the front end, the data, and my steno dictionaries.

I did not, however, share the static lesson generator…

oh no

The static lesson generator evolved in a meandering way as I realised new requirements and bolted extra things on. I didn’t know the things I know now so over the course of 7 years it grew into an unmaintainable mess.

It would have been unapproachable for anyone to navigate and would probably have broken on any other device.

Long build times and unpredictable results

To change any lesson meant rebuilding all 400 lessons. To rebuild every lesson took 6 whole minutes. It was not obvious where to make changes so it would take 6 minutes just to figure out the bit I changed was the wrong place.

Building a new lesson would take several iterations to smooth out surprises in material like stray characters, misspelled words, and backticks used as opening quotation marks. And there’d be surprises in the steno hints too (what should someone press for a capitalised word followed by a question mark?). It was full of unpleasant surprises.

Calling it

I knew it couldn’t go on like this. I was reluctant to create new material and maintenance became an emotional burden.

An interlude on motivation in open-source projects

For 8 years of maintaining this little community project I’ve had 2 main sources of motivation: Building for me, and people’s feedback.

1. Build for yourself

Typey Type solved a problem I had. If I was the only person that ever used this, I knew it would be time well spent.

All the same, I was thrilled when other people took interest in Typey Type and started using it!

2. Let people know you love what they made

I was absolutely delighted when people wrote to me to tell me that they used Typey Type.

I was unbelievably stoked when people told me they loved Typey Type. Or that they used it every day. Or that they learned steno because of it. Or that they used steno for work every day because of it. Or they learned to code in steno because of it.

And the ideas! What else would make it great?

And feedback! What worked, how people used it, how they thought about study and learning.

And hearing about the problems! Within Typey Type or beyond. Hearing about what people are trying to achieve excites me to help make it possible for them.

Every time someone shared something with me, that spurred me on to build something new. To make it better. To thank them for taking the time to share their thoughts.

I had never realised how much development is probably happening in the world purely because people said nice things.

Today’s a good day to send your local indie developer a nice message.

Typey Type’s not for everyone and that’s great

Some people don’t jive with Typey Type. I get that. It has some opinionated choices in it. It’s drill-oriented, which demands diligence, dedication and a willingness to smash through lots of repetitions. (That said, I have started adding some Games for play.) Stenography is a hard skill to learn. Professional stenography schools have a 90% drop out rate!

There weren’t a lot of options for interactively practicing steno when I made Typey Type but now there are heaps!

One of my favourites is Stenojig by Josh Grams. There’s also Practice Plover by Plants, a fantastic new resources that blends theory and practice together.

There’s a good mix of tools now for different learning styles, study habits, and needs. Especially because people learn steno for different reasons like ergonomics, coding or computer use, transcription work, live captioning, and so on.

What gets me down

If Typey Type is not for you then I wholeheartedly recommend checking out Stenojig or Practice Plover.

There are a couple folk, however, that not only dislike Typey Type because it doesn’t suit their preferences (which is fine), but actively and frequently go out of their way to tell people not to use it. (It’s mildly disappointing that they haven’t discussed their reservations with me.)

What used to be a fun community, working together, feels more judgemental and less collaborative now. As much as I’d like to ignore the haters, it’s (literally/technically) unavoidable in the main community space now so I find myself spending less time there. Which in turn means I hear less from people that provide feedback about Typey Type or discuss steno problems they’d like solved. So it actually has the effect of sapping my motivation indirectly by keeping me away from the engaged people that motivate me.

This brings me back to “Let people know you love what they made”. It can help make up for the noise.

Succession planning in open-source projects

These days when I start new projects, I also consider how I want them to end. What might it look like for my project to be wildly successful and then finish? How might I celebrate its impact and hand it over to someone else or unwind it? Interests and capacity change over time so for projects that people depend on, it’s useful to have open-source code or multiple maintainers. So for Typey Type, I knew I would be dissatisfied if I ever parked it without making the code available. It was important to open-source the last piece of the product.

Embarking on a rewrite

Where were we? The lesson generator rewrite!

While I could have refactored it into something sensible in place, there were compelling reasons to rebuild:

• The front end had similar behaviour in places. If I used Node.js, it would be possible to share and reuse code.
• The requirements had changed. I knew a lot more about stenography and I had more ideas about what people needed to learn it, especially their own dictionaries.
• I had been writing JavaScript and Typescript professionally for years with much greater skill and speed with them.

The biggest mistake I made the first time around

I wrote no tests.

Each build would produce lesson content with hints that I committed in git to track differences and spot areas that needed refinement. Every cycle should have prompted me to write a test to capture the complexities of stenography. But it didn’t! I didn’t understand the value of tests back then.

This project has shown me in painfully obvious ways how much better it would have been to have tests.

For the record, my second biggest mistake was mixing up source and target files. A word list was needed to produce a lesson dictionary. The word list and dictionary were then needed to produce a lesson file. The word list, dictionary and lesson was then needed… to build the lesson dictionary…? It was gross.

A rewrite without tests

In the absence of tests, having all the lesson content already served as the goal: if I could perfectly reproduce the same lesson files, I would know the rebuild was successful.

I incrementally added one lesson and one lesson category at a time. I tracked the content in git and copied the target lesson over from the previous project.

Each lesson would add new complexity that would require development like handling capitalised words, punctuation, hyphenated compound words, and so on. For each difference I spotted in the results of building the main material, I created a new test to capture that nuance.

The message here is that in the case of a rewrite, it’s invaluable to have tests, add tests as you rewrite, and if there are no tests, find a substitute for defining ideal behaviour.

Sharing with the front end

As I stumbled into the parts that partially existed on the front end, I pulled them into a directory called “shared”. I refactored them heavily on the backend, quickly seeing the impact any change would make across dozens of lessons at once. I was then able to write tests to use in the front end.

In some cases the front end behaved more cleverly than the Ruby static lesson generator. In others, the Ruby backend created better steno hints. As a result, I ended up completely rebuilding a core piece of functionality into a recursive, comprehensive function that achieved the best of both worlds.

The results

• A publicly published open-source project.
• A NodeJS CLI that can share code with the front end.
• Smarter lesson hints.
• 500 unit tests.
• Comprehensive TypeScript types.
• I can rebuild every lesson and all other project files in 1 minute instead of 6-12 minutes.
• I can rebuild 1 lesson in 1 second.
• I can iterate on individual lessons independently. 10 iterations on one lesson used to take an hour; it now takes 10 seconds.

Left to do

1. I’ve done no performance optimisations even though performance was one of the goals. Most of the speed gains have been achieved by better architecture and using Make to run steps in parallel and only as needed.

2. Extracting the shared code into a node package. For now it’s still copied and pasted across two repos.

Don’t underestimate how long a rewrite takes

It takes way longer than you want.

Even knowing in vivid detail how this now needed to work, there were a lot of details to iron out.

It took about 940 commits and 16,000 lines of code. At the end, the only user facing difference is that some of the lesson hints are smarter and suggest better capitalisation and punctuation. No new “features” appeared as a result. Classic rewrite.

Don’t underestimate how much satisfaction it will bring you

I cannot overstate how happy I was when this was released.

It was a huge milestone to make the first deployment of Typey Type with lessons generated by the new CLI. It was a fantastic moment when I published the repo.

But the best moment came when I built the first new lesson with it. I have a backlog of lesson ideas that have been too hard to tackle in the past. I picked off one at random and built it from start to finish in a couple of minutes. The first build took one second, then I looked at the results and decided I wanted to remove a word, so I deleted the word and rebuilt the lesson again in another second. And it was done. That was it. What had previously been hours of headaches and wrangling was done in moments on a whim.

Seriously, I cannot overstate the satisfaction I felt in that moment. So I built another lesson! Why not! 🥳

Lessons learned

1. Write tests. Especially when you’re learning something new.

2. Indie development is hard work.

3. Rewrites are time consuming and oh so satisfying.

Of course I also learned a bunch of technical pieces along the way. For example, I delved deep into Makefiles. After publishing the code, other people were able to run it. Yay! Which meant I discovered the fun ways it could fail on other machines! Among other things, I found sort behaving inconsistently and learnt about using the LC_ALL environment variable to override all the (user’s) locale settings with a simple, consistent one for a script (the Makefile in this case).

What’s next?

I’m not sure. I’d love to keep tinkering and I probably will. I’ll also admit it’s been a long-running project, there are other steno options out there now, and the haters are getting louder and my motivation is waning.

Aside from the steno of it all, having a successful side project in production that thousands of people actually use is really cool. It’s also a fun place to try things out. For example, it’s the only product I’ve worked on where dark mode is supported in any way. I’ve used a lot of tools in anger that have enabled me to use them effectively in my professional life too.

Anyway, that’s my story about a complete rewrite and being an indie developer. Thanks for reading.