Code Expectations

When you build release a version of your software you should expect that it will compile and run the same way today as it did yesterday. But your code likely depends on one or more libraries/frameworks/what-have-you from other developers and what's worse than having your build break because you made a change is having it break because someone else made a change. An obvious solution is to use a specific version of that dependancy consistently and update only when necessary. This is why we use version numbers (and tag releases) so that we have a common point of reference when using and talking about code. But how different is version 0.2.0 from 0.2.1?

Enter Semver

Semver is a semantic versioning sytem which seeks to standardize the meaning of version numbers. Version numbers come in the form of major.minor.patch and you increment each as such:

• MAJOR version when you make incompatible API changes
• MINOR version when you add functionality in a backwards-compatible manner, and
• PATCH version when you make backwards-compatible bug fixes.

This (hypothetically) should take the guesswork out of versioning and provide at the very least  semblemance of a guarantee to those who would use your software. With this system you could target 1.X.Y or 1.2.Z versions of a library you depend on and have a reasonable expectation that for any value X, Y or Z your code should work. Node even allows for this type of version targeting with their package manager npm. You can target ^1.2.3 which will be valid for versions greater than 1.2.3 and less than 2.0.0. Or you can be safer and target ~1.2.3 which will be valid for versions greater than 1.2.3 and less than 1.3.0.

Problem Exists Between Chair And Keyboard

The main issue here is that you have to be confident that the maintainer of the software you depend on will adhere to this and that they're testing process is robust enough to catch anything that will break how you use it. If breaking changes are made in a patch bump you're effectively SOL if you rely on approximating versions. You can contact whomever maintains the software that broke and inform them - or help them fix if it's open source and you have the time - but who knows when they'll have time to get to it. A simple hotfix for this would be to narrowly target the exact version that was last working. This way the next time you (or your Continuous Integration platform) run npm install or update your tools you won't be surprised with errors.

A best practice to take away from this would be to only use exact versions for dependancies to provide consistency for your build and assurances to your users. All the fancy automated testing pipelines in the world mean nothing when their green badge on your github is only valid for the exact point in time that it was run.

One of the suggested routes to get the API URL is to use their /resolve endpoint which you give a regular URL such as https://soundcloud.com/jackconte and it responds with a 401 redirect to  https://api.soundcloud.com/users/19328797. This is nice, but the reason their flash widget integrated well into hubski was a user just had to give the regular URL and we could generate the code for the widget from it. To use the HTML5 widget - and get the benefits of being able to embed audio in mobile browsers - we would have to set up an extra API call with Soundcloud to get a client id in order to use the endpoint at all.

Another way is to use their /oembed endpoint. You can send the URL and get back a JSON payload which includes the full html for the widget. This simplifies things greatly and you can make a client side request like this

<div id=\"sc\"></div>
<script>
var URL = \"https://soundcloud.com/jackconte\"

$.getJSON(\"http://soundcloud.com/oembed?format=json&url=\" + URL, function( data ) {
  $('#sc').append(data.html);
});
</script>

to render the widget. This is especially useful if they ever change the widget HTML format and you don't need to register a client id.

Ideally, I'd like to do a simple transform on the URL and spit out the HTML for the widget à la youtube. I was trying to set this up and I couldn't find any easier way of doing this so I thought I'd see what would happen if I shoved the URL in like so:

<iframe id='sc-widget' width='100%'' height='166' scrolling='no' frameborder='no' src='https://w.soundcloud.com/player/?url='https://soundcloud.com/jackconte&show_artwork=true'></iframe>

To my surprise it just worked, as you can see here

I'm not that surprised that they haven't documented this that well; there are a lot of benefits of forcing developers to register and go through your API. For now it serves its purpose, but I'll likely change to a more robust client side version using oEmbed when Hubski's frontend is refactored.

Arc's Markdown

What we do use is a modified version of news.arc which has a very simple implementation which only allows for italicizing, code blocks, and clickable links. It goes character by character and checks for a asterisk or double space at the start of a new line and then looks ahead for an applicable end character. After it parses the correct characters it then saves the parsed string in the appropriate file for the comment or submission. If you need to edit your text it then calls an unmarkdown function to convert it back. This works for pg and he and Nick Sivo (kogir) don't seem to have any desire to extend it.

This is the function which converts markdown strings to html:

(def markdown (s (o maxurl) (o nolinks))
  (let ital nil
    (tostring
      (forlen i s
        (iflet (newi spaces) (indented-code s i (if (is i 0) 2 0))
          (do (pr  \"<p>< pre><code>\")
            (let cb (code-block s (- newi spaces 1))
              (pr cb)
              (= i (+ (- newi spaces 1) (len cb))))
            (pr \"</code></pre>\"))
          (iflet newi (parabreak s i (if (is i 0) 1 0))
             (do (unless (is i 0) (pr \"<p>\"))
                 (= i (- newi 1)))
            (and (is (s i) #\\*)
                 (or ital
                     (atend i s)
                     (and (~whitec (s (+ i 1)))
                          (pos #\\* s (+ i 1)))))
             (do (pr (if ital \"</i>\" \"<i>\"))
                 (= ital (no ital)))
            (and (no nolinks)
                 (or (litmatch \"http://\" s i)
                     (litmatch \"https://\" s i)))
             (withs (n   (urlend s i)
                     url (clean-url (cut s i n)))
               (tag (a href url rel 'nofollow)
                 (pr (if (no maxurl) url (ellipsize url maxurl))))
               (= i (- n 1)))
             (writec (s i))))))))

Up until now we have hacked onto that implementation all of the formating we currently have. For things like bolding, quoting and strikethroughs that was easy enough: look for a character (\*, \+, |, or ~), look ahead for its matching character, replace as necessary. As we added more and more functionality it got progressively trickier to keep things from breaking.

Problems

One of the obvious problems that arises is that it doesn't account for nested markdown. For instance, when it sees a code block it prints out the contents of the block with the appropriate tags wrapped. It skips over checking for any more markdown withint the tags.

Another more subtle one is that it stores the html version of the string. What most websites do (from what I'm told) is store both the markdown and html versions. This way you only have to deal with a markdown to html parser and not the reverse. The benefits of this become obvious when you want to have unordered lists. For instance, according to the spec -, \+, and \* are all valid bullets for an unordered list. When converting back which do you choose to display to the user?

Solution

So in order to allow for nested markdown I decided to rewrite how we dealt with it. First we go through our input string and break it up into a list of tokens. Simple tokens are things like \*, \+, |, ~, \\, [, ], (, ), and newlines. More complex ones include image and video links which get embedded. Luckily, if we separate things by spaces we'll get links by themselves. For instance we want to turn, "This is \+bolded\+.
Here's a embedded link: http://i.imgur.com/zAuxnTw.gif" into

(\"This\" #\\space \"is\" #\\space #\\+ \"bolded.\" #\
ewline \"Here's\" #\\space \"a\" #\\space \"embedded\" #\\space \"link:\" #\\space \"http://i.imgur.com/zAuxnTw.gif\")

Links provide their own problem. We want to be able to split up links of the form \[Text\]\(url\) while allowing the url to contain parens. The way I ended up doing it is only tokenizing ('s which are preceeded by a ] and counting the number of open parens in the current buffered text.

From here we can use this list much in the same way as the original implementation but instead go token by token and build an html string. As we go through the list we can keep state through either tail recursion or local variables in a loop (which is what the original uses). I ended up using both in different places to simplify things. In this way we can signal when the text is in a bolded or italic state without skipping over anything.

Another thing we added was an escape character '\\' in case you wanted to use an asterisk or two on their own.

Unfortunately much of our text data is still only stored in html string form. One day I will write a script to go through all our files and add the markdown versions of the text. We plan on moving our data to a proper database anyhow, so we'll probably end up doing that then as it will entail rewriting all our 'database' calls anyway.

Conclusion

This is definitely a better system for dealing with markdown and will make it easier for us to add more things in the future should the need arise. I've come to realize, however, that for the greatest extensibility and generalization I probably should have gone with more formal approach akin to a markdown compiler. When I started this I didn't know anything about how compilers worked, but if I were to do it again I would implement a relatively simple compiler with three phases: Lexical Analysis, Semantic Analysis, and Code Generation. Lexical Analysis would be similar in that it built a list of tokens. Semantic analysis would take each of these token and create a parse tree with appropriate nodes (God I long for a nice typing system). Then we would just walk through the tree and print out the approriate string. I believe with this approach we would be able to represent everything you ever wanted in a markdown. Hopefully with the restructuring of hubski, however, I won't need to.

Making javascript human readable with jQuery

First things first, we want to the to be code easy to understand and thus easier to maintain. For instance, can you guess what this piece of code does?

window.location = document.getElementsByClassName('gridfeed')[0].childNodes[feedSelectionIndex].childNodes[1].childNodes[2].childNodes[1].childNodes[1].href;

It's not easy to tell without following the through each set of children. I think it can be agreed that this is easier on the eyes.

feed.currentNode.find('.savesplit > a:contains(\"hide\")').click();

I'll admit, something as simple as selectors blew my mind when I first saw them. One disadvantage to using jQuery in a userscript is that in Google Chrome the file containing jQuery is out of the scope of the userscript. In order to access the file you need to inject your script into the page.

function addScript(callback) {
    window.onload = function() {
        var script = document.createElement('script');
        script.textContent = '(' + callback.toString() + ')();';
        document.body.appendChild(script);
    }
}

function main() { /* Userscript goes here */ }

addScript(main);

This uses a simplified version of the addJquery function by Erik Void. We don't need to load jQuery ourselves because Hubski does that for us; we merely need it in our scope. Also, loading our own jQuery in addition to hubski's introduces its own set of problems.

Abstracting functionality with modules

Each piece of major functionality can be wrapped into its own module. Each module has the general form

modules['moduleKey'] = (function() {
    var Module = {
        init: function() { /*initialize stuff*/},
        isLoaded: function() {/*determine if module should be run*/}
    };
    privateMember;
    privateFunction() {}
    return Module;
    }());

Every module needs at least two function: init() to initialize the module (attach event handlers, insert spans, etc) and isLoaded() to determine if the module should be run on the current page. This makes loading all applicable modules as easy as

for(mod in modules) {
    if(modules[mod].isLoaded()) {
        modules[mod].init();
    }
}

The eventual goal of using this pattern is to be able to do unit tests which will require that we be able to initialize and cleanup the modules for each test.

Getting rid of massive if/else blocks

The shortcuts module contains all of the keyboard shortcuts for every applicable page on hubski. One of the major changes made to the shortcut functions is instead of using massive if/else blocks for determining which combination of keys was we use objects to map the functions to the key code. We have seperate objects for each applicable page (A single post, the user feed, notifications page, etc) and store keycode/function pairs in each object as such.

var postShortKeys = {
        '65': // 'a'
            function() { $('.longplusminus > a').click(); },
        '82': // 'r'
            function() { $('[name=\"text\"]').focus(); },
        '83': // 's'
            function() { $('.titlelinks > a:contains(\"save\")').click(); }
    };

Once we have all these objects defined we can determine which ones should be available to the user and combine them using the extend function.

var keyMap = {};
function buildKeyMap() {
    $.extend(keyMap,generalShortKeys);
    if(isPost) {$.extend(keyMap,postShortKeys)};
    // ...
}        

This simplifies the function call in the keyup event handler to something like

    keyMap[event.code]();

There are a few drawbacks to using this method as opposed to a if/else or switch/case block, however. For instance, it doesn't look as nice when you are using several different key codes to do the same thing. If you were using a switch statement you could let the those cases just fall through. Also if you want to use key combinations such as Shift + <key> you can't simply rely on the keycode, as Shift + o and o will be detected with the same code.