This blog is dead, long live the new blog!
Shiro Dev Blog
It Lives [12/19/2011]
Download links (kind of important, no?) are now available, and the source code is up on Google Code. I've settled on the MIT License. Development on the language itself has been slow what with the holidays and me starting a new job, but it's in a pretty stable state right now, and the documentation is up to date.
My next projects are going to be reassessing how classes work, specifically the ability the modify a class and have it trickle-down the inheritance chain. I'm also planning on adding some convenience operators like the ternary operator (? :). I love that operator by the way.
In addition, tighter integration with Kayak (right now it's strictly in a testing state) for more HTTP goodness and some long overdue updates to ShiroChan. Know anyone that wants to hack on an IDE? Call me, it's not my favorite kind of work :(.
State of the Site [11/2/2011]
The site is coming up slowly but surely -- plenty of broken links and content placeholders, but most of the content is written and just waiting to be formatted and uploaded. After this weekend all links/downloads/cool-stuffs should be fleshed out, all though there will be more to come.
I don't anticipate visitors this early in the site, since it's not linked anywhere, but just in case I wanted to make sure people knew why links were broken and stuff was missing.
TL;DR [10/29/2011]
If there's anything as satisfying as the creative act of making something new, it's bloviating about the process and end-result afterwards, or in my case, about halfway through. Shiro is more of a hobby to me than anything I did with marketability in mind, but over the last three years of development it's come far enough that I could imagine it being useful to other people, thus this website, this blog, and all the rest. It's still got a long way to go, and even if/when I get there I recognize that pitching a brand-new programming language is almost definitely a dead-end even from the get-go. One wonders why I've put so many hundreds (if not thousands) of man-hours into such a project... I suppose it was more interesting than watching TV at the very least.
This dev blog is pretty much going to contain various exerpts about the development of Shiro, as well as any news, announcements, new stuff, etc. My end-game-goal, which is a tutorial on language parsers similar to an updated "Let's Build a Compiler" (Dr. Jack Crenshaw) may well start life as a series of dev blog posts. This post is going to cover a little of the history behind Shiro. Fair-warning, it will likely run long.
I became obsessed, to some degree, with writing compilers and interpreters at the ripe age of about thirteen. Having conquered QBasic, I was convinced of my own natural genius and just
knew that all it would take is a few weekends of work to make me the next Frank Borland (who is, of course, fictional, a fact I didn't know at the time). This was before the days of amazon.com, so getting a copy of the Dragon Book was out of the question; this left me with the afore-mentioned "Let's Build a Compiler" tutorial, and comp.compilers as a reference. Those 'few weekends' turned into about a year of intermittent effort (thirteen year olds are not, generally, the most consistent workers), just to produce a rather silly system that translated PILOT to QBasic and ran it. If you don't know what PILOT is don't bother to google it, it's terrible.
From good ole Qbasic I went to C, then VB and C++. I wrote probably half a dozen different parsers over the years; even after learning Lexx and Yacc I still preferred to hand-code things, and did so. I re-encountered and re-solved a lot of solved problems that I would later learn about reading textbooks on the discipline. The LCC book and the Dragon Book were both exceptionally useful, after many years of bad habits and self-constructed hacks.
I was 24 when I started Merlin, which would eventually become Shiro. At the time, it resembled Python far more than what it does today. The parser itself started as an experiment in indentation-scoping (me trying to figure out how the hell it worked). I figured it out, but ultimately didn't like the resulting language -- it was too close to Python to really differentiate in my mind. Also, I had no real idea what Merlin was
for. It was neat, it had some cool stuff (specifically the syntax-injection construct), but it had no real purpose. It was back-burnered for a while.
Over the next year and a half, any time I got the "compiler bug", I'd reopen Merlin and work on it a bit. A myriad of raw language features and sematic constructs started working their way into my simple little Python clone. I can trace out my various real-life (ie: bill-paying) learnings and interests and roles as a kind of geological strata in the Shiro project. When I started doing a lot of object-oriented JavaScript, Merlin got in-line object declaration, prototypal inheritance and composition. TDD lead to the built-in assert library and a huge bevy of language-test code. Lambdas in C# translated into a rich "anonymous function" model in Merlin that supported craziness like the parameterization operator and the execution operator. All of these features were barely documented, if at all, and integrated with the core, Merlin syntax in a rather willy-nilly fashion. The functionality was all, and after a while Merlin syntax started to look a bit like APL and a bit like line noise/Perl, except with strange whitespace rules.
I still remember the moment I gave up on Python-style scope levels. I was dealing with injected code (using the ExecOp), which basically just inserts the value of a string at the current point in the source code and continues parsing. I was injecting a multi-line string, and trying to figure out how I should enforce scope levels -- should the string 'start' at a certain indentation level, should every line require indentation? Just the lines after the first? I hated all of the solutions I came up with, and found many of them trashing scope levels and either leaving garbage around in the symbol table or clearing non-garbage out of the symbol table too early.
This is normally sort of an Outside Context Problem for my side-projects. I reach an apex of frustration and then go work on something else. I'm wasn't, after all, getting paid for Merlin; the prospects of anyone ever using it for anything were effectively NULL. It was part hobby and part learning experience.
However, like a bad poker player, I felt I was already pot-committed (Note to aspiring poker players: there is no such thing). All those time-lapse iterations equalled a pretty significant chunk of my time, over almost two years. Tons of refactors, late-night debugging sessions, writing huge test-cases in Merlin to exercise obscure fringe areas of the interpreter, writing and rewriting documentation... Merlin had achieved some sort of critical mass, like Skynet achieving sentience, and it refused to be put out of my mind.
A bunch of IRL crap happened around this time, which basically left Merlin to stew in my subconscious, which is where my best thinking happens. Once things finally cleared up, I'd figured out what I wanted to do with the language, and I set about doing it.
Keeping (almost) all the functionality that Merlin had accrued, I had rejiggered all the sematics and syntax rules into something that felt 'right' to me. Many of the syntactic "options" I added (different keywords to do the same thing) got culled, and everything became kind of... clean. I found myself as productive in Merlin as I was in my desk-job languages, which was a first for me... I'd actually produced something deep enough that it was
useful to me. I could write programs in my own programming language without having to fight through looking up libraries, learning operators and fixing interpreter bugs.
I felt the need to commemorate this moment of productivity, the event horizon between a lot of time and code that was just for fun and a lot of time and code that was something useful. Merlin became Shiro. Since then, the core language has remained largely unchanged -- only the libraries have been built out with the aim of making Shiro capable of Doing Things (tm) that other languages and environments had problems with.
A bit later, I was at work, developing a Javascript front end for a rich call-center/sales application. The back-end and middle tier folks were taking their time providing a service layer, so in order to not get held up, I grabbed a copy of Shiro, implemented a full set of mocked services in about fifteen minutes, and was able to develop the rest of the front-end easily. One of my coworkers (one of the smart ones too!) happened by my desk and asked how I was doing what I was doing, and I showed him Shiro. Two days later he asked for a copy, and used it for a similar purpose. At this point I realized that not only had I created something that was useful to me, but to other people too.
Shiro is still my hobby; I expect it will remain so forevermore, but I firmly believe that even as a 0.9 version, it offers things that a windows developer simply can't get as easily elsewhere. It's node.js without the cygwin and the javascript, and so much more besides. This website represents the beginning of the probably-long road towards making Shiro even better; soliciting feedback, getting new ideas and fresh eyes in the code... the language and libraries are reaching a point where I can't really come up with everything on my own anymore. If it were a novel, I'd be sending it off to my editor. In this case, my editor is the internet at large.
This site may end up being a digital monologue -- someplace where I put content and no one consumes it. I've neither the time nor the temperment to be an evangelist for Shiro. It is my hope that the language, libraries and design philosophy behind it evangelize themselves, and if not, well, people have websites for their cats... I have one for my programming language. It's not
that weird, is it?