Intended Effect

A kind of cliché-but-true complaint common among programmers in the late 2010’s was that there were too many new JavaScript things every week. Frameworks, libraries, and bits and pieces of tooling relentlessly arrived, exploded in popularity, and then receded like waves on a beach. My sense is that this pace has slowed over the last few years, so I wonder: was this, too, a ZIRP phonemon, and thus a victim of rising interest rates along with other bubbly curiosities like NFTs, direct-to-consumer startups, food delivery discounts, and SPACs?

I’ll admit: I can’t say I’m basing this on anything but anecdotes and vibes. But the vibes were strong, and they were bubbly. People complained about the proliferation of JS widgets not because of the raw number of projects, but because there was a sense that you had to learn and adopt the new hotness (all of which still seem to be going concerns!) in order to be a productive, modern, hirable developer. I think such FOMO is a good indicator of a bubble.

And I think other ZIRP-y factors helped sustain that sense that everyone else was participating in the permanent JavaScript revolution. What could interest rates possibly have to do with JavaScript? Let’s say you have $100 to invest in either a startup that will return a modest 2x your investment in 10 years, or a bank account. If interest rates are 7%, your bank account will also have (very nearly) doubled, and of course it’s much less risky than a startup! If interest rates are 2%, your $100 bank account will only end up about $22 richer after a decade, so you might as well take a chance on Magic Leap. It also means that a big company sitting on piles of cash might as well borrow more, even if they don’t particularly know what to do with it. So all of that is to say: low interest rates meant there was cheap money for startups and tech giants alike.

Startups, for their part, are exactly the kind of fresh start where it really does make sense to try something new and exciting for your interactive demo MVP. If you’re a technical founder with enough of a not-invented-here bias—a bias I’ll admit to being guilty of—then you might well survey the JavaScript landscape and decide that what it lacks is your particular take on solving some problem or other. Here’s that XKCD link you knew was coming.

As for the acronymous tech giants, there’s been recent public accusations that larger firms spent much of the 2010s over-hiring to keep talent both stockpiled for future needs and also safely away from competitors (searches aren’t finding me much beyond VC crank complaints, but I’d also gotten this sense pre-pandemic). Everyone loves a good “paid to do nothing” story, but I think most programmers do want to actually contribute at work. A well-intentioned, ambitious, and opinionated programmer might well pick up some slack by refactoring or reimplementing some existing code into a brand new library to share with world. If nothing else, a popular library is a great artifact to present at your performance review.

I started thinking about this in part because it frustrates me how much programming “content” (for lack of a better word) seems to be geared towards either soldiers in corporate programmer armies or individuals who are only just starting their company or their career: processes so hundreds can collaborate, how-to’s fit only for a blank page, or advice for your biannual job change. There’s probably good demographic reasons for this; large firms with many programmers have many programmers reading Hacker News. But something still feels off about this balance of perspectives, and you know what? Maybe it’s all been a bubble, my opinions are correct, and it’s the children who are wrong.

As interest rates proved to be less transient and some software-adjacent trends were thus shown to have been bubbles, a popular complaint was to compare something like crypto currency to the US railroad boom (and bust) of the 19th century—the punchline being that at least the defunct railroads left behind a quarter-million miles of perfectly usable tracks.

So what did we get for the 2010’s JavaScript bubble? Hardware (especially phones), browsers, ECMAScript, HTML, and CSS all gained a tremendous amount of performance, functionality, and complexity over the decade, and for the most part a programmer can now take advantage of all that comparatively easily. The Hobbesian competition of libraries left us with the strongest, and it’s now presumably less complicated trying to decide what to use to format dates.

Then again, abandonware is now blindly pulled into critical software via onion-layered dependency chains, and big companies have taken frontend development from being something approachable by an eager hobbyist armed with a text editor, and remade it in their own image as something to be practiced by continuously-churning replaceable-parts engineers aided by a rigid exoskeleton of frameworks and compilers and processes.

So yes, “in conclusion, web development is a land of contrasts“. To end on a more interesting note, there are two projects that I think could hint at what the next decade will bring. Microsoft’s Blazor is directly marketed as a way to avoid JavaScript altogether by writing frontend code in C# and just compiling it to WebAssembly (yes, that’s not always true because as usual Microsoft gave the same name to four different things). Meanwhile htmx tries to avoid this whole JavaScript mess by adding some special attributes to HTML to enable just enough app-ness on web pages. I would never bet against JavaScript, but it’s noteworthy that both are clearly geared towards people frustrated with contemporary JS development, and it’ll be interesting to see how much of a following they or similar attempts find over the next few years.

Bungie released Marathon in 1994, the same year Doom II came out. Marathon was also a first-person “run around picking up guns to kill aliens with” game, but had a comparatively robust plot with multiple conflicting AI characters, multiple settings, a relatively fleshed-out picture of the bad guys, effectively moody atmospheric music, and some additional universe flavor text here and there.

I never really played much of Doom II, so I’m not sure if Marathon’s gameplay was also a step above. But it was beloved by Macintosh users because it was a rich, advanced FPS with fantastic network play levels, which was exclusive to a platform that was usually left behind by game publishers. Apple wasn’t at the top of their game in that beige Performa, pre- Steve Jobs return era, but Marathon was a celebrated bright spot.

Not long before Microsoft bought them in 2000, Bungie GPL’d the Marathon 2 game engine, which is what the current Aleph One project originated from. Bungie later released the whole trilogy of Marathon games for free, so the scenario/content files can now be distributed with the engine (though I don’t know the license details). They’re still 25-30 year-old games, so (for example) there aren’t many save points, but the open source project has been very successful at making them pleasant to play decades later.

Since the pandemic my household has used a Raspberry Pi 4B running RetroPie as a retro game emulation system, so I was excited to see that RetroPie has a built-in option to install the Aleph One Marathon games under “ports”. These installs do run, and are playable with a keyboard, but they never worked with our 8BitDo bluetooth controllers. Until today, that is! I finally figured out how to get them running with our controllers, so I thought I should write down what I did.

Aleph One uses SDL, a software library used by many games (among other things) to work with hardware components like controllers, keyboards, displays, etc. If your Linux install of Aleph One is also failing to work with controllers, you might find log entries like this:

No mapping found for controller "8Bitdo SN30 Pro" (05000000c82d00000161000000010000) (joystick_sdl.cpp:66)

On our RetroPie this was in the file /opt/retropie/configs/ports/alephone/Aleph One Log.txt. The “joystick_sdl.cpp” in the log entry is the clue that this is an SDL issue, and that led me to a helpful guide on Reddit for making controllers work with SDL games on Linux.

That included a link to a script to download the SDL game controllers database, grab the Linux section, save it to a dot file in the home directory called .controller_config, and then add a line to your .bashrc (or similar) to add those controller mappings as an environment variable called “SDL_GAMECONTROLLERCONFIG”. I ran that script on our RetroPie box.

To briefly explain what this is for: one of SDL’s jobs is to take input from many different kinds of controllers and present them to games as, simply, “D-Pad Up”, “D-Pad Down”, “A Button”, “B Button”, etc. That game controllers database has the actual mappings for that—it basically says, “controller XYZ identifies itself with the number 1234, and on that controller the right trigger is button code b4”, etc, etc, for all of the buttons on all of those controllers. By adding those mappings as an “environment variable” (a value available to every program that you run), you’re telling the SDL code in the game you’re trying to play how to work with the buttons of your controller.

Adding that entire controllers database as an environment variable presumably must work in many cases, but it’s also a bit blunt. The Linux section of that controller database is over 500 lines long! In fact, it’s so long that adding all of that to your environment variables will make ordinary bedrock tools like rm or mv break with “argument list too long” errors. If that happened to you and now you’re panicking because you can’t even delete your .controller_config file, you just need to un-set that environment variable by running unset SDL_GAMECONTROLLERCONFIG in a terminal.

Anxious to see if I could get things working, tonight I just took the .controller_config file generated by the script above and deleted everything except for the couple dozen 8BitDo lines (since that’s the brand of all of our assortment of controllers). You should similarly trim your controller file down—just remember to leave the very first line and the very last line of the file untouched, or to otherwise make sure that the file still begins with export SDL_GAMECONTROLLERCONFIG=" and ends with a closing quote mark.

You could improve upon my blunt edit by looking in .controller_config for the long numeric string(s) in your Aleph One Log.txt “joystick_sdl.cpp” error, and just including the matching line(s)—in my error example above the number to look for would be “05000000c82d00000161000000010000”.

So now you should have a .controller_config file with a small subset of the giant game controller mapping database, and a line like this in your .bashrc (or similar) file:

. "$HOME/.controller_config"

To make your controllers work with Marathon when launched from EmulationStation (the frontend used by RetroPie), however, you’ll need to copy that line to the launch scripts for the Marathon ports. Those are in ~/RetroPie/roms/ports/, with names like Aleph One Engine - Marathon.sh. In that file, you need to add that line from your .bashrc as the second-to-last line (just before the long final line that runs the game). Here it is again:

. "$HOME/.controller_config"

That’s a period for the first character—you’ll need to copy that line exactly, including that period and the quotes.

But once you do that, you should be able to run the Marathon ports from RetroPie and have your wireless controllers work!

Another time I’ll have to improve upon this. PortMaster (a system for installing ports on handheld emulator consoles) does a great job of handling controller mapping for games that use SDL, so their scripts are probably something I should look at. Ideally, the Marathon launch scripts would look at what controllers are attached (possibly with evtest), look up only those lines in the game controller database, and then set those mappings to the SDL_GAMECONTROLLERCONFIG environment variable before launching the game. Presumably that controllers database already exists somewhere in a normal RetroPie distribution? Who knows—for now I’ve got a score to settle with the Pfhor still occupying the colony ship Marathon.

It might well be my naive, beginner’s implementation of it, but it sure seems like Logstash is pretty easy to confuse. Elasticsearch temporarily unavailable? Hang. Some weird formatting sneaks in? Hang. Some as-yet-undiscovered circumstance that causes java to eat all your memory? Hang. And each time it hangs, Logstash becomes unresponsive, never recovering. So for the moment, I’ve given up, and I’m resorting to a dumb, sledgehammer-like solution:

#! /bin/sh
FILELENGTH=`stat --format=%s /var/log/logstash/logstash.err`

if [ $FILELENGTH != "0" ]
then
        killall -9 -u logstash
        service logstash restart
        logger Logstash Killed and Restarted Due To Non-Zero Error Log
fi

Again, I’ve only had about a week of actually sending a lot of input to Logstash, but so far, there’s always something in the error log when it hangs, and there’s rarely anything in there when things are going well. So this barely-a-script checks the size of the error log, and if it’s non-zero, it SIGKILLs every process belonging to the logstash user, and then tries to start Logstash. The restart part assumes you have a Logstash script in init.d, which I certainly hope will be included as part of a proper .deb distribution of Logstash someday soon. I run it every five minutes with cron:

*/5 * * * * /etc/logstash/kick_logstash.sh

At Knowledge Architecture, we may not operate at the scale of Facebook or Flickr, but Synthesis sees enough image uploads that we’ve run into some pretty strange image rotation problems now and then. I’ve spend a lot of time with the issue, and while I’m not naive enough to declare victory over something as devilish as EXIF rotation gremlins, we’ve got an update rolling out this week that will at least avoid the species we’ve seen so far.

First, a little background, and we’ll imagine a portrait-orientation photo of the Sutro Tower as our example image. Digital images are stored in a given rotation, which generally corresponds to the default rotation of the image sensor—so our Sutro Tower photo is probably sideways on disk. If we open the image in any half-decent viewer, though, the tower will be right-side up because most digital cameras annotate image files using the EXIF Orientation tag. That’s used by most image-viewing things to recognize situations where image may be stored on disk like this, but when displayed it should be (say) flipped, and rotated 90° to look like that.

It’s gotten better, but support for that Orientation value isn’t perfect. So when someone uploads an image to Synthesis, we look for the Orientation EXIF field, rotate the image data itself so that it’s stored the way it should be displayed, and then we remove the Orientation field, because its instruction to rotate the image on display is no longer needed.

The problem we’ve run into is that some image editors apparently don’t update EXIF data properly; they rotate the image data itself (so the Sutro Tower is right-side up), but leave the Orientation flag in place. So our uploading process would dutifullly rotate the image one more time, and the tower would end up pointing sideways. It was a frustrating situation.

Luckily, those EXIF-unaware image editors seem to leave all the other EXIF data unchanged, too. So what our solution does is compare the so-called “EXIF Width” and “EXIF Height” field values to the width and height of the image data on disk. If it looks like the image has already been rotated, we simply remove the Orientation field.

So here’s the important part of the code. In short, we:

• calculate the ratio of actual image width to height, and the ratio of EXIF-reported height to width,
• round them both to one decimal place (to account for resized images subtly changing that ratio),
• and check to see if one equals the other. If so, the image has been rotated already.

This is C# but it’s mostly math so it should be easy enough to port to whatever language you’re using. Not shown is fetching the orientation, exifWidth, and exifHeight values; they’re EXIF fields 0x0112, 0xa002 and 0xa003, respectively. I found this list of EXIF tags very helpful during this whole process.

// Useless data check (this check doesn't work on squares)
if (exifWidth > 0 && exifHeight > 0 && exifWidth != exifHeight)
{
    // To see if the bits were rotated, we calculate a height/width ratio, because 
    // we might be dealing with a resized image. We round, becuase resizing might 
    // slightly change this ratio
	
    decimal actualRatio = Math.Round(
		((decimal)img.Height / (decimal)img.Width)
		, 1); // height / width
		
    decimal exifRatio = Math.Round(
		((decimal)exifWidth / (decimal)exifHeight)
		, 1);  // width / height

    if ( orientation >= 5 && actualRatio == exifRatio )
    {
        // SO, if the orientation is rotate90 or rotate270 (with or without flip),
        // and the image seems to have already been rotated 90 or 270 (the width 
	// is now the height), we're going to assume that some bad software rotated 
	// the pixels and never removed the orientation flag. In such a case, we'll
	// trust the lousy software's rotation, and remove the flag.

        orientationPropItem.Value = new byte[1] { 1 };
        img.SetPropertyItem(orientationPropItem);

        // We return true because in our implementation, we just need to know if the 
	// image file we're storing has changed at all, whether that means rotation 
	// or just removing an EXIF value.
        return true;
    }
}