Porting Meteor apps to Sandstorm in three minutes
By Kenton Varda - 12 Aug 2014
As you know, Sandstorm apps can be written using any tech stack that runs on Linux. We have apps written in PHP, Python, Node.js, C++, Go, and even Rust.
But if you’re starting from scratch and need to choose, the easiest way to write a Sandstorm app today is using Meteor. Meteor is a modern open source web framework that makes it ridiculously easy to build high-quality, low-latency, real-time web apps quickly. Sandstorm’s own front-end is built on Meteor.
I recently gave a lightning talk at Meteor Devshop where I demonstrated porting a Meteor app to Sandstorm. The app I chose was written by independent developers with no intent to target our platform. As you’ll see, it took me under three minutes to port live on stage:
Note: The video shows the process for older versions of Meteor, in which packages had to be installed with the separate tool Meteorite (mrt). This blog post has since been updated for Meteor 0.9.x, which has its own built-in package repository.
When it comes to writing a Sandstorm app, Meteor works particularly well as a platform because it tends to produce self-contained apps. Meteor’s tools already know how to construct a “bundle” containing all of the app’s dependencies. From there, we just need to add Node and Mongo, and we have ourselves a Sandstorm package. We don’t even need to use our trick to automatically detect dependencies.
To make this even easier, we’ve written a tool called meteor-spk which automates this. We’ve also published a Meteor package kenton:accounts-sandstorm which integrates Meteor’s accounts system with Sandstorm’s unified login. Together, these mean that porting a Meteor app to Sandstorm usually involves three commands:
meteor add kenton:accounts-sandstorm
meteor-spk init
meteor-spk pack myapp.spk
Our intrepid fan Jake Weisz – who previously contributed a port of EtherCalc – has used this tool to port Meteor Blocks, a simple voxel-based model editor written using Meteor. You can find it on the Sandstorm app list now – try the demo if you don’t already have your own server.
We hope to see tools like this developed for other frameworks in the future. meteor-spk is just a simple bash script wrapping Sandstorm’s spk tool and providing a dependency bundle to merge into the package. If you’d like to contribute a similar script for your own favorite framework, let us know!
Disclosure: Jade Wang, a member of the Sandstorm project, also works for Meteor, but Sandstorm and Meteor are not affiliated.
Roundcube; and avoiding walled gardens
By Kenton Varda - 11 Aug 2014
Today we’re releasing our port of Roundcube, another open source e-mail web app. You can go install it from the app list now, either on your own Sandstorm server or on our demo.
To this you might ask: What happened to Mailpile?
Nothing happened. We support both! We like Mailpile, but we also want to give you choices. With Sandstorm, you can mix and match apps from competing developers as you see fit, while still retaining the unified experience of Sandstorm.
Let’s end walled gardens
Usually, when you sign up for a company’s suite of “cloud apps”, you have two choices:
- Restrict yourself only to the apps that company provides and no others. This way you get a nicely integrated experience, but if one or more apps in the suite aren’t to your liking, you are stuck with them.
- Also use apps from other developers. But you probably have to log into those apps separately, and they won’t integrate well with the first company’s suite. You end up with a fragmented experience in which you must remember multiple passwords and re-enter the same data into each service.
We call this the “walled garden” problem. Big SaaS (Software-as-a-Service) providers tend to build walled gardens of apps that only integrate with other apps from the same developer.
It would be easy and convenient for us to say that these companies are Evil and just want to lock you in, but that’s not true. The reality is that integration across security realms is hard. When each company has their own notion of users that can’t easily be mapped to each other, and understandably no company is willing to trust other companies with their security, how can you integrate apps containing private data? The answer in practice is OAuth, but OAuth permissions requests are clunky and disruptive to the UX. Some users are scared off by it – while others aren’t scared enough and just click “OK”.
But developers within a company are perfectly willing to trust other developers in the same company to get security right, and so the entire OAuth dance can be skipped in these cases. This makes first-party integrations easy. Now imagine what would happen if a single company’s own apps had to OAuth to each other before integration. Imagine, for example, if Google Docs had to ask you for permission to fetch your contact list from GMail before enabling auto-complete of contact names in the share dialog. Probably, Google Docs wouldn’t bother, and there would be no autocomplete. Or imagine if the Facebook profile page had to ask for permision to connect to your Facebook photos before you could set your profile photo. This would be a huge pain! And even then, it wouldn’t necessarily end the walled garden: with OAuth, an app says “I need permission to talk to GMail,” not “I need permission to talk to an e-mail app.” There’s still no opportunity for you to substitute a third-party app in its place.
With Sandstorm, we’re solving a lot of these problems:
- All your apps run under the same roof, whether they are from the same developer or competing developers. You can use a diverse set of apps without having to remember a separate password for each one.
- In place of OAuth, we’re building the Powerbox. This UI integrates better into the app’s UX by simultaneously acting as a picker and permissions granter. Apps usually need to present a picker anyway, so this is less disruptive. For example, if an app wants you to choose some contacts to which to send invitations, instead of first asking for permission to all your contacts just to display a picker UI back to you, the app will ask Sandstorm to display a Powerbox picker for contacts. Only the contacts you choose will be passed to the app. So, the UX is exactly the same, except without the separate permissions check, and yet it is more secure. The same idea would apply to choosing photos, or choosing anything else.
- When using the Powerbox UI, the requesting app does not name an exact receiving app it wants to access, but instead specifies a protocol. So instead of saying “I need access to Mailpile,” the app would say “I need access to an e-mail app,” and you could potentially choose Mailpile, Roundcube, or any other e-mail app. Apps do have to implement compatible protocols, which can be difficult, but this approach at least leaves open the possibility of a newcomer app emulating the protocols already used by other apps in the space, thereby breaking down the network effect that would otherwise prevent a new app from gaining a foothold.
We hope these design choices mean you get a unified experience no matter which apps you choose to use, and new players can break into the market more quickly, leading to faster innovation.
EtherCalc: Online spreadsheets
By Kenton Varda - 05 Aug 2014
Today’s release is a pleasant surprise even to us. Last week, a Sandstorm fan, Jake Weisz, e-mailed me out of the blue to tell me he had ported EtherCalc, an online collaborative spreadsheet editor, to Sandstorm.
I hadn’t heard of EtherCalc before, but Jake included a link to a spreadsheet running on our own alpha server, so I was able to try it out immediately, and I was impressed. Sure enough, it’s a spreadsheet editor, it does the things you expect spreadsheets to do, it supports real-time collaboration, and it’s Free Software.
Jake professes to have had no idea what he was doing. “I have never even seen Node.js before, I don’t even know JavaScript, and I have only a passing familiarity with using Linux.” Yet, by just following the porting guide, he managed to port an app. Jake attributes this to EtherCalc’s simple and elegant design: “The way it was already designed, it was incredibly easy to port to Sandstorm. I think I changed three or four lines of code tops to get it working.”
Speaking of EtherCalc’s design, developer Audrey Tang has written a long and fascinating history of EtherCalc (and its previous iterations, WikiCalc and SocialCalc) for the book series The Architecture of Open Source Applications. These chapters have also been reproduced on EtherCalc’s web site. Check it out!
Meanwhile, Jake’s port is on the Sandstorm app list now. As always, you can install it on your Sansdtorm server, or try the demo.
MediaGoblin: Photo sharing and more
By Kenton Varda - 01 Aug 2014
Today we’re releasing a port of yet another crowd-funded open source web app, MediaGoblin. You can think of MediaGoblin as a Free Software alternative to Flickr, SoundCloud, YouTube, and others.
The app is on the app list now – try it on the demo server.
MediaGoblin is our first app to take full advantage of Sandstorm’s new user identification features. Whenever a request comes in from a user, Sandstorm’s front-end proxy adds a few headers related to authentication and authorization:
X-Sandstorm-Username: Specifies the user’s display name, e.g. “Kenton Varda”. This name is not unique nor stable; it is meant only for displaying to other users.X-Sandstorm-User-Id: A hex string identifying the user. This ID is both unique and stable, so it can be used to identify when the same user returns multiple times.X-Sandstorm-Permissions: A list of permissions that have been shared with this user via the Sandstorm sharing interface. An app can define what permissions are available to share in its package config. However, note that we have not yet implemented the full sharing UI, so for now the creator of the app instance is granted all permissions and everyone else is granted none. It’s up to the app to actually implement this permissions, so for now you can simply use it to distinguish between the owner and non-owners.
Our MediaGoblin port uses these headers to automatically populate MediaGoblin’s internal user list. This way you get MediaGoblin’s commenting and other multi-user features without the need to separately log into it.
Note that there are a few things missing from this port, which we plan to fix in the future:
- Currently, under the hood, apps are exposed only at ephemeral hostnames; the hostname may change every time you use the app. This is great for security but means that apps cannot expose APIs, since API clients want a persistent URL to talk to. Atom feeds are a kind of API. So, for now, we have disabled Atom feeds in our port. This is unfortunate, as it changes MediaGoblin from “federated” to “isolated”. That said, enabling APIs is high on our wish list of features, so expect this to return in the future.
- Related to the above, it’s currently not possible to deep-link into an app,
since it runs in an iframe. So, sending someone an individual photo with
MediaGoblin is hard. We can fix this by automatically passing through any
additional path added to the outer app URL, and creating a
postMessage()-based API for apps to tell the Sandstorm shell what the current path should be. - Due to a limitation in our proxy implementation, we currently do not support very large file uploads. As a result, we had to disable video uploads for now. We plan to fix this soon.
- Currently, if you wish to share your MediaGoblin content, you must share the app instance by copy/pasting the URL. When another user visits that URL, your MediaGoblin server starts up to handle the request. This differs from Ghost and Hacker CMS which are able to publish web content in such a way that the server does not need to start up for every visitor, only for the editor. Unfortunately, this means that a widely-shared MediaGoblin instance may be expensive as it may have to run frequently; Sandstorm normally tries to cut costs by only running servers a small fraction of the time. We’d like to change the app to take advantage of Sandstorm’s web publishing features like Ghost and Hacker CMS do, so that the server only needs to run when a visitor posts a comment, uploads media, or otherwise makes a change. However, this may require some invasive changes to MediaGoblin, so we’ll need to discuss them with MediaGoblin’s developers.
- MediaGoblin implements a number of permissions which can be granted to users, such as permissions to comment, upload media, etc. We eventually want permissions to be set using Sandstorm’s unified sharing model, but as described above it is not yet complete. So, for the time being, you have to set users’ permissions inside the MediaGoblin user management interface.
All that said, despite some missing features, Sandstorm is by far the quickest way to get MediaGoblin up and running. We hope our port allows this great piece of Free Software to reach a wider audience than it could before.
And if you’d like to see the missing features fixed in the future, be sure to fund our Indiegogo campaign. :)
ShareLaTeX: Collaborative scientific typesetting
By David Renshaw - 25 Jul 2014
Today we are releasing our port of ShareLaTeX, a real-time collaborative editor for typesetting scientific documents.
You can install it on your Sandstorm server now, or try it on our demo server.
LaTeX has long been a standard tool in the world of academic publishing, and we think that ShareLaTeX does a great job at making it more accessible and useful.
Using ShareLaTeX in Sandstorm to typeset Rob Simmons's PhD thesis.
We also think that, as an open source web app, ShareLaTeX makes a fascinating case study.
On the one hand, it’s hard to imagine any of the giant internet corporations ever offering anything like it, as its potential user base is probably not large enough to merit their attention. On the other hand, within its niche ShareLaTeX addresses a very real need – a need that’s certainly large enough to merit the attention of small teams of indie developers.
ShareLaTeX illustrates some of the barriers faced by such developers. Its core functionality consists of an in-browser editor and a server-side compiler, but to be a viable product it has had to do a fair bit more than that. ShareLaTeX also implements a secure login system, an interface for managing multiple projects, and a sharing model. Moreover, there is the question of hosting.
The striking thing about our port of ShareLaTeX is that it simply bypasses this non-core functionality. Sandstorm already handles login, already has a file selection and organization interface for managing multiple projects, and already implements sharing. The main changes we made to the ShareLaTeX code – and we did not in fact change very much at all – were to remove these features and delegate their responsibilities to Sandstorm instead.
Sandstorm, of course, also resolves the hosting question, making it easy for users to deploy ShareLaTeX to servers they control.
We take all of this as evidence that Sandstorm can eliminate many of the less-interesting and potentially costly parts of developing web apps. We hope that Sandstorm will enable developers to create many more apps like ShareLaTeX, catering to audiences of all sizes.