Sandstorm News

Sandstorm is a self-hostable web productivity suite. Host your own

Recommended Posts

Further Reading

Let's Encrypt support for Sandstorm and Sandcats

By Kenton Varda - 13 Jun 2020

Sandstorm now has built-in support for fetching certificates from Let’s Encrypt. This applies both to Sandcats and to custom domains.

Let’s Encrypt with Sandcats

Backstory

Sandcats.io is Sandstorm’s free dynamic DNS and TLS (aka SSL) certificate service. Since 2015, Sandcats has been making it easier to set up self-hosted Sandstorm servers, by letting anyone claim a subdomain of sandcats.io on which to host their server, automatically configuring DNS and HTTPS.

Sandstorm requires a wildcard host so that it can create sandboxes by assigning random hostnames. So, the certificates issued by Sandcats.io have always been wildcard certs. Back in 2015, that was a challenge: at the time, Let’s Encrypt did not yet offer wildcard certs, so the only way to get one was to pay for it. Unfortunately, many CAs considered wildcards to be an “enterprise feature” and charged excessive amounts of money for them, often hundreds or even thousands of dollars.

So how were we able to offer wildcard certificates for free? Well, we negotiated a deal with GlobalSign. Recognizing that subdomains of Sandcats.io were unlikely to be enterprise customers, and recognizing a growth opportunity if Sandstorm took off, GlobalSign offered us a deal that worked. Sandstorm was a funded startup at the time, and we were happy to pay a few bucks per certificate-year and call it “customer acquisition cost”. We paid for thousands of certificate-years upfront, anticipating growth.

But, the growth we hoped for didn’t happen, and we only ever had about 500 self-hosted servers using Sandcats. In 2017, Sandstorm failed as a startup and reverted to being just an open source project. Having never seen the growth we were hoping for, we hadn’t even come close to using up the first block of certificates we had paid for from GlobalSign. Since we had already paid, we left the system running, happily issuing certificates.

Fast forward to 2020. Finally, our contract is running out. In fact, it did run out, in early March – embarrassingly, I had miscounted how much time we had left. However, GlobalSign was nice enough to give us a small extension, giving us time to migrate our users without disruption. And it turns out that, these days, Let’s Encrypt supports wildcards. So, moving to them is the obvious choice.

What’s New

Starting a few days ago, all Sandstorm servers using Sandcats.io are in the process of switching to Let’s Encrypt for future certificates. The process is designed to happen slowly so that we can address any problems that arise, but so far everything has been smooth. All servers should be transitioned by the end of the month.

If you’d like your server to start using Let’s Encrypt immediately, visit the TLS certificates admin page at /admin/certificates on your server, click the button to create an ACME account (you will be prompted to agree to Let’s Encrypt’s Terms of Service), and then click “Fetch Certificate Now”.

Note that at present, we have not yet updated the install flow, so newly-installed Sandstorm servers will still use a GlobalSign certificate initially and then will switch to Let’s Encrypt later. We’ll be updating the installer soon, and then we will decommission the GlobalSign flow.

Let’s Encrypt with non-Sandcats domains

To implement Let’s Encrypt support for Sandcats.io, it made sense for your self-hosted Sandstorm server to directly talk to Let’s Encrypt via the ACME protocol. This differs from the old GlobalSign flow, in which Sandstorm would talk to a central Sandcats.io server, which then talked to the GlobalSign API on your behalf. This was necessary for GlobalSign since Sandstorm the company was paying for the certificates, so our credentials were needed when talking to the API. But for Let’s Encrypt, the Sandcats.io central server only needs to set some DNS records to pass an ACME DNS-01 challenge; everything else happens on your own server.

Given this implementation, it was straightforward to support domains other than sandcats.io. I chose to use the ACME.js library to implement the ACME protocol, and it turns out this library already had a suite of plugins to support a variety of popular DNS providers. If your domain uses any of the DNS providers supported by ACME.js, then Sandstorm can now automatically obtain TLS certificates for your domain from Let’s Encrypt.

However, at present, initial setup is difficult, because of the chicken-and-egg problem: How do you access the admin UI to configure certificates, without a certificate? And than brings me to…

Help make it better!

Currently, there are two major problems with setting up Sandstorm TLS on your own domain:

  1. The UI is very bad. I am a terrible designer, and I didn’t have much time to work on it. Especially bad is the fact that to configure any DNS plugin, currently, you need to enter a JSON blob into a textarea, where the JSON blob’s format is different for every plugin and documented in their respective READMEs as a JavaScript method parameter (not even JSON)… Only experienced programmers could possibly understand what to do. We should make the UI better. See issue #3300.
  2. There is a chicken-and-egg problem at install time, as the current UI to configure TLS is accessed over HTTP – which implies that you already need a TLS certificate for it to be secure. We need a CLI configuration option as an alternative. See issue #3367.

If you’d like to help implement either of these, click on the issue links above and comment!

Announcing the release of vagrant-spk 1.0

By Jacob Weisz - 22 Feb 2020

Hello! I’m Jacob Weisz, a member of the Sandstorm community, a long-time contributor, and the new maintainer of the vagrant-spk tool. I’m thrilled to announce the 1.0 release of vagrant-spk.

What’s vagrant-spk?

vagrant-spk is the premier tool for packaging Sandstorm apps. Unlike the spk tool built into Sandstorm, vagrant-spk creates a virtual environment within which to build your app. This provides a reasonable measure of reproducibility and maintainability, along with default templates (or “stacks”) of common configurations apps likely need to run.

What’s new?

Why go to 1.0 now?

Historically, vagrant-spk was frequently released alongside Sandstorm releases and was regularly dependent on those Sandstorm releases for functionality. At the time, the convention was to release vagrant-spk with versioning complimentary to Sandstorm releases.

However, times have changed. vagrant-spk improvements are rarely strongly correlated with Sandstorm releases, and of course, both Sandstorm and vagrant-spk releases are less frequent in nature.

vagrant-spk is also now a mature tool, having been used to package a large portion of Sandstorm apps. The natural progression at this time is to move from 0.236 to 1.0. This release is fairly small because the priority for 1.0 is to deliver a polished, stable release that we can then iterate upon.

The future of vagrant-spk

We have a list of improvements and features we are beginning to think about for vagrant-spk’s future. As part of renewed community efforts to revive the Sandstorm project, we are planning a regular cadence of releases for vagrant-spk. We have already begun designating goals for vagrant-spk 1.1, which we’d like to deliver later this year.

Reviving Sandstorm

By Ian Denhardt - 03 Feb 2020

Hi! I’m Ian Denhardt, a long-time Sandstorm user and community member. I have some exciting news to share. Let’s start by talking about some recent history.

It’s no secret that development on Sandstorm has been pretty sparse since Sandstorm-the-business shut down in 2017. The tone of that announcement was optimistic; while Sandstorm had failed as a business, as an open source project it had attracted an vibrant community, and there was hope that the project could continue to be successful outside of a for-profit setting.

Things didn’t go as smoothly as we’d hoped, however. While Sandstorm had had a healthy community of folks using it and building apps, there had been fewer contributions to Sandstorm itself from outside the company, and no one but Sandstorm employees had been doing major core development on a regular basis. Additionally, Kenton has had less time than he’d hoped to work on Sandstorm.

The project never quite died. It has continued to receive security and maintenance updates. Internationalization support was added, and several folks have stepped up to translate the UI into other languages. A few other features landed, and a few apps continued to see updates as well. But it would be more than fair to say that the project had stagnated, and while I and many others were still using it, it was clear that development wasn’t going anywhere fast.

That looks to be changing.

About two months ago, Lyre Calliope sent an email to the sandstorm-dev mailing list, starting a discussion on how to get the project moving again. Since then, I and others have been contributing code and documentation, and have had weekly “office hours” meetings. In spite of a full time job, moving across the country, and a new baby to take care of, Kenton has still managed find a bit of time to review and merge pull requests, and help plan. Here are some highlights of what’s been happening in terms of development:

As a community we’re once again very hopeful, and I personally am committed to spending what time I can making Sandstorm’s vision a reality. If you want to help, join us in the #sandstorm IRC channel on libera.chat, and sign up for the sandstorm-dev mailing list.

Sandstorm Oasis is Shutting Down

By Kenton Varda - 15 Sep 2019

On December 31st, 2019, Sandstorm’s paid hosting service, Sandstorm Oasis, will begin winding down.

It’s time to go Self-Hosted

If you’re an Oasis user, fear not! You can keep using Sandstorm on your own server, and you can easily transfer all your Oasis data to it.

Indeed, today, there’s almost no reason to prefer Oasis over a self-hosted Sandstorm server. Consider:

In order to make it extra-easy to transfer your data to a new server, I have added a new “Mass transfers” feature to Sandstorm. Find it by clicking the button at the top of your Grains list:

Screenshot showing the mass transfer button, located at the top of the Grains list between the "Restore backup..." button and the "View trash" button.

Then follow the on-screen directions to specify a destination server, review the list of grains to transfer, and then execute the transfer.

To recap:

  1. Set up your own self-hosted Sandstorm server now »
  2. Transfer your grains from Oasis »

Why shut down Oasis?

The story of the last few years

Sandstorm, as a company, mostly shut down two years ago. The company had run out of investor money while having achieved essentially no revenue, and no hard evidence that we’d ever achieve any. While Sandstorm was popular on Hacker News, that popularity never really converted into paying users. Meanwhile, in the market where we imagined we’d find real profits – enterprise software – we’d made no real progress whatsoever. In this state, we were unable to attract new investors, and we were unable to find a company to acquire the business.

The Sandstorm team was forced to look for new jobs. Most of us were hired by Cloudflare, though some chose to go elsewhere. Personally, I chose Cloudflare because I had always liked the technical culture I saw in their blog posts, and because I was interested in the project they wanted me to work on.

Originally, my plan had been to keep developing Sandstorm as an open source project. I felt – and still feel – that if only some rough edges could be smoothed out and some key missing features filled in, Sandstorm could really be a plausible replacement for the set of web services people use every day. I set a goal of getting myself off of Google services by replacing the key bits with Sandstorm apps – especially email. I thought that if I could really get that working, maybe we’d be in a position to relaunch the company.

I made some progress. On nights and weekends, I managed to clean up one of the hairiest rough edges of Sandstorm, fixing the identity system. I also rewrote the basics of how Sandstorm handles HTTP traffic, making it much faster and cleaner and removing JavaScript from a part of the system where it had no business being.

For a while, Oasis was costing far more to operate than it was taking in in revenue, with me making up the difference out-of-pocket. But, between the HTTP rewrite (which saved several machines), and discontinuing the Oasis free plan, I was able to bring things to the point where Oasis is mildly profitable, earning a few hundred dollars a month.

But, meanwhile, at my new job at Cloudflare, I am the lead engineer / architect of a project called Cloudflare Workers, a “serverless” platform that simultaneously deploys your code to 193 (and growing) locations around the world. Starting from scratch when I joined, I built a first prototype in a few months, had a public demo and beta customers shortly thereafter, and launched it to the world exactly (by coincidence!) a year after joining. Today, Cloudflare Workers handles something like a million times the traffic Sandstorm ever did. Meanwhile, the team has grown from just me to a literal bus-load of people. And we’re really just getting started.

As much as I love Sandstorm, it’s hard to come home from my successful day job to work on an unsuccessful side project. And so, I have been spending less and less time on Sandstorm. I still push updates every month to keep the dependencies fresh, but hadn’t worked on any new features in about a year and a half before adding mass transfers recently.

Meanwhile, without leadership, the community has mostly disbanded. The only app that gets regular updates anymore is Wekan, thanks to its maintainer Lauri “xet7” Ojansivu. Jake Weisz heroically continues to carry the Sandstorm flag, reviewing app submissions (mostly from Lauri), replying to questions and bug reports, and advocating Sandstorm around the internet. A couple others lurk on the mailing list and IRC. Most people have moved on.

Why not leave Oasis running?

Oasis is mildly profitable: it brings in about $1800 in revenue each month, while costing around $1400 each month between infrastructure, services, fees, and business upkeep (e.g. tax prep). Almost 200 people are paying for it and it appears most of them are in fact using it. As long as it isn’t losing money, why not let it be?

First, the obvious reason: It still takes time to operate. Once a month I have to spend my Saturday afternoon testing and pushing an update. Several times, changes in dependencies have broken things, requiring debugging time. In fact, Oasis’s cluster management and storage back-end (known as “Blackrock”) is still running a build from October 2018! For reasons I’ve been unable to determine, newer builds after that point start crashing under moderate load. I’m unable to reproduce such load in a test environment, so the only way to test potential fixes is to push out a full release, watch it fail, and roll it back. After several tries, I’ve mostly given up. Luckily, this component of Oasis has had no major changes and does not have any directly-exposed attack surface, so pinning the old version is mostly fine… but it’s a fragile position to be in.

On a related note, I am on call 24/7 for Oasis. It rarely breaks, but when it does, I have trouble fixing it in a timely fashion. For one example, in January, an unexplained Google Cloud hiccup forced me to transfer Oasis to another zone, which it wasn’t designed for (whoops, yeah, it’s not multi-homed, we never got that far). It was down for hours. Luckily it was a weekend and I was at home, or it could have been days. In another incident, I discovered that GMail had been routing all my monitoring alerts (and e-mail to support@, security@, contact@, etc.) directly to spam for months.

But, more important than the time burden on me is that I no longer feel good about charging money for this product. Almost all the app packages are from 2015-2016; many of those apps have had significant updates in their standalone versions since then which are missing on Sandstorm. Apps load super-slowly on Oasis. Many have significant missing features vs. their stand-alone versions, due to not having adapted to Sandstorm’s security model. And the Sandstorm UI itself remains woefully incomplete and janky. I constantly worry that most of the people paying for Oasis signed up by mistake and never noticed it on their credit card statements – that may sound far-fetched, but in fact I have had at least a few complaints from people who did just that (which I then refunded). I worry that it seems like we have European customers and I wonder if they realize Sandstorm is located in the US and may not comply with relevant European regulations. I feel embarrassed that people who haven’t read the blog assume the product is supported by full-time staff. Would Oasis still be profitable if it were only used by people who fully understand the state of the company? I’m not sure.

Finally, Oasis today provides almost no advantages over self-hosting. The price of virtual servers has come down to the point where self-hosting Sandstorm on an equivalently-priced server will give you a much better experience than Oasis can. Sandstorm was always supposed to be about owning your own server anyway. In fact, in retrospect, I think we never should have created Oasis, but should instead have focused entirely on self-hosting all along.

What’s next?

Sandstorm will continue to exist as an open source project. I personally plan to transfer my Oasis grains to a self-hosted server, and keep using it. I have to admit, building the mass transfer feature was kind of fun – I’d forgotten how little time it takes to build significant features in Meteor. And I’m still interested in self-hosting my email, if I can cobble together a decent UX. Maybe I’ll be inspired to build something on Sandstorm… we’ll see.

However, after the shutdown of Oasis, the project should be understood to be a hobby project, not a business. People should no longer rely on me, working in my spare time, to safeguard your data or keep it accessible.

Results of discontinuing the free plan

By Kenton Varda - 28 Oct 2018

Recently we made the tough decision to end Sandstorm Oasis’s free plan. This change has now been made and the dust is settled. (If you’re affected and are unsure how to export your data, see my previous blog post.)

Today, for the purpose of transparency, I wanted to show you the results of this change.

First, I’m happy to report that revenue increased more than I expected:

Graph of revenue showing when change was announced and implemented.

Timeline:

Second, server resources were reduced. In particular, utilization of Oasis’s “worker” machines (where users’ apps actually run) dropped in half:

Graph of CPU usage showing when change was implemented.

We previously had four worker VMs, each of which was a GCE “n1-highmem-2” machine costing $60.50 per month. We were able to cut two of those machines for a savings of $121.

Sandstorm currently runs 13 other VMs – six others to operate Oasis itself, three to run our web site and app market, two for Sandcats.io DNS, one for monitoring and one for metrics aggregation. It’s likely that we could further consolidate some of these, although these machines run smaller-sized instances with bespoke purposes meaning it will take a lot more work for comparatively smaller savings.

In August I estimated that the cost to continue operating Sandstorm (including servers and corporate maintenance) at about $1560 per month. With the server reduction, we’re now at $1439 per month – just barely above the $1428 in revenue. So, we went from a $700/month deficit ($8400/year) to break-even by making this change.