By Jacob Weisz - 03 Nov 2023
Sandstorm has been on a fairly slow and gradual transition from a dead startup to a community-operated open source project. Last week we posted our first update about our plans for the future of Sandstorm here on sandstorm.org, and now we’re going to explain what the domain change means going forward.
The .io TLD has been a popular choice for startups for a number of years. However, it is a problematic TLD choice. One of our earliest key sponsors, draw.io, moved away from the domain a few years ago.
While the .org TLD has had its own share of problems, it’s large enough and established enough that the watchful eyes of the Internet have so far prevailed in protecting it. And the sandstorm.org domain better represents us as a nonprofit project mission-focused on bringing self-hosting to everyone.
Software that keeps itself updated, or “evergreen” is convenient and popular today, but it comes with a big risk: You trust the developer not just for the code they ship today, but for years into the future. You also are placing trust in the developer not to sell or give away that trust to someone else. Evergreen software like browser extensions have been known to be bought out in order to inject adware on unsuspecting users down the road.
Every Sandstorm server installed in the past ten years has entrusted Kenton Varda or Sandstorm Development Group with that level of access. While the members of our community have contributed to Sandstorm’s ecosystem for a number of years, we can’t ethically take over maintenance of Sandstorm servers without permission. While we are looking to extend and grow the ecosystem, server operators deserve the right to choose whether to trust us.
So as we develop a community-based path forward, we are going to ask server operators to opt-in to utilizing our releases and services, rather than taking them over.
The final, practical reason to make this fork now, is ultimately because of the project we are undertaking to upgrade Sandstorm’s database. This is a project we have found uncomfortable to automatically deploy, in case it leaves someone’s server in a non-working state.
By releasing that upgrade as a “fork”, it will allow administrators to take a backup of their server prior to both agreeing to entrust our team with future updates as well as migrating to the new version of the database. We plan to replace the references in the code to services currently hosted at sandstorm.io with sandstorm.org alternatives at the same time. And that minimizes the disruptive work of this process.
Sandcats is one of Sandstorm’s few “centralized services” which still runs today, so transferring it to our community would require a handoff of user data. We don’t want to do that without consent, so if we take over administration of the sandcats.io domain, we’ll provide significant advance notice and ensure only consenting users’ data is included.
The other issue with Sandcats is that its current infrastructure is fairly old, and before we look at taking over operation of the service, we need to update the platform and stand up a modern version of it.
This will be a very important process to get right, so we will talk about it often as we get closer to making any change that impacts users.
Finally, I wanted to say thank you to our new backers who have contributed to our OpenCollective. We still have a ways to go in order to fund our database upgrade, but we are a lot closer today than we were a few weeks ago. Some people have reached out to ask how they can contribute to Tempest or app development, and it has been a delight to work with some great open source developers on app updates this month.
By Jacob Weisz - 23 Oct 2023
Over the last couple of months, we have had a lot of people asking for an update on the project. We’ve had to take a bit of time for self-care before putting together our plans going forward, culminating in this update.
Note: Astute observers may notice this blog is on a new domain, sandstorm.org, along with subtle changes to this version of the website. We will discuss why and how this will impact our plans next week.
As people closely following the project may already be aware, about three months ago one of our core project contributors, Ian Denhardt, sadly and unexpectedly passed away. It is impossible to overstate the significance of Ian’s contributions to this project, from maintaining and fixing bugs in Sandstorm itself, to packaging, re-packaging, and building a number of apps and tools, to starting a next-generation platform for Sandstorm (more on this below). Ian has ultimately been the primary contributor to our ecosystem as a whole for the past three years.
Ian was an incredibly gifted developer whose projects spanned from calendaring apps and backup tools to operating system kernels and programming languages. But far more than that, Ian was an incredibly humble and kind mentor and friend. Countless times he helped me with problems that came down to something close to “How do I navigate a Linux machine” without a hint of irritation. Steve Pomeroy shared a story about Ian recently that exemplified the sort of thought and care that he put into everything that he did.
We intend to honor Ian’s memory and vision by carrying forward his work and ensuring that we continue to make self-hosting viable for everyone.
Since late last year, Ian had been working heavily on an experimental alternative software to run Sandstorm apps that we refer to as Tempest. Tempest is a modern take on the Sandstorm platform, supporting existing apps but dropping various long-disused compatibility shims and embracing modern technologies and expectations. It is written in Go, which is significantly more accessible to volunteer development than Meteor and C++. Another key feature is the use of WASM to allow the client side of the platform to also be written in Go and run Cap’n Proto directly in the browser.
Whereas Sandstorm was built with organization or SaaS provider scale as a focus, Tempest development was planned with individual or small community use at the forefront. One of the major goals Ian had intended to strive for with Tempest was the ability to communicate with decentralized communication networks like ActivityPub and Matrix, which has proven challenging for Sandstorm thus far.
As of this post, Tempest can run many Sandstorm apps and can import grains directly from a Sandstorm install. However, it is still far from being a production-usable application, and a lot of important API features for apps like external network access and web publishing are not implemented.
Multiple community members have stepped forward in the past two months to let us know that they would like to continue development of Tempest including Troy Farrell and Dan Krol. We plan to bring the current work on the project under the Sandstorm organization to coordinate that effort. If you are interested in contributing, please reach out on the project repository or the Sandstorm mailing list.
While Tempest has been exciting as a potential leap forward, everyone using Sandstorm today is still using “Sandstorm classic”. Sandstorm development has moved more slowly, but it has not stood still. Several security improvements have been added in the past couple of years, including improved sandboxing both on the server and client side and a number of small but impactful UI improvements. Last month, Sandstorm’s 308th release included Ian’s final contribution as well as additional localization work that Troy contributed.
Unfortunately, as we hope to continue to support Sandstorm long-term, we’ve had an increasingly challenging problem with the legacy MongoDB database underneath, as Meteor has dropped support for the version we use. Ensuring a seamless migration on end-user servers has proven to be a challenging problem. Dependency upgrades are not fun and this has become a fairly unpleasant project to undertake for unpaid open source contributors.
Over the past few weeks, we have spoken to a number of potential outside parties about taking on this project and building a migration so that we can update the databases of existing Sandstorm servers and allow us to continue moving forward. We believe this is the best way to ensure that Sandstorm servers continue to be the safest way to self-host. The cost estimates we’ve discussed for the current Sandstorm aren’t unreasonable, but they are significant.
We are looking to raise the funding for this project through our OpenCollective. If you are using Sandstorm, especially at an organization level, and can help ensure that we are able to provide long-term support, please donate to the project. If you believe that you or your organization may be willing to fund this effort in large part or its entirety and would like additional details about the amount we would need to fund this, please contact me directly.
Sandstorm is approaching its tenth birthday, and there’s still nothing else like it. Capability-based security and a tireless focus on abstracting away complexity from the user continue to make it the only platform which holds hope of letting everyone, not just developers and IT professionals, self-host. Thank you for your continuing interest and support.
By Ian Denhardt - 17 Jun 2021
Hey Everyone!
Members of the Sandstorm community are doing some fundraising!
Specifically,
The organizers are familiar faces, but we aren’t and haven’t ever been formally affiliated with Sandstorm, Inc; we’re just long-time community members looking to chart a path for the Sandstorm project going forward. We all miss the days when there were several people working on Sandstorm full-time, and want to find ways to support continued development of new features and additional app packaging efforts.
If you want to help us out here’s how you can:
By Ian Denhardt - 08 Aug 2020
This past week I updated the Sandstorm package for Tiny Tiny RSS. In addition to updating to the latest upstream code, I’ve also made some improvements to the package’s integration with Sandstorm. In particular:
The latter of these is critical to enable Sandstorm’s goal of full network isolation for applications. It has always been the intent that apps running on a Sandstorm server should not be able to phone home or otherwise access the network without the user’s consent, but early on the team implemented some temporary hacks that apps could make use of until Sandstorm’s Powerbox API was actually available. Now it is, so it’s time to migrate existing code away from these legacy APIs. Once we’ve done that, we can remove the holes, and finally deliver on Sandstorm’s security promise of network isolation for server code.
In order to make this happen, I wrote a daemon that runs inside of the grain, intercepting HTTP traffic and making powerbox requests for access to the relevant hosts on-demand. The daemon is re-usable and could be used as a quick way to port other existing apps that have legitimate need to access a variety of different hosts at runtime, which are not known in advance.
Unfortunately, this transition does come with some growing pains, and there are a few things users should be aware of when upgrading.
First, since existing TTRSS grains will already have subscriptions to feeds that they haven’t requested access to, the first time you start the new version of TTRSS you’ll see a flood of requests for all of your existing feeds. This is annoying, but fortunately after this initial setup you’ll only need to grant access when adding new feeds.
Second, a more serious limitation of the current implementation is that it makes it impossible to add new feeds through mobile & desktop clients – at least without also having a browser window open somewhere. This is because, without the Sandstorm UI open, Sandstorm has no way of actually showing a user the powerbox dialog. We’d like to find ways to improve on this.
Finally, while the daemon is exciting for app packagers, in that it can make basic porting of apps not designed for Sandstorm much quicker, the integration is imperfect, resulting in a sub-par UX: as it stands, users will see an extra prompt (or two) when first subscribing to a feed.
Maintaining security while avoiding annoying extra prompts like this is one of the design goals of Sandstorm’s powerbox, but to fully take advantage of it will require more invasive changes to TTRSS; rather than prompting the user for a feed URL and then trying to fetch it (causing the daemon to request access), it should just ask Sandstorm for a feed directly. If you’re an enterprising PHP hacker and want to help make this happen, get in touch on the sandstorm-dev mailing list or the #sandstorm IRC channel on libera.chat.
By Kenton Varda - 13 Jun 2020
Sandstorm now has built-in support for fetching certificates from Let’s Encrypt. This applies both to Sandcats and to custom domains.
Sandcats.io is Sandstorm’s free dynamic DNS and TLS (aka SSL) certificate service. Since 2015, Sandcats has been making it easier to set up self-hosted Sandstorm servers, by letting anyone claim a subdomain of sandcats.io on which to host their server, automatically configuring DNS and HTTPS.
Sandstorm requires a wildcard host so that it can create sandboxes by assigning random hostnames. So, the certificates issued by Sandcats.io have always been wildcard certs. Back in 2015, that was a challenge: at the time, Let’s Encrypt did not yet offer wildcard certs, so the only way to get one was to pay for it. Unfortunately, many CAs considered wildcards to be an “enterprise feature” and charged excessive amounts of money for them, often hundreds or even thousands of dollars.
So how were we able to offer wildcard certificates for free? Well, we negotiated a deal with GlobalSign. Recognizing that subdomains of Sandcats.io were unlikely to be enterprise customers, and recognizing a growth opportunity if Sandstorm took off, GlobalSign offered us a deal that worked. Sandstorm was a funded startup at the time, and we were happy to pay a few bucks per certificate-year and call it “customer acquisition cost”. We paid for thousands of certificate-years upfront, anticipating growth.
But, the growth we hoped for didn’t happen, and we only ever had about 500 self-hosted servers using Sandcats. In 2017, Sandstorm failed as a startup and reverted to being just an open source project. Having never seen the growth we were hoping for, we hadn’t even come close to using up the first block of certificates we had paid for from GlobalSign. Since we had already paid, we left the system running, happily issuing certificates.
Fast forward to 2020. Finally, our contract is running out. In fact, it did run out, in early March – embarrassingly, I had miscounted how much time we had left. However, GlobalSign was nice enough to give us a small extension, giving us time to migrate our users without disruption. And it turns out that, these days, Let’s Encrypt supports wildcards. So, moving to them is the obvious choice.
Starting a few days ago, all Sandstorm servers using Sandcats.io are in the process of switching to Let’s Encrypt for future certificates. The process is designed to happen slowly so that we can address any problems that arise, but so far everything has been smooth. All servers should be transitioned by the end of the month.
If you’d like your server to start using Let’s Encrypt immediately, visit the TLS certificates admin page at /admin/certificates on your server, click the button to create an ACME account (you will be prompted to agree to Let’s Encrypt’s Terms of Service), and then click “Fetch Certificate Now”.
Note that at present, we have not yet updated the install flow, so newly-installed Sandstorm servers will still use a GlobalSign certificate initially and then will switch to Let’s Encrypt later. We’ll be updating the installer soon, and then we will decommission the GlobalSign flow.
To implement Let’s Encrypt support for Sandcats.io, it made sense for your self-hosted Sandstorm server to directly talk to Let’s Encrypt via the ACME protocol. This differs from the old GlobalSign flow, in which Sandstorm would talk to a central Sandcats.io server, which then talked to the GlobalSign API on your behalf. This was necessary for GlobalSign since Sandstorm the company was paying for the certificates, so our credentials were needed when talking to the API. But for Let’s Encrypt, the Sandcats.io central server only needs to set some DNS records to pass an ACME DNS-01 challenge; everything else happens on your own server.
Given this implementation, it was straightforward to support domains other than sandcats.io. I chose to use the ACME.js library to implement the ACME protocol, and it turns out this library already had a suite of plugins to support a variety of popular DNS providers. If your domain uses any of the DNS providers supported by ACME.js, then Sandstorm can now automatically obtain TLS certificates for your domain from Let’s Encrypt.
However, at present, initial setup is difficult, because of the chicken-and-egg problem: How do you access the admin UI to configure certificates, without a certificate? And than brings me to…
Currently, there are two major problems with setting up Sandstorm TLS on your own domain:
If you’d like to help implement either of these, click on the issue links above and comment!