This post was originally published on mainmatter.com
Embroider is of critical importance for the entire Ember ecosystem, yet has been in development for many years with no clear end in sight. The Embroider Initiative is spearheaded by Mainmatter with the support of a group of sponsors to help speed up development on Embroider so that we can make it the default build system for newly generated Ember apps as soon as possible. This will allow Ember developers to continue leveraging its advantages while benefiting from a modern build system. We hope to see Ember apps being built with Vite before the end of the year.
The Embroider Initiative aims to:
- Finish Embroider itself by assigning experienced Ember engineers full-time to the project. This covers development work on Embroider and its dependencies, as well as helping backers setting it up in their repos to uncover and fix edge cases.
- Make Embroider maintainable by decentralizing technical knowledge beyond the project's core developer Ed Faulkner and thus improving the bus factor. Mainmatter makes use of the apprenticeship model to onboard and train new developers on the intricate parts of the project.
- Shift the ecosystem and make Embroider mainstream by making it easier to generate Embroider-optimized Ember apps and supporting addon developers make their addons compatible with Embroider.
To reach these goals and keep up the momentum, the project needs your financial support. Please get in touch to learn how to directly benefit from this investment.
Mainmatter founder and director Marco Otte-Witte recently talked about the Embroider Initiative and open source funding in general at EmberFest in Madrid. He detailed how important the Initiative is to the future of Ember itself and to all companies that have built on Ember and need to secure the investments they have made in Ember. He also went into detail about the Embroider Initiative in his original blog post that announced the initiative. This blog post will dive into the things our team has been able to achieve up until now.
Getting Ember to work with Vite
At EmberConf this year Ed Faulkner announced that we were closing in on a Vite plugin for Embroider. While that was true at the time, we have learned a lot about the Vite build process since then and we know more about the steps that are still required to get the Vite integration working.
The Vite app that Ed demoed at EmberConf was a trivial app that is a package in the Embroider monorepo and if you wanted to test it yourself then you could either clone the Embroider monorepo, or you could clone this repo which is essentially just extracting the same test app into an independent repo. It works, and you can even see the incredible rebuild speeds in action.
The issue with this trivial demo is that it doesn't represent an average Ember application. There aren't any Ember applications out there that don't have a single addon installed. While it's not exactly true that the demo doesn't have any addons installed, it doesn't have any addons that are doing any real work. And, as it turns out, getting dependencies to work right is the challenge with the Vite build.
Ed and Mainmatter Senior Engineer Chris Manson have been pairing weekly, plugging away at the remaining things that are required to fix the Vite build. We are now optimistic we can get to a point where it's possible to build the first real-world Ember applications with Vite until the end of the year.
While the main focus of the Embroider Initiative was always going to be the Embroider code base, there are other parts of the ecosystem that will require some work to bring them more in line with how we want people to build their apps.
If you're already using Embroider, you will know that a lot of the work to
package your app is done by Webpack. If you're still on a classic build, you may
not be aware that ember-auto-import uses Webpack under the hood to allow you to
seamlessly import from
node_modules. This has been a very useful feature but
since the acceptance of the
v2 addon spec RFC
we have noticed that we have a bit of a blindspot in classic builds. Since v2
addons can't influence the build in any way (effectively making them static
packages) addon authors need to add extra installation instructions to detail
how to add a Webpack plugin to their application build config if they still
wanted to influence the build process in any way. This is perfectly legitimate
in Embroider but it does not work for classic apps.
The issue is that ember-auto-import was originally designed to only work with npm packages, so that means that classic apps couldn't add a Webpack plugin that would influence the build process for any files controlled by their app, only for files controlled by npm packages or addons. This has been a blocker for some addon developers who want to upgrade their addons to the new v2 format and our solution to this problem has been to add a new config to ember-auto-import to allow you to specify parts of your app that should be under its control.
While this work has been done to facilitate v2 addons having the same install instructions for classic ember-cli builds and Embroider apps, this functionality could also be considered a way to allow you to opt-in to Embroider on a folder by folder basis and when your whole app is being controlled by ember-auto-import (and Webpack) the move to embroider should technically require no changes to the app.
Progress for Embroider Initiative backers
For the Embroider Initiative to be successful, it needs backing from companies that would benefit from a modern and thriving Ember ecosystem. But the benefits of sponsoring the project go beyond the end goal.
While all sponsorship tiers (starting at 3k€ for individual supporters) include a backer's logo in Mainmatter communication around the Initiative, companies committing 18k€ to the project get access to the weekly 1-hour call with the team and get opportunities to discuss their technical needs and challenges related to Embroider.
Premium tiers of 36k€ and above include a 2-hour weekly private session with Mainmatter engineers focused on improving the backer's Ember build. In practice, these backers have seen a great return on their investment as the Mainmatter Embroider team has been able to deliver dramatic speed improvements to their build, removing a bottleneck in their development process. Some issues specific to these backers also resulted in solutions that the larger Ember community can now benefit from.
Let's see some success stories in details:
Like many companies, Ticketsolve has an internal addon that they use throughout their other applications. Internal addons are a great place to start when you are thinking about upgrading to Embroider for two reasons:
- It's an isolated place to opt into Embroider's stricter requirements
- There is a clear specification of what your v2 addon can do.
Chris' work on improving the ergonomics of the v2 addon blueprint in the early weeks of the Embroider Initiative, proved very useful during the process of converting Ticketsolve's internal addon.
After the internal addon was fully converted to v2 and deployed to all of
Ticketsolve's apps, our next step was to add support for
in both the internal addon and their applications. The team also worked on the
ecosystem PRs required to enable GJS support in v2 addons in both a
PR and a
PR to the
Before the Embroider Initiative, Ticketsolve had one of their three apps already
running Embroider. We worked to update the Embroider version of that app, then
started to convert the remaining two apps. There were some challenges along the
way, mostly related to addons that just didn't support Embroider. A great
example of this is the
which uses the
postprocessTree() hook that doesn't work with Embroider. We
updated the addon locally using
pnpm patch <pkg>
submitted a PR with the same changes
to help the rest of the community.
This is only a fraction of the changes that came out of these pairing sessions, but it gives a flavour of the kinds of things that we have been able to achieve. While our Chris Manson has been instrumental in helping speed up this work, Ticketsolve's engineer Andreas Minnich deserves a shout-out for the colossal amount of work they have put into this effort between pairing sessions.
Intercom also had an internal addon they wanted to convert to v2 before working
on converting their main application. Mainmatter worked with them to convert
that and, incidentally, we also helped them to convert the tests app that was
documenting the v2 addon to use Embroider with the
staticAddonTrees flags turned on.
While this work was going on, Intercom Engineer Aaron Chambers had setup a CI build step that would keep track if the main Intercom app could build with Embroider and if any tests were passing. Because of this CI job, Aaron identified a 10x slowdown in build times between Embroider v3.1 and v3.2. Chris used a number of pairing sessions to dive into the issue and produce a number of flamecharts for their Embroider builds to get an insight into what was causing their specific slowdown. Those flamecharts were pointing at a particular part of Embroider's package cache not working exactly as we intended it to. Ed Faulkner managed to pinpoint the problem (while we were discussing it in the "hallway track" of EmberFest) and opened a pull-request with the fix. This completely fixed the performance regression for Intercom.
The Mainmatter Embroider team also noticed that Intercom's build was behaving
differently on CI compared to running the exact same build locally. We tracked
it down to the fact that Intercom's CI was configured to create a symbolic link
to an existing
node_modules folder rather than running a new
npm install for
every job. As it turns out, this was the same reason that the CI was failing for
Chris' pull request to make Embroider optimised the default
when running the command
ember new --embroider. Chris identified the cause of
the issue inside Embroider and
fixed it without the
need for any assistance from Ed, which also shows the success in making
Embroider more maintainable by increasing the team size and distributing
technical knowledge (more about this in the next section).
The rest of the pairing sessions were spent working through issues that were
causing tests to fail. Some of these issues were systemic and when they were
fixed they would turn hundreds of tests green (Intercom has a lot of tests),
while other problems were highly specific and required multiple pairing sessions
to identify and fix the problem. An example of this would be Intercom's use of
the charting library Highcharts. When a Highcharts
user needs to include timezones in their Date axis, the library will check for
the presence of
window.moment and use it for any timezone-related
calculations. This allows the user to have a chance to setup
moment-timezone and correctly configure it with the right set of timezone
data for their application. With the move to Embroider,
window.moment is no
longer accidentally set to the correct instance of
moment-timzone as a
side-effect of the build system, so Highcharts wasn't finding the right instance
during its initialisation. It turns out that Highcharts
provides a config option,
time.moment, to cover this exact case and, as soon as the team set that
correctly in the application's charts base class, the x-axis started behaving
Again, these are only a fraction of the changes that came out of these pairing sessions. Both Aaron Chambers and Peter Meehan were able to achieve a lot of big things over the course of the Embroider Initiative for Intercom and our team was more than happy to help speed the process along.
Improving the bus factor
The goals of Embroider may seem simple from the outside, i.e. "just use Webpack or Vite to build your Ember app", but when digging a bit deeper, it's easy to see how complex of a project it really is. This complexity arises from the project having challenging design constraints which pose a significant challenge to anyone who would like to contribute to the core of Embroider. The main design constraint that causes a lot of this complexity arises from the fact that the team wants to provide an easy on-ramp for existing Ember apps to convert to Embroider, and then slowly move those apps from full-compatibility mode to a "fully static" build that can automatically benefit from tree-shaking and code-splitting. This means that we need to provide systems that can automatically convert the still-supported conventions of an Ember app to fully standard compliant ESM code. This is a significant challenge since some of the patterns that are still officially supported today date back to the 1.x series of Ember which was released in 2015.
Because of the overall complexity of project and the rapid iteration of the internal architecture, it hasn't been practical to create any documentation that could communicate what is going on under the hood. This is illustrated by the fact that Aaron Chambers and Peter Meehan put together a document trying to explain the full architecture of the project, which was completely outdated only less than a month later when Ed Faulkner merged the first of many internal refactors.
To address this challenge, we have adopted an apprenticeship model where Ed Faulkner pairs with Chris every week, collaborating on solving complex problems deep within the heart of the Embroider codebase. This has made a significant improvement to the bus factor of the Embroider project and has also had a deep impact on its velocity.
With our second full-time engineer on the Embroider Initiative, Andrey Mikhaylov, we have extended the apprenticeship model by having Chris and Andrey pair-program for half a day each week. This has helped both Chris and Andrey ramp up their knowledge of the codebase while improving the overall project's bus factor.
General stability and ecosystem improvements
Providing watch-mode tests for Embroider
Recently, Godfrey Chan has been discovering some places in Embroider and our Webpack plugin that were causing crashes when certain files were added or deleted. He has already come up with a fix for some of the cases, but it showed a blind spot in the team's testing infrastructure that meant we weren't testing "watch mode" in Embroider. Preston Sego and Chris paired together to add some basic watch mode tests that would highlight the problem and prove the effectiveness of the fix, but they were hit by some strange quirks that prevented the watch-mode tests from ever exiting properly on the Windows CI job. Chris spent most of his weekly streaming session on Twitch trying to figure out the solution. The team finally got the PR merged and is now ready to start adding more expansive watch-mode tests.
ember-cli-update supporting v2 addons
ember-cli-update is a great asset to the Ember community, and it's a vital
tool during the push for Embroider and encouraging the community to use the new
v2 addon blueprint because a lot of the recent changes require updating both a
dependency version and related configuration.
One issue the team worked through was to fix CI, which had been broken for a
ember-cli-update interacts with the npm APIs to discover versions, but a
change on the npm side prevented it from functioning properly. The solution to
fixing CI was ultimately updating a dependency,
boilerplate-update, to use
pacote to interact with the npm API.
Once the upstream dependency was updated, the team only needed to
add a few tweaks to ember-cli-update
to get CI to finally pass. This gave us time and confidence for the rest of the
ember-cli-update works by generating a diff between the version of the
blueprint that the app or addon was generated with, and the version to update it
to. Under the hood, this means that
ember-cli-update will generate a pristine
new version of the app/addon at its current version, then generates a new
pristine copy of the target version. It generates a
git diff between those
versions then applies that diff to the current app.
Generating these pristine "from" and "to" versions of the blueprint involved a
custom code-path in
ember-cli-update that aimed at working around some bugs in
Node 8. As those workarounds were causing it to fail, Chris'
fix for custom blueprints
involved removing these workarounds.
This fix was released in
then followed by
that fixed a small bug. These two releases are a big deal for the wider Ember
community, especially as its members are encouraged to migrate their existing v1
addons to v2 addons. They bring an important DX functionality that the Ember
community has come to expect.
Embroider optimised with the
Creating a new Ember app with the
ember new my-super-app --embroider) generates a "full compat" app were none
Embroider optimisation flags
are passed to the build for the developer.
While more existing apps will work with a "full compat" mode, Ember has reached a point where it makes sense for newly generated apps to start with a high water mark so that developers don't accidentally or unconsciously add functionality that won't work in a fully optimised Embroider application. Developers have the option to turn off any of the optimisation flags, but it would be a deliberate choice as they would need to add a specific dependency or functionality to their app.
The Mainmatter Embroider team opened a
PR to switch the functionality
so that passing the
--embroider flag uses Embroider optimised by default. This
involved working through issues with some of the slow test suites that rely on a
custom package caching mechanism. The PR got merged and the functionality will
be part of Ember v5.5.
scenario-tester ESM compatibility
Scenario tester is a
testing tool that the team makes use of a lot when testing Embroider and
ember-auto-import. It allows us to generate many scenarios with different
combinations of dependencies. We tried using it outside of Embroider, or more
specifically outside of a Typescript project, and saw it doesn't work in a CJS
Chris started the effort to move the Typescript build to output an ESM-compatible build
so it can be consumed directly in ESM without a build step. The only remaining
task is to test if it will work in Embroider before merging and releasing the
Documenting the scenario-tester library.
scenario-tester is to Embroider
ember-try is to Ember CLI: it's a tool that lets us perform automated
tests with various combinations of dependencies, configs and circumstances. The
scenario-tester is different: instead of reinstalling dependencies
for every test case, it has all dependencies (including all versions of
dependencies) set up once, saving a lot of time. It leverages
create and emit to filesystem Ember apps and addons with predefined dependencies
and configuration, in order to run tests on them.
Working on the reverse-exports
During the build, Embroider needs to expose Ember internals to Vite and Webpack
in a way they can understand and consume. Modern Ember apps can have multiple
points in their
package.json configs. This poses a peculiar challenge for
Embroider: it needs to know how to reorganize files in an Ember project in such
a way that they would resolve into paths defined as
Essentially, this requires resolving
exports in reverse, and this it what the
package is for.
Call to action
We are hoping we can extend the initiative's budget and timeline to keep up the momentum in 2024 and finish the work we started. Please get in touch to learn how to support the Embroider Initiative and directly benefit from this investment!