Packages folder - diskspace


#1

Each new Native Application has it’s own directory under src/packages and within that directoryt its own node_modules folder. Even though I’ve got commonly used modules such as jQuery loaded globally I’m getting over 150Mb per application, and much of it is duplicated in every Native Application.

Am I doing something wrong here? What is the philosophy supposed to be in this kind of hierarchy of related npm “applications”. Should some of these packages be installed globally?


#2

This is something everyone that uses npm curses over every now and then. I don’t know if there’s a nice way to solve this.

Not sure if you’ve seen this joke before:

A way around this is to have the node_modules/ in src/packages/ and have all your dependencies there, shared among all your packages.

I’m 99% sure that both node and webpack will traverse downward in the file tree until it locates node_modules/ and use that as the primary location.

Another solution as you said is to install packages as global and then use npm link (directly using global packages is a big no-no and not even possible when using nvm)… but this is wonky at best.

You might also be able to configure webpack include paths to a shared location you store all your dependencies.


#3

https://github.com/lerna/lerna is supposed to make this a bit easier, but I’m not that sure, heh.


#4

Just adding to this. You can make “groups” of packages by adding more discovery paths – it does not have to be src/packages. This way you can have a monorepo or something like that for your packages that share deps.


#5

My first level take-away from this is that there is a small problem here, and there are some candidate recipes to mitigate. We need to document one such recipe. That should help with our builds as we scale up to having quite few Native Apps.

I get an itchy feeling that even then each app is going to have in its dist folder copies of the same thing and hence at runtime we’re going to be fetching more than we really need to, and the browser is going to be needlessly bloated.


#6

Remember that most of the node_modules/ stuff that is pulled is for development only. Pretty much nothing will end up in the resulting bundle by default.

Have you researched Externals in Webpack ?

The externals configuration option provides a way of excluding dependencies from the output bundles. Instead, the created bundle relies on that dependency to be present in the consumer’s environment. This feature is typically most useful to library developers , however there are a variety of applications for it.

https://webpack.js.org/configuration/externals/

Another way to solve some of these cases is to make a Service Provider in OS.js.

That way you can bundle libraries in the core distribution bundle (dist/vendors~main.js) and call for them at application runtime.


#7

Just wanted to add to this since I’ve faced this scenario on a couple of occasions.

You can build a “shared” bundle using https://webpack.js.org/plugins/dll-plugin/. This can be added to your OS.js package webpack configurations so you don’t have to use package.json as the source of your deps.

This requires an extra build step, but seems to work fine. And if you put your npm provided modules in here, build times are reduced as well.


This might not belong here, but you can take this even a step further by creating a fully mono repository if you want to have a bit more minimalist setup.

This will make use of a single webpack.config.js for the entirety of your project. If you combine this with the dll method above you can also reduce your bundle sizes.

1 - Remove individual builds

Then, remove the webpack.config.js files from all of your custom packges.

2 - Project-wide babel configuration

Then, switch over to a project-wide babel configuration by moving .babelrc to babel.config.js (in OS.js root) and change the format:

# From
{
  "json": "data"
}

# To
module.exports = {
  "json": "data"
}

3 - Project-wide webpack configuration.

You can now use the base webpack.config.js file to build everything:

const path = require('path');
const {merge} = require('lodash');

const packages = ['MyPackageName'];

const baseConfiguration = {
 mode: 'development',
 module: { /* ... */ }
 // ...
};

const clientConfiguration = merge({}, baseConfiguration, {
  entry: {
    osjs: [
      path.resolve(__dirname, 'src/client/index.js'),
      path.resolve(__dirname, 'src/client/index.scss')
    ]
  },
  plugins: [
    // You want the html webpack plugin here etc.
  ]
});

const packageConfigurations = packages.map(filename => {
  const abs = path.resolve(__dirname, 'src/packages', name);

  return merge({}, baseConfiguration, {
    context: abs,
    entry: path.resolve(abs, 'index.js'),
    output: {
      path: path.resolve(abs, 'dist')
    },
  });
});

module.exports = [
  clientConfiguration,
  ...packageConfigurations
];

Heck, there’s even another solution for this, which is to bundle the applications inside the client bundle and use a service provider to register them all. If that sounds interesting, let me know, and I’ll provide a sample.