Minimal distribution


#1

Trying to figure out whether we can optimise our deployment scenario. Our application has a substantial client-side, many apps, many view. The server side is mostly implemented in enterprise services with OS.js activing as a pass-through. Can’t go into all the details but we need to distribute copies of the application to many sites. So far as I can see this means distributing the entire set of npm depedencies even though at run-time virtually everything we are “running” is already built into the “dist” folders - I think we’ve already established that once the app is built much of the npm tree is not referenced. Right now we have over 40,000 files (the xkcd cartoon was appreciated) and we are by no means finished developing.

I guess the question is whether there is some way to differentiate between the items that OS.js needs at runtime and that which is only needed at build time.


#2

You don’t have to distribute the npm packages, only their contents. When you run the package:discover command, the npm sources gets symlinked into dist/.

What you could do is to copy the npm package contents directly into the dist/ folders – that way that’s the only thing you have to deploy.

Now, saying this. The OS.js server will require some of the npm dependencies – but you don’t need it for the client-side stuff.


#3

I could possibly make a CLI command to automate this :slight_smile:


#4

I could possibly supply several beers to a person who did that :sunglasses:

Seriously, I think finding a way to get a more manageable deployment package would be a really big deal for us. I have a little time to help, but I think I would probably slow things down because I don’t really know what I’m doing - however I stand ready.


#5

I just got off work, so I’ll take a stab at this after dinner. I’ll let you know when it’s done :slight_smile:


#6

In the newly published version of @osjs/cli you can now run:

npx osjs-cli package:discover --copy

# or

npm run package:discover -- --copy

This will make physical copies of all the {package}/dist directories into dist/{type}/{package}/{dist}. Together with npm run build you have everything you need in dist/.

If your deployment also requires the server, you could make a minimal package.json file:

{
  "dependencies": {        
    "@osjs/server": "^3.0.1"
  }
}

You could probably put this file inside src/ and run npm install in there instead of overwriting the original file in the root.

Hope this helps :slight_smile: Let me know if I missed anything or got something wrong :nerd_face:


#7

Another solution is just to split out the server from your base distro entirely using git repo or something like that. That way you don’t even need the src/ directory – just deploy dist/ and server in parallel if required :slight_smile:


#8

Hm. I just come to think that this solution is not entirely complete. Package server scripts are placed in their respective sources.

I’ll have to also make a CLI command that extracts the server file and re-generates the packages.json (notice: plural) in the root directory. :thinking:


#9

Seems like we have various pieces and some other ideas to try. I need to carve out some focused time to experiment … I’m pretty sure that a workable solution can emerge. Thanks.


#10

I’m not entirely sure what the scenario is here though.

Do you need to deploy both the server and the client on to these different sites (servers) ?

Also, are you providing your packages via npm or do you have a monorepo ?

Because depending on these cases, there’s quite a few ways to optimize the deployment.


#11

I actually think that the best solution in these cases is to structure the packages a bit differently and using some of my previously mentioned methods.

I’ve published another @osjs/cli update to help this process. Seen in action below.

Step 1 - Restructure packages

# Standard
/package.json - All dependencies
/metadata.json
/server.js

# Separated
/package.json - Client/Build dependencies
/metadata.json
/server
  /package.json - Server dependencies
  /server.js

Then update server in metadata.json

Step 2 - Create some deployment files

A custom dependency list src/deploy/package.json:

{
  "dependencies": {        
    "@osjs/server": "^3.0.1"
  }
}

And a script to automate the process bin/pack.sh:

NOTE This does not add NODE_ENV=production to the client build. Ideally you should do this in your packages as well before deploying.

#!/bin/bash
#
# Dependencies: `jq`
#
dest="distro-packed"

# Create required directories
mkdir -p $dest/dist $dest/src

# Build client with custom path
npx webpack --output-path $dest/dist

# Discover packages with custom paths (including copy)
npx osjs-cli package:discover --copy --dist $dest/dist --discover $dest/packages.json

# Copy our deployment file templates
cp -r src/deploy/* $dest/

# Copy OS.js server
cp -r src/server $dest/src/

# Copy package server files
list=$(cat $dest/packages.json | jq -r '.[]')

for d in $list; do
  metadata=$(cat $d/metadata.json | jq -r '.server')

  if [[ $metadata != "null" ]]; then
    check="${d}/${metadata}"
    dirname=${check%/*}
    out="${dest}/${dirname}"

    if [[ -a $check ]]; then
      if [[ ! -d $check ]]; then
        echo "Warning: $check is a file.... dependencies might be missed"
      fi
      echo "Copying $check"
      mkdir -p $out
      cp -r $check $out/
    fi
  fi
done

Step 3 - Deploy

Now, copy the distro-packages folder onto the server, then:

# Base dependencies
npm install

# Then install required dependencies in each package:
# You can probably automate this as well
cd src/packages/<name>/server/ && npm install

# Then run server
node server/index.js

Notes

You can even put all of the package server dependencies in the root package.json file.

Depending on your requirements, you can also do the npm install before you copy the newly created distriution.


Example output:

distro-packed/src/server/index.js
distro-packed/src/server/config.js
distro-packed/src/packages/osjs-xterm-application/server/...

distro-packed/dist/f4cfc6708cf6bfc17d4b1e775f594ae2.png
distro-packed/dist/881c86876af63b732063d7bdd51bb226.png
distro-packed/dist/vendors~osjs.js.map
distro-packed/dist/vendors~osjs.js
distro-packed/dist/vendors~osjs.css.map
distro-packed/dist/vendors~osjs.css
distro-packed/dist/metadata.json
distro-packed/dist/index.html
distro-packed/dist/osjs.css.map
distro-packed/dist/osjs.css
distro-packed/dist/osjs.js.map
distro-packed/dist/osjs.js

distro-packed/dist/fonts/...
distro-packed/dist/sounds/...
distro-packed/dist/themes/...
distro-packed/dist/icons/...
distro-packed/dist/apps/...

distro-packed/package.json
distro-packed/packages.json

Is this of any use ?