Shaun Xu

The Sheep-Pen of the Shaun


News

logo

Shaun, the author of this blog is a semi-geek, clumsy developer, passionate speaker and incapable architect with about 10 years experience in .NET. He hopes to prove that software development is art rather than manufacturing. He's into cloud computing platform and technologies (Windows Azure, Aliyun) as well as WCF and ASP.NET MVC. Recently he's falling in love with JavaScript and Node.js.

Currently Shaun is working at IGT Technology Development (Beijing) Co., Ltd. as the architect responsible for product framework design and development.

MVP

My Stats

  • Posts - 96
  • Comments - 344
  • Trackbacks - 0

Tag Cloud


Recent Comments


Recent Posts


Archives


Post Categories



In one of my previous post I described and demonstrated how to use NPM packages in Node.js and Windows Azure Web Site (WAWS). In that post I used NPM command to install packages, and then use Git for Windows to commit my changes and sync them to WAWS git repository. Then WAWS will trigger a new deployment to host my Node.js application.

Someone may notice that, a NPM package may contains many files and could be a little bit huge. For example, the “azure” package, which is the Windows Azure SDK for Node.js, is about 6MB. Another popular package “express”, which is a rich MVC framework for Node.js, is about 1MB. When I firstly push my codes to Windows Azure, all of them must be uploaded to the cloud.

Is that possible to let Windows Azure download and install these packages for us? In this post, I will introduce how to make WAWS install all required packages for us when deploying.

 

Let’s Start with Demo

Demo is most straightforward. Let’s create a new WAWS and clone it to my local disk. Drag the folder into Git for Windows so that it can help us commit and push.

Please refer to this post if you are not familiar with how to use Windows Azure Web Site, Git deployment, git clone and Git for Windows.

And then open a command windows and install a package in our code folder. Let’s say I want to install “express”.

image

And then created a new Node.js file named “server.js” and pasted the code as below.

   1: var express = require("express");
   2: var app = express();
   3:  
   4: app.get("/", function(req, res) {
   5:     res.send("Hello Node.js and Express.");
   6: });
   7:  
   8: console.log("Web application opened.");
   9: app.listen(process.env.PORT);

If we switch to Git for Windows right now we will find that it detected the changes we made, which includes the “server.js” and all files under “node_modules” folder. What we need to upload should only be our source code, but the huge package files also have to be uploaded as well. Now I will show you how to exclude them and let Windows Azure install the package on the cloud.

First we need to add a special file named “.gitignore”. It seems cannot be done directly from the file explorer since this file only contains extension name. So we need to do it from command line. Navigate to the local repository folder and execute the command below to create an empty file named “.gitignore”. If the command windows asked for input just press Enter.

   1: echo > .gitignore

image

Now open this file and copy the content below and save.

   1: node_modules

Now if we switch to Git for Windows we will found that the packages under the “node_modules” were not in the change list. So now if we commit and push, the “express” packages will not be uploaded to Windows Azure.

image

Second, let’s tell Windows Azure which packages it needs to install when deploying. Create another file named “package.json” and copy the content below into that file and save.

   1: {
   2:   "name": "npmdemo",
   3:   "version": "1.0.0",
   4:   "dependencies": {
   5:     "express": "*"
   6:   }
   7: }

Now back to Git for Windows, commit our changes and push it to WAWS.

image

Then let’s open the WAWS in developer portal, we will see that there’s a new deployment finished. Click the arrow right side of this deployment we can see how WAWS handle this deployment. Especially we can find WAWS executed NPM.

image

And if we opened the log we can review what command WAWS executed to install the packages and the installation output messages. As you can see WAWS installed “express” for me from the cloud side, so that I don’t need to upload the whole bunch of the package to Azure.

image

Open this website and we can see the result, which proved the “express” had been installed successfully.

image

 

What’s Happened Under the Hood

Now let’s explain a bit on what the “.gitignore” and “package.json” mean.

The “.gitignore” is an ignore configuration file for git repository. All files and folders listed in the “.gitignore” will be skipped from git push. In the example below I copied “node_modules” into this file in my local repository. This means,  do not track and upload all files under the “node_modules” folder. So by using “.gitignore” I skipped all packages from uploading to Windows Azure.

“.gitignore” can contain files, folders. It can also contain the files and folders that we do NOT want to ignore. In the next section we will see how to use the un-ignore syntax to make the SQL package included.

The “package.json” file is the package definition file for Node.js application. We can define the application name, version, description, author, etc. information in it in JSON format. And we can also put the dependent packages as well, to indicate which packages this Node.js application is needed.

In WAWS, name and version is necessary. And when a deployment happened, WAWS will look into this file, find the dependent packages, execute the NPM command to install them one by one. So in the demo above I copied “express” into this file so that WAWS will install it for me automatically.

I updated the dependencies section of the “package.json” file manually. But this can be done partially automatically. If we have a valid “package.json” in our local repository, then when we are going to install some packages we can specify “--save” parameter in “npm install” command, so that NPM will help us upgrade the dependencies part.

For example, when I wanted to install “azure” package I should execute the command as below. Note that I added “--save” with the command.

   1: npm install azure --save

Once it finished my “package.json” will be updated automatically.

image

Each dependent packages will be presented here. The JSON key is the package name while the value is the version range. Below is a brief list of the version range format. For more information about the “package.json” please refer here.

Format Description Example
version Must match the version exactly. "azure": "0.6.7"
>=version Must be equal or great than the version. "azure": ">0.6.0"
1.2.x The version number must start with the supplied digits, but any digit may be used in place of the x. "azure": "0.6.x"
~version The version must be at least as high as the range, and it must be less than the next major revision above the range. "azure": "~0.6.7"
* Matches any version. "azure": "*"

And WAWS will install the proper version of the packages based on what you defined here. The process of WAWS git deployment and NPM installation would be like this.

image

 

But Some Packages…

As we know, when we specified the dependencies in “package.json” WAWS will download and install them on the cloud. For most of packages it works very well. But there are some special packages may not work. This means, if the package installation needs some special environment restraints it might be failed.

For example, the SQL Server Driver for Node.js package needs “node-gyp”, Python and C++ 2010 installed on the target machine during the NPM installation. If we just put the “msnodesql” in “package.json” file and push it to WAWS, the deployment will be failed since there’s no “node-gyp”, Python and C++ 2010 in the WAWS virtual machine.

For example, the “server.js” file.

   1: var express = require("express");
   2: var app = express();
   3:  
   4: app.get("/", function(req, res) {
   5:     res.send("Hello Node.js and Express.");
   6: });
   7:  
   8: var sql = require("msnodesql");
   9: var connectionString = "Driver={SQL Server Native Client 10.0};Server=tcp:tqy4c0isfr.database.windows.net,1433;Database=msteched2012;Uid=shaunxu@tqy4c0isfr;Pwd=P@ssw0rd123;Encrypt=yes;Connection Timeout=30;";
  10: app.get("/sql", function (req, res) {
  11:     sql.open(connectionString, function (err, conn) {
  12:         if (err) {
  13:             console.log(err);
  14:             res.send(500, "Cannot open connection.");
  15:         }
  16:         else {
  17:             conn.queryRaw("SELECT * FROM [Resource]", function (err, results) {
  18:                 if (err) {
  19:                     console.log(err);
  20:                     res.send(500, "Cannot retrieve records.");
  21:                 }
  22:                 else {
  23:                     res.json(results);
  24:                 }
  25:             });
  26:         }
  27:     });
  28: });
  29:  
  30: console.log("Web application opened.");
  31: app.listen(process.env.PORT);

The “package.json” file.

   1: {
   2:   "name": "npmdemo",
   3:   "version": "1.0.0",
   4:   "dependencies": {
   5:     "express": "*",
   6:     "msnodesql": "*"
   7:   }
   8: }

And it failed to deploy to WAWS.

image

From the NPM log we can see it’s because “msnodesql” cannot be installed on WAWS.

image

The solution is, in “.gitignore” file we should ignore all packages except the “msnodesql”, and upload the package by ourselves. This can be done by use the content as below. We firstly un-ignored the “node_modules” folder. And then we ignored all sub folders but need git to check each sub folders. And then we un-ignore one of the sub folders named “msnodesql” which is the SQL Server Node.js Driver.

   1: !node_modules/
   2:  
   3: node_modules/*
   4: !node_modules/msnodesql

For more information about the syntax of “.gitignore” please refer to this thread.

Now if we go to Git for Windows we will find the “msnodesql” was included in the uncommitted set while “express” was not. I also need remove the dependency of “msnodesql” from “package.json”.

image

Commit and push to WAWS. Now we can see the deployment successfully done.

image

And then we can use the Windows Azure SQL Database from our Node.js application through the “msnodesql” package we uploaded.

image

 

Summary

In this post I demonstrated how to leverage the deployment process of Windows Azure Web Site to install NPM packages during the publish action. With the “.gitignore” and “package.json” file we can ignore the dependent packages from our Node.js and let Windows Azure Web Site download and install them while deployed.

For some special packages that cannot be installed by Windows Azure Web Site, such as “msnodesql”, we can put them into the publish payload as well.

With the combination of Windows Azure Web Site, Node.js and NPM it makes even more easy and quick for us to develop and deploy our Node.js application to the cloud.

 

Hope this helps,

Shaun

All documents and related graphics, codes are provided "AS IS" without warranty of any kind.
Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.

Comments

Gravatar # re: Install NPM Packages Automatically for Node.js on Windows Azure Web Site
Posted by Nabarun on 1/15/2013 8:15 PM
Hi Shaun,
Great post! It helped me a lot. However I am facing some issues in deploying the oauth Node.js module. Any suggestions regarding that will help a lot.
Thanks
Nabarun
Gravatar # re: Install NPM Packages Automatically for Node.js on Windows Azure Web Site
Posted by Joakim Månsson on 4/29/2013 6:47 PM
VERY nice post,

However this is for Azure Websites right? How about when running a Node-server on Cloud Services which I am and I have issues with the msnodesql-plugin.
Gravatar # re: Install NPM Packages Automatically for Node.js on Windows Azure Web Site
Posted by Joakim Månsson on 4/29/2013 8:37 PM
Never mind. I fixed it!

https://github.com/WindowsAzure/node-sqlserver/issues/114
Post A Comment
Title:
Name:
Email:
Comment:
Verification: