Distributed Computing

Digipede Webinar, November 1

Robert W. Anderson — Fri, 28 Oct 2005 15:55:00 GMT

Dan Ciruli and Nathan Trueblood will be running a webinar on the DN, showing how to speed up applications. They do a great job of giving an overview of its power and functionality. The focus of this webinar will be financial applications.

Go here for details on what will be covered and to sign up. It is scheduled for 10AM Pacific and will be about 30 minutes.

Dan is off to Code Camp

Robert W. Anderson — Fri, 21 Oct 2005 21:54:00 GMT

Dan is off to Code Camp in Seattle this weekend. He'll be talking on Sunday @ 3 about grid-object oriented programming (what he calls GOOP). He'll be talking about that and showing the Digipede Network at the same time.

If you are going to be there drop by and see him.

Distributed / Grid / Controls?

Robert W. Anderson — Tue, 11 Oct 2005 10:52:00 GMT

Dan's post (Grids? Clusters? Distributed Computing?) explains why we refer to the Digipede Network as “Grid Computing for Windows“.

Several speakers at the recent Microsoft eScience Workshop defined Grid as well and many others steered away from the term altogether. As Ian Foster and Carl Kesselman are generally credited with the definition of this term (from his book), there are many people who think of Grid as being exactly Globus. To these people, Grid means large, complex, multi-organizational, cross-platform (mostly Unix and its pretenders) computing systems that integrate sensors and visualization with distributed computing power.

As we are trying to bring simplification to this kind of computing, I wanted to steer away from calling our product a “Grid“. I truly thought it would be more confusing if we did so; however, I was wrong.

We found again and again that when we described our product we often ended with “you know, like a computing grid“. That was when people began to get it.

So, we decided to go directly to the punchline.

Of course, we have also had a few people ask us: your whole company is based on a grid (i.e., tabular UI) control? ;)

eScience / Savas Parastaditis

Robert W. Anderson — Mon, 10 Oct 2005 09:46:00 GMT

Savas Parastaditis gave a talk on his White Dwarfs application and the WS-GAF work he did while at University of Newcastle. He had the worst timeslot of anyone in the workshop: It was after dinner and we were all getting pretty tired. The workshop had started over 12 hours before (and aside from a few breaks) was packed with content. Anyway, Savas did a good job with his talk. The White Dwarfs application is an interesting application that brought together existing data to help astronomers search for white dwarfs.

I chatted with him for a few minutes about MEST: like many, I am eager to read his coming paper (or book?). He has gotten me thinking about a mechanism that we use to simulate MEST-like behavior in a world where the clients are not services (nor are they assumed to be reachable from the service). Unfortunately, I can only guess if such methods for simulation are MEST- or XEST (i.e., something else altogether). I will blog my thoughts on this and see what Savas as to say about it.

I had been in contact with the Savas before eScience and was happy to finally meet him: he is a real nice guy.

eScience / Marvin Theimer

Robert W. Anderson — Mon, 10 Oct 2005 09:44:00 GMT

I am getting behind in my blogging: I missed the second day of the eScience event, took most of Friday off and went camping with my family. What a beautiful weekend!

Here is the start of my (short) wrap-up to the event.

Marvin Theimer, architect for the new offering (I believe it is now called Compute Cluster Server) from the Microsoft HPC group, stood in for Kryil Faenov to talk about Microsoft’s view of the changes coming in the HPC industry. For version 1.0, they seem to be more and more focused on what they called the “personal supercomputer.” – an 8 or 16-node cluster in a single box complete with a dedicated head-node loaded with applications specific to a particular vertical. Think the Apple Workgroup Cluster for Bioinformatics.

I think this is a reasonable place for them to focus: there certainly is a market for this pre-packaged hardware / software solution.

eScience / Patrick Hogan

Robert W. Anderson — Fri, 07 Oct 2005 19:31:00 GMT

Patrick Hogan showed off the work his group is doing at NASA: World Wind. Think Virtual Earth and Google Earth packaged in a slick smart client layered with scientific data fetched from data warehouses from around the world. Check it out here: http://worldwind.arc.nasa.gov/download.html

He really wowed the audience with:

Multiple satellite views
USGS topographic maps
Place information (the kind of stuff you see on a political map)
Elevations in relief
Seismic / weather data

And more, all in a zoomable 3d view off the earth.

Plus a version of it is available for Windows Mobile!

Probably the most impressive part was that it was all done in one year by one programmer straight out of college. That says a lot about the productivity one can achieve with the Microsoft tools (it is all .NET and DirectX).

Apparently the CEO of Google praised the NASA group for the accomplishment. It took them 30 programmer years to achieve the same result.

eScience / Dan Werthimer

Robert W. Anderson — Thu, 06 Oct 2005 16:17:00 GMT

Dan Werthimer talked a bit about SETI@Home and BOINC.

This is a very well-known project run from my alma-mater.

He talked about several ways over time that people have tried (or suggested trying) to tell extra-terrestrial life that we are here. None of these were funded. He makes the point that there was no funding until Voyager, which he called "Porno in space". If you don't get this, look for the plaque on the side of Voyager (or recall V'ger).

His whole talk was pretty amusing. A mix of SETI science and humor.

eScience / David Lifka

Robert W. Anderson — Thu, 06 Oct 2005 13:34:00 GMT

David Lifka leads the Cornell Theory Center (i.e., the CTC). If you are (still) questioning the viability of Windows for cluster computing, you should check out the work they have done. They offer a free workshop – I went last year. Interesting content plus you get to check out their visualization cave!

He spoke about what they do there and more of the story: Windows, IIS, SQL Server, Web services, Visual Studio, . . . All together this makes an excellent platform for computing.

David brought up the problem with network load balancing (NLB) applied to distributed computing. It is transaction based. If transactions/second go down, then the server is loaded higher. Long running tasks (e.g., a large BLAST search) will have a low t/s resulting in more tasks assigned to that machine. Then you get a grinding halt. This approach works well for short transaction-based requests (think Web-sites) and poorly for computation-intense applications.

He brings this up for a good reason: many people think that application distribution can be achieved by simply implementing NLB across a set of Web servers. The answer, as is often the case, is: "well, under some circumstances, maybe."

He also uses an alternate to the term “grid”: “Heterogeneous Wide Area Distributed Computing”. Meaning, a Grid that isn't necessarily based on Globus.

eScience / Marty Humphrey

Robert W. Anderson — Thu, 06 Oct 2005 13:11:00 GMT

Marty Humphrey directs the group that builds WSRF.NET.

WSRF.NET is both an alternative to the Globus reference implementation (built on Java / Tomcat, . . .) and a toolkit for developing WSRF-compatible Web services using ASP.NET. See this paper for a good overview of WSRF/WSRF.NET (note that isn't the purpose of this paper, but it does it just the same).

The grid-standards community owe these UV guys a big thanks for taking the standards and building an alternate implementation – further validating the standards work. This proves (again) that the standard can be implemented, that it is functional, that it works and is not biased towards a specific language, application server, or OS. Of course, the implementation also provides feedback into future standards efforts.

It also validates a basic tenet of our approach: .NET is a terrific platform for distributed computing.

A great challenge for this industry is to take the excellent work done by computer scientists (and other domain experts) to solve these greatly hard problems and distill them down into something that is accessible to/by "normal" people to solve their real problems.

The standards are definitely helping make this a reality, but the tools built on these standards will really make the difference.

eScience / Jim Gray

Robert W. Anderson — Thu, 06 Oct 2005 11:27:00 GMT

Here at the Microsoft eScience event – my special thanks to the organizers. Apparently I wasn't registered! No problem – they will get me a badge during the break.

Jim Gray kicked off the event with a historical view of science – that we have now gone into the era of data analysis. This is clearly the case in many different scientific domains: the sheer amount of measurements being taken, collected, and stored is phenomenal. Analyzing this data.

He described how on a 15-20M astronomy project (I think he was talking about the Sloan Digital Sky Survey (http://Skyserver.sdss.org), up to a 1/2 of that is spent on software development. This doesn't count the programming done by the astronomers! Certainly, some of this software development includes that related to the science, but most of it is the development of an entire distributed computing system to support this one project.

His point: the tools aren't good enough yet. I agree that this is true across the grid/distributed computing domain. There is too much computer science and IT skill necessary to build and maintain these systems. We are trying to make this easier in our own domain.

I hadn't heard Jim's pragmatic side: He talked a bit about how we computer scientists need to engage the domain experts in science:

find someone who is desperate (who has a lot of need)
don't try to go deep into go from working to working (i.e., deliver as you go)
give 100x value
give 20% of a solution (the other 80% will take 10 years)
Don't make your project's success depend on another project's success.

Replace “domain experts in science“ with customer, and Jim Gray sounds like a sales manager! Maybe we can get him hawking the Digipede Network!