Geeks With Blogs

News This is the *old* blog. The new one is at blog.sixeyed.com
Elton Stoneman
This is the *old* blog. The new one is at blog.sixeyed.com

Apache Storm went to GA on HDInsight yesterday, and the latest Azure SDK has Visual Studio integration for creating Storm applications.

You can select the Storm Application project template:

image


And that creates you a basic topology with a simple spout and bolt. You can publish that new project to Azure from Visual Studio (right-click the project in Solution Explorer and select Submit to Storm on HDInsight), and verify it's all working correctly from the Storm Topologies view:

image


But there's a gotcha if you follow the typical .NET naming conventions - Storm doesn't let you have full-stops in a topology name, and by default the template uses the project name as the topology name. So if you create project StormApplication1 the topology name is valid and all is well. If you create My.EventProcessor.RealTime.Storm then you'll get a strange error in the output when you try to submit the app:

Error: Could not find or load main class "My.EventProcessor.RealTime.Storm"

If you play around with the name you use in TopologyBuilder, you can get this error instead - which tells you the restricted characters:

Exception in thread "main" InvalidTopologyException(msg:Topology name cannot contain any of the following: #{"." "/" ":" "\\"})
    at backtype.storm.generated.Nimbus$submitTopology_result.read(Nimbus.java:2737)

Behind the scenes, VS generates a .spec file from the configuration you build up in TopologyBuilder:

{:name "StormApplication1"
 :topology
  (nontx-topology "StormApplication1"
 
    {
      "Spout"
      (spout-spec
      (scp-spout
        {
          "plugin.name" "SCPHost.exe"
          "plugin.args" ["StormApplication1.dll" "StormApplication1.Spout" "Get"]
          "output.schema" {"default" ["count"]}
        })
    :p 1)
} 

That tells the runtime (SCPHost.exe) which assemblies to load for spouts and bolts, as well as the schema config. But for the generated spec to work, the assembly name and topology name need to match, and they need to be free of full stops.

So when you create the project My.EventProcessor.RealTime.Storm, first change the assembly name in the project properties to MyEventProcessorRealTimeStorm and then change topology name the builder uses to be the same:

  TopologyBuilder topologyBuilder = new TopologyBuilder("MyEventProcessorRealTimeStorm");

Now it will publish correctly, and you can still use the full project name (with full-stops) for your namespace.

Posted on Thursday, February 19, 2015 4:21 PM Annoying problems , Azure , BigData | Back to top


Comments on this post: Submitting HDInsight Storm Applications from Visual Studio - Gotcha!

No comments posted yet.
Your comment:
 (will show your gravatar)


Copyright © Elton Stoneman | Powered by: GeeksWithBlogs.net