Joe Mayo

LINQ to Twitter Maintenance Feedback

Joe Mayo — Sun, 16 Jun 2013 20:43:42 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/06/16/linq-to-twitter-maintenance-feedback.aspx

It’s always fun to receive positive feedback on your work. If you receive a sufficient amount of positive feedback, you know you’re doing something right. Sometimes, people provide negative feedback too. There are a couple ways to handle it: come back fighting or engage for clarification. The way you handle the negative feedback depends on what your goals are.

Feedback Approaches

If you know the feedback is incorrect and you need to promote your idea or product, you might want to come back fighting. The feedback might just be comments by a troll or competitor wanting to spread FUD. However, this could be the totally wrong approach if you misjudge the source and intentions of the feedback.

In a lot of cases, feedback is a golden opportunity. Sometimes, a problem exists that you either don’t know about or don’t realize the true impact of the problem. If you decide to come back fighting, you might loose the opportunity to learn something new. However, if you engage the person providing the feedback, looking for clarification, you might learn something very important. Negative feedback and it’s clarification can lead to the collection of useful and actionable data.

In my case, something that prompted this blog post, I noticed someone who tweeted a negative comment about LINQ to Twitter. Normally, any less than stellar comments are usually from folks that need help – so I help if I can. This was different. I was like “Don’t use LINQ to Twitter”. This is an open source project, the comment didn’t come from a competing project, and sounded more like an expression of frustration. So I engaged. Not only did the person respond, but I got some decent quality feedback. What’s also interesting is a couple other side conversations sprouted on the subject, which gave me more useful data.

LINQ to Twitter Thread

Actions

Essentially, this particular issue centered around maintenance. There are actually several sub-issues at play here: dependencies, error handling, debugging, and visibility. I’ll describe each one and my interpretation.

Dependencies

Dependencies are where a library has references to other libraries. This means that when you build your application, you need DLLs for the entire dependency graph for your application. There are several potential problems with this that include more libraries for configuration management, potential versioning mismatches, and lack of cross-platform support.

In the early days of LINQ to Twitter, I allowed developers to contribute and add dependencies, but it became very problematic (for reasons stated). It was like a ball and chain that kept me from moving forward. So, I refactored and pulled other open-source into my project to eliminate external dependencies. This lets me fix the code in my project without relying on someone else to upgrade or fix their DLL. The motivation for this was from early negative feedback that translated as important data and acted on it. Today, LINQ to Twitter has zero dependencies.

Note: Rejecting good code from community members who worked hard to make your project better is a painful experience in itself. I have to point out that any contribution was not in vain because they had a positive influence on my subsequent refactoring that resulted in a better developer experience.

Error Handling

Error handling has been a problem in the past. I have this combination of supporting both synchronous and asynchronous (APM) processing that can be complex at times. Within the last 6 months, I did a fair amount of refactoring to detect errors and process them properly. I also refactored TwitterQueryException so it includes important data from Twitter. During this refactoring, I’ve made breaking changes that I felt would improve the development experience (small things like renaming a callback property to Exception, rather than Error). I think the async error handling is much better than it was a year ago.

For all the work I’ve done, there is more to do. I think that a combination of more error handling support, e.g. improving semantics, and education through documentation and samples will improve the error handling story. Because of what I’ve done so far, it isn’t bad, but I see opportunities for improvement.

Debugging

Debugging can be painful. Here’s why: you have multiple layers of technology to navigate and figure out where the real problem is – Twitter API, Security, HTTP, LINQ to Twitter, and application. You can probably add your own nuances to that list, but the point is that debugging in this environment can be complex.

I think that my plans for error handling will contribute to making the debugging process easier. However, there’s more I can do in the way of documentation and guidance. Some of the questions to be answered revolve around when something goes wrong, how does the developer figure out that there is a problem, what the problem is, and what to do about it.

One example that has gone a long way to helping LINQ to Twitter developers is the 401 FAQ. A 401 Unauthorized is the error that the Twitter API returns when a use isn’t able to authenticate and is one of the most difficult problems faced by LINQ to Twitter developers. What I did was read guidance from Twitter and collect techniques from my own development and actions helping other developers to compile an extensive list of reasons for the 401 and ways to fix the problem. At one time, over half of the questions I answered in the forums were to help solve 401 issues. After publishing the 401 FAQ, I rarely get a 401 question and it’s because the person didn’t know about the FAQ. If the person is too lazy to read the FAQ, that’s not my issue, but the results in support issues have been dramatic.

I think debugging can benefit from the education and documentation approach, but I’m always open to suggestions on whatever else I can do.

Visibility

Visibility is a nuance of the error handling/debugging discussion but is deeply rooted in comfort and control. The questions to ask in this area are what is happening as my code runs and how testable is the code.

In support of these areas, LINQ to Twitter does have logging and TwitterContext properties that help see what’s happening on requests. The logging functionality allows any developer to connect a TextWriter to the Log property of TwitterContext to see what’s happening. Further, TwitterContext has a Headers property to see the headers Twitter returns and a RawResults property to show the Json string Twitter returns. From a testing perspective, I’ve been able to write hundreds of unit tests, over 600 when this post is published, and growing.

If you write your own library, you have full control over all of these aspects. The tradeoff here is that while you have access to the LINQ to Twitter source code and modify it for all the visibility, LINQ to Twitter *will* change (which is good) and you will have to figure out how to merge that with your changes (which is hard). The fact is that this is a limitation of any 3rd party library, not just LINQ to Twitter. So, it’s a design decision where the tradeoff is between control and productivity.

That said, there are things I can do with LINQ to Twitter to make the visibility story more compelling. I think there are opportunities to improve diagnostics. This would be a ton of work because it would need to provide multi-level logging that can be tuned for production and support any logging provider you want to attach. I’ve considered approaches such as how the new Semantic Logging application block connects to Windows Error Reporting as a potential target. Whatever I do would need to be extensible without creating native external dependencies. e.g. how many 3rd party libraries force a dependency on a logging framework that you don’t use. So, this won’t be an easy feat, but I believe it can be part of the roadmap.

I think that a lot of developers are unaware of existing visibility features, so the first step would be to provide more documentation and guidance. My thought are that this would lead to more feedback that will help improve this area.

Summary

Recent feedback highlights some of items that are important to LINQ to Twitter developers, such as dependencies, error handling, debugging, and visibility. I know that there are maintenance issues that have been problems for LINQ to Twitter developers in the past. I’ve done a lot of work in this area, such as improving error handling, adding visibility features, and providing extensive API documentation. That said, there is more to be done to make LINQ to Twitter the best Twitter API experience available for .NET developers and I welcome anyone’s thoughts on what I’ve written here or new improvements.

@JoeMayo

Goodbye Twitter API v1.0

Joe Mayo — Wed, 12 Jun 2013 15:22:40 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/06/12/goodbye-twitter-api-v1.0.aspx

Hello Twitter API v1.1! Actually, Twitter API v1.1 has been around since September 2012. That’s when Twitter deprecated v1.0. Yesterday, Twitter shut down v1.0. You can read more details of yesterday’s events in on the Twitter API blog at API v1 Retirement is Complete - Use API v1.1.

Soon after Twitter API v1.1 was available, LINQ to Twitter upgraded to Twitter API v1.1 and changed its version from 2.0 to 2.1. You can download the latest version of LINQ to Twitter, which is v2.1.06 as I write this, and have full support for Twitter API v1.1.

Here’s a summary of important changes:

Rate Limits apply per API, as opposed to every API call. Statistically, this works out to a better rate limit overall, but it also means that you might need to adjust your design on how it scales to match the new policies. You can get rate limits with a HelpType.RateLimits query.
Json is now the only available response format. Generally, this doesn’t change your code because LINQ to Twitter deserializes most queries into entities. However, if you use Streaming, Raw commands/queries, or reading the TwitterContext.RawResults property, and you were working with XML, you’ll need to deserialize Json instead. The good news is that your application performs and scales better because queries and commands execute more quickly and use less bandwidth.
You must authenticate for all API calls. I know this is painful if you’ve used Search or another unauthenticated API. The good news is that LINQ to Twitter has extensive support for all Twitter OAuth options, so you don’t have to write your own OAuth code.
Search merged with the main API and is more consistent. LINQ to Twitter’s Search option also changed extensively. Primarily, search results and Status entities, which are the same type as Status query results. Though it requires a code change, it adds consistency to the API and improves the development experience.

As the Twitter API matures and moves forward, so will LINQ to Twitter. Thanks to all the people who have contributed code, submitted pull requests, and made constructive changes over the years.

@JoeMayo

Released LINQ to Twitter v2.1.06

Joe Mayo — Mon, 10 Jun 2013 15:27:59 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/06/10/released-linq-to-twitter-v2.1.06.aspx

Released LINQ to Twitter v2.1.06: http://bit.ly/11CHNdb.

Here are a few highlights of this release:

Updated MVC sample (in source code) to MVC 4.0.
Added support for Symbol Entities.
Added StallWarning param to Streaming APIs.
Caught up adding all missing parameters for all APIs (could be a few breaking changes with modified overloads – e.g. trimUser for UpdateStatus).
Changed cache_age on StatusType.OEmbed response to integer to fix unannounced breaking change in Twitter API.
Deprecated old/unused TwitterContext constructors.
Added constructor for SL/WP/Win8 apps to support custom IWebRequestCreate, allowing 3rd party providers. Also enables GZip support on WP7.1/8 through 3rd party providers, such as SharpGIS.WebRequestCreator.GZip.
Refactored TwitterContext arrangements in Unit Tests.

@JoeMayo

Windows 8 for Developers Online Camp

Joe Mayo — Mon, 20 May 2013 23:01:52 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/05/20/windows-8-for-developers-online-camp.aspx

On May 21st, Ziff Davis is hosting an eSeminar, Windows 8 for Developers Online Camp. Speakers include Greg Levenhagen, @GregLevenhagen, and myself. Here’s the agenda:

- Building a Windows 8 App in 15 minutes or less.
- Where is Microsoft headed, how it differs from others (Apple/Google), and why it makes sense.
- Win8 Development versus Android, iOS, and Pre-Win8 Dev
- What’s Different in Building for Windows 8?
- Understanding the potential to Make Money With Windows 8
- What it takes to submit an app (It’s not that hard)
- What help is available? (There is some very good, free help and tools available)
- Next Steps / Getting Started

@JoeMayo

Webinar: Async Programming in C#

Joe Mayo — Tue, 07 May 2013 16:40:52 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/05/07/webinar-async-programming-in-c.aspx

Tomorrow, I’m doing a live webinar with Learn Now on Async Programming in C#. More details here: http://bit.ly/17JT792.

@JoeMayo

Global Windows Azure Bootcamp–4/27/2013

Joe Mayo — Tue, 23 Apr 2013 18:32:36 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/04/23/global-windows-azure-bootcampndash4272013.aspx

Denver Windows Azure Bootcamp

On April 27th, 2013, join us at a Windows Azure Bootcamp in Denver, just one of the 95+ locations hosting this massive event. This free one day deep dive class will get you up to speed on developing for Windows Azure. The class includes a trainer with deep real world experience with Windows Azure, as well as a series of labs so you can practice what you just learned.

Register for the event today at https://gwabdenver.eventday.com. There are only 100 seats available so act fast!

This free training event will be hosted at Microsoft - Denver Tech Center - 7595 Technology Way Suite 400. Want more information? Check out http://globalwindowsazure.azurewebsites.net
and follow hashtag #GlobalWindowsAzure.

Not enough reasons to sign up? How about awesome door prizes:

JetBrains	- ONE (1) winner at each event gets a license of one of their products. The winner may choose from ReSharper, dotTrace, dotCover, PhpStorm, PyCharm, IntelliJ IDEA, AppCode, WebStorm.
PluralSight	- ONE (1) winner at each event will get a 1 Year Annual Subscription. - ALL Attendees will get a 7 Day training pass via a unique code.
Telerik	- ONE (1) winner at each event will get a DevCraft Complete (all their products).
Cerebrata	- ALL Attendees will get a license to the new Azure Management Studio product. There will be an email address they can use to request it.
Blue Syntax	- ONE (1) winner at each event can win a Cloud Backup Advanced Edition license.
MyGet	- ONE (1) winner at each event can win a 1 year Starter Subscription - ALL Attendees will get a 1 month starter subscription redeemable via an online sign up form.
Cloud Berry	- FIVE (5) winners at each event can win a Cloud Berry Explorer License - FIVE (5) winners at each event can win a Cloud Berry Drive License
AzureWatch	- ALL Attendees will get a free 30 day unlimited license, redeemable via a unique code.
Inner Workings	- ALL Attendees will get 90 days Free Training, redeemable online.
Zudio	- ALL Attendees will get a 3 month free trial redeemable via a unique code. - ALL Organizers will get a 1 year subscription.

Bonus points to take part in an experiment to set a record for the largest "Shared Workload" run by a community in a public Cloud. The general idea is to launch a ready-to-go Windows Azure deployment package in every event location around the world. The workload is a video rendering farm which will render videos from a set of depth frame images - a very CPU intensive workload that takes a lot of time to process. Progress will be measured by country and location and each participant will be able to follow this progress.

You know you want in. Register today!

Twitter API v1.0 Deprecation and Blackout Schedule

Joe Mayo — Tue, 16 Apr 2013 19:55:14 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/04/16/twitter-api-v1.0-deprecation-and-blackout-schedule.aspx

Twitter has officially announced API v1 Retirement: Final Dates. The final date will be May 7th 2013. So, if you still have code running on Twitter API v1.0, you’ll need to upgrade to Twitter API v1.1. LINQ to Twitter has supported Twitter API v1.1 for months and you can download it from codeplex.com or NuGet.

To help people understand the impact, Twitter will be having a blackout period starting at 4:00pm PDT today. This is an opportunity to verify that all of your endpoints have been re-written to use Twitter API v1.1. If you’re using the latest version of LINQ to Twitter, you shouldn’t have a problem. This will mostly affect folks who have written their own plumbing code or are working with an older library that only supports v1.0.

@JoeMayo

Released LINQ to Twitter v2.1.05

Joe Mayo — Mon, 08 Apr 2013 15:36:04 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/04/08/released-linq-to-twitter-v2.1.05.aspx

Today, I released LINQ to Twitter v2.1.05, which is available via NuGet or the LINQ to Twitter Download page.

This was a critical release because Twitter changed their OAuth handling logic, as explained in the discussion, “oauth/access_token Stopped working today” on their developer site. In particular, this change requires all access_token endpoint calls to include the oauth_verifier for out-of-band, OOB, authorization. What this means is that any applications using PIN authorization with LINQ to Twitter will break or are already broken.

Impact-wise, I wasn’t seeing a problem with Console applications, but it did break Windows Phone apps. What this tells me is that the change isn’t fully implemented across the entire Twitter API, yet. However, if you’re using PIN authorization on any application type, you’ll want to upgrade to LINQ to Twitter v2.1.05 or later because this change is likely to eventually affect everyone using PIN authorization.

@JoeMayo

Truth, Equality, and the Meaning of Null

Joe Mayo — Thu, 28 Mar 2013 23:04:49 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/03/28/truth-equality-and-the-meaning-of-null.aspx

There’s been some discussion recently about whether it’s proper for a language to support null values or not. My guess is that the people having these discussions are a whole lot smarter than myself, so it’s probably best to defer to their judgment on any matters that might conflict with whatever I have to say. That said, there might be semantic differences in what is being referred to when discussing null. Perhaps one discussion regards the logical operation or state of code, whereas my perspective in this post is on data.

There was a time when I was fascinated by subjects such as truth, equality, and the meaning of null. Especially when thinking about data, null is an interesting subject. In a relational context, null means the absence of a value. When writing code, the meaning of null is determined by your domain. You typically won’t have a specific requirement, use case, or story that tells you how to work with a null value when first writing the code and your logic is often inferred. It’s often when a bug appears in your code that a formal requirement might be specified, but I say “might” loosely because handling of null is a technical requirement and the bug would probably be stated as the functional requirement to satisfy. This is why null values are often so aggravating because they bubble up later in the application lifecycle where they’re more painful to deal with. This seemingly mundane topic of existence appears to be more important than one may think.

Today I encountered a null in the wild and all the related discussion compelled me to investigate. I was adding a new field to the LINQ to Twitter Status entity named FavoriteCount. This relates to the Twitter Tweets favorite_count field. What caught my eye is the definition of favorite_count identifies the field as Nullable. I know what Nullable means in relational and .NET parlance, but Twitter is a different world. So, I went to the Twitter developer’s site and asked, “What is the Meaning of null favorite_count?”. I’ll quote an important part of @episod’s answer:

The state of null is usually reached because it was not possible at the time of servicing the request to provide a value in that field -- an underlying system could have taken too long to respond, or a null value got cached at some other point in time when that happened.

One could say that this is the same thing as the “absence of a value” definition I mentioned earlier, but there’s an important nuance here in the context of the data source and nature of the application. The “absence of a value” comes from a relational ACID transaction perspective where you have what you’re going to get. However, Twitter is a massively scalable system that uses a NoSQL data source. From the answer and the scalable NoSQL perspective, the true value of favorite_count may not be null at the point in time that you’re looking at it, but you don’t know for sure. I believe the proper term for this phenomenon is Eventual Consistency. There’s some hair splitting that can be done here, but I’m speaking in general terms.

In my initial question about the favorite_count field, I wondered whether it was safe to assume that a null value meant 0. Making a blanket assumption like that could certainly cause some trouble and it would be unwise for me to make that assumption on behalf of the developers using LINQ to Twitter. e.g. What if an application would rather monitor a tweet for N minutes to see what the real favorite_count was after it initially appeared as null? There are variations of this theme, but it’s a situation where I would want to expose functionality that was meaningful and allowed the developer to know the true state of an entity.

To understand what null is and put it into context is an important endeavor, especially in keeping customers from hating you and reducing project lifecycle costs. However, you still need language support unless you regress to kludges of earlier platforms. In the first version of .NET you could encounter code that either treated a value as a string or used some other mechanism, such as an alternate value to represent null values. I can’t count the number of different representations of DateTime value system I encountered in those days and the headaches of accepting defaults of different databases with null values. Of course, ADO.NET has the DBNull type and there were 3rd party libraries that special types that allowed you to represent nullable value types. .NET 2.0 introduced the Nullable<T> type and added special language support to C#, where you could add a Nullable suffix, such as DateTime?. These days, it’s easy to represent null data in a consistent way with both .NET Framework and language support.

Null data is a reality. Old databases often have null values. These old databases also have applications with logic that relies on the null data. The cost/benefit of a re-write can result in leaving things as is. There are also current scenarios where null makes sense, such as when having Created and Modified fields on a record where it doesn’t make sense to provide a value for Modified because the record hasn’t changed since it’s initial value. One could argue that Modified should have the Created time and application logic would detect this fact or perhaps another design would be more appropriate, but the fact is that you’ll find people with differing opinions on what the data design should be in any given situation and sometimes the argument for null will win. I’m not one to fall on my sword for a null value when it’s possible to write code to deal with it. Another perspective, as mentioned earlier is the NoSQL world with eventually consistent data. Regardless of what arguments you have on that, the reality is that you will be working with APIs, public data, and situations where the need for scalability results in null data making sense or being the best choice to meet requirements.

So, I’m not trying to say that someone else’s opinion of null is wrong. However, it seems like there are nuances to consider before declaring that “Null is Evil!” Our industry is plagued with never-ending battles of who’s platform or language is best and the true answer can sometimes be found in-between. While Scientists debate What is Nothing, VB developers can relate and C# developers can say that’s equivalent to null. Or is nothing really something to care about?

@JoeMayo

IOException on WebRequest.Create in .NET 4.5

Joe Mayo — Thu, 28 Mar 2013 03:59:54 GMT

Originally posted on: http://geekswithblogs.net/WinAZ/archive/2013/03/27/ioexception-on-webrequest.create-in-.net-4.5.aspx

I was testing .NET 4.5 compatibility in LINQ to Twitter when I encountered an IOException with the message: “The specified registry key does not exist.” This occurred during a call to System.Net.WebRequest.Create().

This is something that happened after a Windows update, kb2745030. Since this is a security issue, you’ll want to read Microsoft’s article and make your own determination on what to do. Someone has already reported the issue on Connect with a work-around similar to the Microsoft Kb article, WebRequest.Create eats an IOException on the first call.

@JoeMayo