QA checklist to run before installing a new SlickEdit build?

[Kaiser]

I love SlickEdit, but the product has issues.  QA is a joke, and every release it seems that more features are added while older features which don't work quite right are never fixed.  Perhaps this is due to core talent having left the company or the "cowboy" nature of development.

I, for one, am tired of upgrading only to find some glaring flaw.  For some reason, other editors such as Emacs, Vim, and Eclipse are more stable.  Of course, they don't purport to do as much as SlickEdit does, but I don't think that accounts for most of the difference in quality.  Perhaps we, as a community, could come up with a basic QA checklist which could be run before each build is released?  Perhaps it could be then automated using Slick-C.  This shouldn't be the job of the community but if SlickEdit developers themselves are not professional enough to test their own code, perhaps we can.

[evanratt]

Out of curiosity, what glaring flaw are you referring to? The newly-released build went through 4 beta versions, and a new build is due in the next few days that fixes some late-noticed bugs. I know that any bugs I reported during the beta were fixed by the time the final version was released. I'm not going to claim that I've never had issues with SlickEdit working the way I wanted or expected, but both the community and the developers have always been very responsive on these boards and through support email, and have generally helped to fix my issues or figure out how to make the program work the way I want.

I'm not saying you don't have a point, but without explaining what you feel is lacking or not working, it doesn't seem quite fair to make a blanket statement like that. Yes, in previous releases, I've found that certain aspects of the program didn't quite work right, but as I said, the SlickEdit developers were always very helpful in resolving my issues. From my viewpoint, this newest release adds some very nice functionality (I'm really enjoying Symbol Coloring), and it also seems more solid than previous releases, so far (at least, in the areas of the program that I use regularly).

[buggyfunbunny]

Too harsh, by half.  SE tries to be all things to all coders, and at that no one can succeed.  SE comes closer than its extinct competitors.  The problem with SE is that it supports so many languages and editor definitions, that the cartesian product is such a big number (and likely each element is just one SE customer) that getting 90% coverage of what a specialized IDE/editor would do for each is impossible.  I, for one, have always wanted multiple embedded languages supported because that is what java/jsp/servlet coding requires.  Hasn't happened.  Now that I do Python/Pylons/TurboGears, my whining will change, I'm sure.  SE has edged toward java IDE land at times, but never quite all the way.  I'd rather it didn't, even when java was mostly what I did with it, because I knew that I wouldn't always be doing java.  Raw meat chewing coders still build their own emacs environments, but my fingers only know vi(m).

[ScottW, VP of Dev]

I certainly understand Kaiser's perspective. Putting together a quality release is important to us, and it's very challenging. I think we were doing fine until we made what seemed like a minor change after beta 4 to eliminate the "Maximize first window" option. We wanted to drive that setting off of the state of the last window to be closed. Testing after that change didn't turn up any issues. But a problem was quickly found when the release went out: generating a list of references and clicking on them caused the File tabs and the File tool window to get messed up. My apologies to anyone who was inconvenienced by this. We take a great deal of pride in our work on SlickEdit, and we are all very upset that this happened.

The problem wasn't found under test because you had to click on several references before the problem manifested itself. In my testing, it was around the 10th reference clicked. This highlights the difference between use and testing. During testing, how many clicks is enough? Also, our customers use SlickEdit in ways that are too numerous and too creative for us to replicate in testing.

Because Emacs, Vim, and Eclipse are open source, there are a lot of users who work with the intermediate builds. You could say that they are always in beta. For us, we run the beta once we've gotten enough changes to warrant the time from our customers. The mistake we made was to put something in after the last beta--though the change did not seem risky at the time.

As for features versus bug fixes, that is the age-old tension in product management. Existing customers primarily want bug fixes. New features are important to attract new customers. We try to provide a balance of both.

I'd be interested in the list of old features that you think need fixing. I am aware of many that need work, but I'm curious to know which ones you are refering to. Our x.0.1 releases typically focus on bug fixes with very few new features. So, I'm setting priorities for those right now.

Please give us any additional feedback you have on quality and the things you'd like to see changed in SlickEdit. While we may not always be able to fix something quickly, we do take your input seriously.

[chrisant]

I see that two replies came in while I was typing, but I'll post anyway:

If the core goal is to improve QA on the SlickEdit product, it seems counterproductive to offer insults (e.g. "unprofessional", "joke").  Insults distract from pragmatic points.

I can understand frustration at bugs, especially if there are bugs that persist over multiple releases or if there are patterns of regressions.  Feedback isn't actionable if it doesn't cite specific examples of bugs and/or patterns.

Peeling aside the insults and rhetoric to find a diamond in the rough, I do see a potentially interesting suggestion in the original post.  I'll try to rephrase it in a more approachable manner:



This seems less relevant after ScottW's post and buggyfunbunny's post, but I'll include it at least for illustrative purposes if not its stated purpose:

@SlickTeam:  I imagine that with as many features SE has and the sometimes complex interactions between the features, it may be difficult to know which features (and in what configurations) are relied upon most heavily by customers, and where to emphasize QA investment.  Would it be helpful to the SlickTeam for the user community to make a thread where users can suggest areas or specific tests that are of particular interest to their usage patterns?  I can imagine the actionable nature of the feedback would vary, as would the relative priority or cost/benefit ratio, and I acknowledge that basing a QA strategy around a statistically insignificant sample set isn't a good idea.  If that kind of feedback would have any value to the SlickTeam, though, I'm sure the user community would be happy to contribute suggestions over time.

[jimlangrunner]

...The mistake we made was to put something in after the last beta--though the change did not seem risky at the time...

Been there.  Done that.  Went running.

I'm sure that many of us have made "safe" changes to code about to go into production.  The fact that you pulled it right away with an explanation speaks volumes about your pride and professionalism. I may not have a huge interest in this release, but I do have a tremendous interest in your commitment to me as a customer and in the features yet to come that will be of greater interest to me.

I think @chrisant has a good idea. I do not know how well it will work, but it is certainly worth trying.

[Tim Kemp]

Here's a suggestion that I'm sure will be met with universal scorn.   

How about a lightweight profiler built into SE that keeps track of how often different features are used.  Then once a month or once every so many hours of use, a report could be automatically sent back to a server at SlickEdit.  Of course you'd need to be able to disable it for security or paranoia purposes, but it ought to be enabled by default when SE is installed with a pop-up the first time it's run allowing you to turn it off.

With this information SE could know where to focus their efforts in new development, bug fixes, regression testing and I'm sure many other things I haven't thought of.

[jimlangrunner]

Here's a suggestion that I'm sure will be met with universal scorn.   

How about a lightweight profiler built into SE that keeps track of how often different features are used.  Then once a month or once every so many hours of use, a report could be automatically sent back to a server at SlickEdit.  Of course you'd need to be able to disable it for security or paranoia purposes, but it ought to be enabled by default when SE is installed with a pop-up the first time it's run allowing you to turn it off.

With this information SE could know where to focus their efforts in new development, bug fixes, regression testing and I'm sure many other things I haven't thought of.

Not universal scorn.  While there are certainly software companies (and others) that I do not trust, I would be willing to give SE usage information.  Why?  Because customer usage is the best way to determine the value of a feature and what a user really needs, even if they won't ask for it.

With the product we develop, we asked a user permission to record a desktop session to help track down some bugs.  (We couldn't reproduce them in testing).  She consented, and we got 3 hours of usage, and found the steps to reproduce the bug and get it fixed. We also watched the user switch to another of our applications and check on information we never imagined they'd need.  A couple of builds later we had a window into the other app and made it a point to let everyone know that we put it in because we saw that a customer had a "need".  It was so cheap (in terms of time taken) that it made no sense not to put it in.  Good PR for us and hours saved every week for the customer. She wouldn't have asked for it because it seemed so wasteful (to her) when there's a perfectly good way to get the information already.

I'll disagree with "on by default" because Google & Yahoo toolbars are 'way to often on by default, and if I rush through an install, I have to go back & take them out.  Let the user CHOOSE to help out, and you'll get more useful information with fewer folks griping about not seeing it in the rush to install.  (That's my $0.02).

In short, I think it's a pretty good idea. 
Jim.

[ScottW, VP of Dev]

Tim's idea is one I've often talked about. I wouldn't enable it by default, though. I'd want this to be something you have to turn on yourself. I would have it gather the following information:
1) What programming languages are you editing? This would need to have some kind of count so that we can tell which languages get the most useage. So, if you primarily work in Java but you ocaisionally open a Perl file, I would want something that conveys that.
2) What Tool Windows do you use? Again, I'd like something more than just which ones are docked. I'd like to know which ones you actually use and how much.
3) What other features are you using? The most basic version of this could be to monitor which commands SlickEdit executes. These don't have to be commands typed into the command line. Commands underly all actions taken through the UI. Though it would be useful to know if you launched the command from with a keystroke, a menu, an icon, or the command line.

All of this would be completely anonymous with no source code sent as part of the information. Also, it would not include anything that could be used to identify the user.

[Phil Barila]

Here's a suggestion that I'm sure will be met with universal scorn.   

How about a lightweight profiler built into SE that keeps track of how often different features are used.  Then once a month or once every so many hours of use, a report could be automatically sent back to a server at SlickEdit.  Of course you'd need to be able to disable it for security or paranoia purposes, but it ought to be enabled by default when SE is installed with a pop-up the first time it's run allowing you to turn it off.

With this information SE could know where to focus their efforts in new development, bug fixes, regression testing and I'm sure many other things I haven't thought of.

Far from scorning it, I like it a lot.  But then, I allow various apps to call home with usage stats, something the privacy police swear is giving them everything they need to steal my soul.  

[Ryan]

While the motivation is dead on, I don't know about this  .  I feel like we would experience the same outrage (or at least similar) that we were subjected to during the great trusted storage debacle of version 13.

- Ryan

[jimlangrunner]

While the motivation is dead on, I don't know about this  .  I feel like we would experience the same outrage (or at least similar) that we were subjected to during the great trusted storage debacle of version 13.

- Ryan

I'd have to disagree.  If it's upfront and opt-in, there is nothing to complain about.  If it's a downloadable slick-c macro or stand-alone executable, even better. Make it clear that noting goes out without the customer's approval.

Look, I hate statistics gathering.  Mostly because it's used as a means of profiling me to get more of my money.  I like to make that hard.  But now I'm paying $60/yr (something like that) so that I can stay current.  Now it's on the other foot. I WANT you to profile me to serve me better.  I want you to profile as many of us as possible so you can keep us happy.  Keep us happy and your revenue stream stays in place, you stay in business, and you don't pull a Borland (CodeWright) on me.  I'm in.  Play it right and you'll get enough buy-in to make the statistics you gather worthwhile & meaningful.

Just be sure to leave Macrovision out of it.  I have nothing for them.  Their opinion of the end-user is a big part of the reason there was such an uproar.  (You let them do WHAT to our boot sector?)

Just another $0.02 from Jim

[Ryan]

I hear you...I just talked to Scott about this briefly and I think we are on the same page about this type of feature.  My concerns were mainly anything automatically "phoning home", and additionally not getting enough data because we couldn't get enough people running whatever it is we distribute. 

I think if we could have some type of feature that works like the Slick-C profiling does would be good...maybe we could just have users send in their data into a huge repository that we have here that we can data mine.  Also, Scott said maybe we could distribute it with the product and just allow users to turn it on or off in the Quick Start.  I think that would give this more exposure, because the majority of our users are not on these forums.  Just some thoughts...

- Ryan

[mako]

Tim's idea is one I've often talked about. I wouldn't enable it by default, though. I'd want this to be something you have to turn on yourself. I would have it gather the following information:
1) What programming languages are you editing? This would need to have some kind of count so that we can tell which languages get the most useage. So, if you primarily work in Java but you ocaisionally open a Perl file, I would want something that conveys that.
2) What Tool Windows do you use? Again, I'd like something more than just which ones are docked. I'd like to know which ones you actually use and how much.
3) What other features are you using? The most basic version of this could be to monitor which commands SlickEdit executes. These don't have to be commands typed into the command line. Commands underly all actions taken through the UI. Though it would be useful to know if you launched the command from with a keystroke, a menu, an icon, or the command line.

All of this would be completely anonymous with no source code sent as part of the information. Also, it would not include anything that could be used to identify the user.

Other info you should collect are metrics on sizes of projects and workspaces and performance of size dependent features. e.g. number of files, lines of code, number of symbols, sizes of tag files, time taken to look up symbol definitions, references, and symbol help, how often lookups fail because of symbol ambiguity, etc. I have used SlickEdit for many years in large part because of the integrated tagging capability for C++, but as the projects I've worked on have grown in size and complexity, I find the lookups more likely to get confused by ambiguity due to multiple namespaces, languages, and overloading, and more likely to take "too long". Collecting these size and performance metrics would help to characterize the behavior better, and get a picture of the distribution of project sizes across the customer base.

[thefrogger]

While the motivation is dead on, I don't know about this  .  I feel like we would experience the same outrage (or at least similar) that we were subjected to during the great trusted storage debacle of version 13.

In addition to agreeing with jimlangrunner's response to this, you need to keep in mind the motivations and conditions behind "trusted storage" in v13, compared to that of this new proposal. The former was implemented for *your* benefit, and was a mandatory requirement of installation. This usage gathering would primarily benefit the users that decide to enable the tracking, with only a secondary benefit to SE corporate of maintaining its customer satisfaction and potential growth.

I acknowledge that there are plenty of users out there who would not believe that rhetoric, and they will assume some ulterior motive for data gathering. Those users will not enable the feature, so it's hard to see why they would feel damaged by it. (Thus, I strongly agree with those who suggest keeping the feature off by default).

OTOH, the biggest risk on the downside would be having the product identified as "spyware" by some automated detection program. Even if users have to enable the tracking manually, they may not fully understand all the ramifications and still get freaked out when their paranoia protection program cries "foul". This is particularly the case if the spyware detection is installed sometime later on, when the user may have forgotten about the SE tracking and its intention. I'd love to think that programmers who use SE would be in a class above such misunderstanding, but it only takes one person, and the damage caused by some public outcry like "SlickEdit now includes spyware" could be considerable.

The harder it is to enable the feature (like a downloadable macro), the less likely it might be to get misinterpreted. But it also might skew the results further away from the general consensus since now a smaller fraction of the user base will contribute to the data. For the data to be valid for its intended purpose, it should represent a fair sampling of everyone who uses SE, and the proportions might be skewed significantly as the circle of participation grows smaller. (It's better to not know, than to know incorrectly).  (Edit: looks like Ryan is already on top of this concern in a message that came in while I was writing this. But I'll leave the thought here for completeness).

On rereading this post, it's pretty clear that I've both advocated as well as warned against implementing such a feature. That wasn't my initial intention, but in a roundabout way perhaps I supported Ryan's concern about the idea having a potential for backfiring.

--
John