Author Topic: Tagging has trouble with "void ::Foo()"  (Read 9817 times)

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Tagging has trouble with "void ::Foo()"
« on: April 18, 2009, 03:14:30 am »
It has taken me a whole year to finally track down why in my team's product code base Ctrl+. on some functions/methods can find the prototype but not the declaration.  It's not SlickEdit's fault, but I think I need help from SlickEdit to solve the problem.

On the surface...
When tagging encounters a function declaration like this:
Code: [Select]
void ::Foo()
{
}
It believes it is seeing a function "void::Foo()" with no return value.
It thinks "void" is a namespace, not a return type (and that's probably the Correct interpretation).
And when pressing Ctrl+. on a call to "Foo();" the editor knows it's not calling "void::Foo()" so it ends up deciding it doesn't know where "Foo()" is declared.

In reality...
In reality the declaration is this:
Code: [Select]
void SomeNamespace::Foo()
{
}
The usercpp.h file defines "SomeNamespace" to an empty string.  The intended effect was to make tagging no longer differentiate between the default namespace and the SomeNamespace namespace.
Why would I do that?  An excellent question...

The Spaghetti Effect...
The trouble is that certain folks are in love with macros.  Worse, they love to #include a particular header file in every source file, and inside that header file they invoke a macro that resolves to "namespace SomeNamespace {}; using namespace SomeNamespace;".  Only, sometimes they explicitly declare the namespace.  And sometimes they invoke the macro at the top of the .cpp file instead of including the header.  :'(  It's a tangled mess.

So the effect is that sometimes SlickEdit can see the namespace declaration, and sometimes SlickEdit can't.  So the tag database ends up with a schizophrenic view of which functions/classes/etc are in which namespaces.

I know this is the fault of certain developers who love to do it this way, but I need a way for SlickEdit to make sense of the situation.  Under the circumstances I would accept if SlickEdit just pretended that certain namespaces didn't exist.  That's what I was trying to accomplish by using preprocessor hints to define away certain namespaces.

Help  ???
Any ideas how to work around this, short of convincing multiple teams with different intra-office-political agendas to agree to format the code sensibly?  My Jedi powers are too weak to control that many minds.

Technically the source files are fine.  For performance reasons the tagger doesn't dig into headers, so the tagger doesn't have an accurate enough picture of the mess to be able to make sense of it.

If I could define away "SomeNamespace::" (i.e. include colons as part of the identifier to be preprocessed away) that would solve my dilemma.
Or if I could provide a list of namespaces that should be ignored, and the tagger dropped them whenever it ran across them.
Or if I could provide a list of files that need specific additional hints.
Etc.

Also, I am strongly growing to wish there was a global usercpp.h, but also a workspace-specific preprocessor hint file, and also project-specific preprocessor hint file.  Different projects have different quirks, and I've having to make compromises where I chose to mess up tagging in one project to improve tagging in another project, just because a generic symbol name like "Utilities" or "Error" or "Verify" is used completely different in two different projects.

jb

  • Community Member
  • Posts: 37
  • Hero Points: 0
Re: Tagging has trouble with "void ::Foo()"
« Reply #1 on: April 18, 2009, 11:24:28 am »
It has taken me a whole year to finally track down why in my team's product code base Ctrl+. on some functions/methods can find the prototype but not the declaration.  It's not SlickEdit's fault, but I think I need help from SlickEdit to solve the problem.

On the surface...
When tagging encounters a function declaration like this:
Code: [Select]
void ::Foo()
{
}
It believes it is seeing a function "void::Foo()" with no return value.

I wonder whether the function definition above is legal C++. According to the Visual C++ 2005 compiler ::Foo() is only allowed in declarations and calls, but not in the definition. (Where I write this post I have no other compiler at hand.)
I think the SlickEdit tagger should ignore illegal code instead of inventing something such as interpreting "void   ::Foo()" as "void::Foo()". Furthermore C++ does not support default-int as return type and "void" would be an illegal namespace name because "void" is a keyword.

In reality...
In reality the declaration is this:
Code: [Select]
void SomeNamespace::Foo()
{
}
The usercpp.h file defines "SomeNamespace" to an empty string.  The intended effect was to make tagging no longer differentiate between the default namespace and the SomeNamespace namespace.
Why would I do that?  An excellent question...

You explained in detail HOW some programmers made your workaround ("#define SomeNamespace" in SlickEdit's usercpp.h) fail by placing "namespace SomeNamespace {}; using namespace SomeNamespace;" at the top of some cpp files (in three different ways), but you did NOT explain WHY your workaround is desirable in the first place. Or did I miss something?
With your macro in usercpp.h you make the SlickEdit tagger sees something else than the compiler. Why?

(As far as I remember I used usercpp.h once to make the SlickEdit tagger always recognize calls of questionable macros such as "#define INTERFACE class". But your case seems different. "void SomeNamespace::Foo() {}" is plain legal C++ source code.)
« Last Edit: April 18, 2009, 12:18:54 pm by jb »

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #2 on: April 18, 2009, 06:02:05 pm »
I wonder whether the function definition above is legal C++. According to the Visual C++ 2005 compiler ::Foo() is only allowed in declarations and calls, but not in the definition. (Where I write this post I have no other compiler at hand.)
I think the SlickEdit tagger should ignore illegal code instead of inventing something such as interpreting "void   ::Foo()" as "void::Foo()". Furthermore C++ does not support default-int as return type and "void" would be an illegal namespace name because "void" is a keyword.
Exactly.  I said "it is probably the Correct interpretation".  It's not SlickEdit's fault.  It's not legal C++.  And it isn't what's really in the code.  I was just illustrating what the tagger ends up seeing.


You explained in detail HOW some programmers made your workaround ("#define SomeNamespace" in SlickEdit's usercpp.h) fail by placing "namespace SomeNamespace {}; using namespace SomeNamespace;" at the top of some cpp files (in three different ways), but you did NOT explain WHY your workaround is desirable in the first place. Or did I miss something?
With your macro in usercpp.h you make the SlickEdit tagger sees something else than the compiler. Why?
I tried to explain why the workaround is needed, in the section "The Spaghetti Effect".

Clearing up a couple things:
- Placing "namespace ... using ...;" at the top of cpp files did not cause the workaround to fail.  If people had done that consistently, everything would have been well and the workaround wouldn't have been needed in the first place.  Or even if they had used a macro to do so.
- The root cause of the tagging mishap is that 1/3 of the time people do not express anywhere in the cpp file (neither written out, nor with a macro) that the file should be "using SomeNamespace".  Instead they let some included header do that for them.  So nowhere in the cpp file is there any mention of the namespace.  The tagger does not recurse into #include directives.  Or maybe it's not the tagger getting confused, maybe it's the find-tag or push-tag commands and the local analysis they do.  I don't know, it didn't seem important to me to figure out exactly which piece of the tagging system is being misled.
- What causes the workaround to fail is that it ends up spoofing invalid C++, so the tagger tries to make sense of the situation and ends up treating the return type as if it were a namespace name, and then the symbol gets recorded as being in a bogus namespace, which means nothing will ever find it.  I don't expect the tagger to do anything better than that -- it's a reasonable interpretation, and doing symbolic analysis to realize the true situation seems like not only a performance problem but also a problem that requires multiple passes (suppose it encounters that in the first file being tagged; there are 36399 more files yet to tag and the return type might be declared in one of them, and until the later file is tagged it won't be possible to look up the return type in the first file).

So without a workaround:
- 1/3 of the time SlickEdit doesn't know to be looking in the "using SomeNamespace".
- So auto-complete doesn't work, tag lookup doesn't work, etc.
- And I tend to get duplicate declarations -- one in the namespace, one not in the namespace.  Maybe that comes from not having found all of the macros I need to add hints for in the usercpp.h file.

So the solutions I've thought of are either some way to either hint that files should be treated as if they contain "using SomeNamespace", or some way to completely ignore namespaces.  At this point I don't see way to accomplish either one without some kind of help from the editor.

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #3 on: April 18, 2009, 10:45:10 pm »
I reverted the tagging workaround to investigate the original root issues again.
Turns out to be interesting stuff, it is pretty easy for namespaces to confuse tagging.

I've opened a Support Case and sent a tiny workspace that reproduces the problems.
The workspace is also attached below -- open the workspace, open repro.cpp, and read the comments.

It took me a year of wrestling with usercpp.h and tagging to finally track down the root problem.  The effect is that when using namespaces in reasonable ways, the reliability of tagging becomes significantly impaired.  Hopefully the issues in the sample workspace can be fixed in the core SlickEdit, without needing tricksy manual workarounds.

jb

  • Community Member
  • Posts: 37
  • Hero Points: 0
Re: Tagging has trouble with "void ::Foo()"
« Reply #4 on: April 18, 2009, 10:59:41 pm »
- The root cause of the tagging mishap is that 1/3 of the time people do not express anywhere in the cpp file (neither written out, nor with a macro) that the file should be "using SomeNamespace".  Instead they let some included header do that for them.  So nowhere in the cpp file is there any mention of the namespace.  The tagger does not recurse into #include directives.

As far as I understand now, your problem boils down to the following:
1. There are cpp files where a "using namespace SomeNamespace;" statement is placed indirectly by an #include directive.
2. The SlickEdit tagger does not recurse into #include directives (unfortunately).
3. Consequently when a reference to a symbol defined in SomeNamespace occurs in such a cpp file, then SlickEdit cannot find the definition of the symbol.
Is that correct?

- What causes the workaround to fail is that it ends up spoofing invalid C++, so the tagger tries to make sense of the situation and ends up treating the return type as if it were a namespace name, and then the symbol gets recorded as being in a bogus namespace, which means nothing will ever find it.  I don't expect the tagger to do anything better than that -- it's a reasonable interpretation, and doing symbolic analysis to realize the true situation seems like not only a performance problem but also a problem that requires multiple passes (suppose it encounters that in the first file being tagged; there are 36399 more files yet to tag and the return type might be declared in one of them, and until the later file is tagged it won't be possible to look up the return type in the first file).

OK, it seems that I judged your workaround ("#define SomeNamespace" in SlickEdit's usercpp.h) too mildly in my previous post. Actually your workaround does not solve any problem because it makes the code invalid for the SlickEdit tagger. Generally the situation is even worse than before.
As already mentioned, I think it is not reasonable if the tagger interprets invalid code (such as taking the return type name as namespace name).

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #5 on: April 18, 2009, 11:53:10 pm »
Aha!  And the other major tagging problem I've been facing for the past year, having to do with smart pointers, also turns out to be due to a problem with namespaces.  Only in this case there isn't even anything questionable or inconsistent about the namespace usage.

See the attached Repro2.zip file for a second sample workspace that demonstrates this other tagging problem.

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #6 on: April 19, 2009, 12:04:18 am »
As far as I understand now, your problem boils down to the following:
1. There are cpp files where a "using namespace SomeNamespace;" statement is placed indirectly by an #include directive.
2. The SlickEdit tagger does not recurse into #include directives (unfortunately).
3. Consequently when a reference to a symbol defined in SomeNamespace occurs in such a cpp file, then SlickEdit cannot find the definition of the symbol.
Is that correct?
#1 -- Turned out to be a red herring.  The "Repro.zip" sample workspace I posted earlier demonstrates the root problem, without any macros or indirect "using"s.
#2 -- Yes.  And I don't want the tagger to recurse.
#3 -- Yes.  The sample workspace shows it's more than that, too, though.

The simplest way I can show the 4 problems, is in the Repro.zip sample workspace.


OK, it seems that I judged your workaround ("#define SomeNamespace" in SlickEdit's usercpp.h) too mildly in my previous post. As already mentioned, I think it is not reasonable if the tagger interprets invalid code (such as taking the return type name as namespace name).
Yes.  (I've acknowledged that from the beginning :)).
What I'm looking for is a solution that does work.

Actually your workaround does not solve any problem because it makes the code invalid for the SlickEdit tagger. Generally the situation is even worse than before.
Give me a little credit.  ;)  Empirically the results of the workaround are much better than not using the workaround.  It is a tradeoff between a huge set of unrecognized symbols without the workaround, or a much smaller (but still significant) set of misplaced symbols with the workaround.
Without the workaround, things go wrong everywhere.  But with the workaround things go wrong only when explicit namespace qualification is used "void SomeNamespace::Foo()" instead of using scoping like "namespace SomeNamespace {".
So under the circumstances, the workaround makes the situation much better.
Also note that the workaround does not make the code "invalid" for the SlickEdit tagger, it just causes the tagger to think the return type is a namespace.

Anyway, you can see the specific problems that I'm facing, in the Repro.zip and Repro2.zip sample workspaces I posted in previous replies.  They illustrate the problems concisely and completely.

jb

  • Community Member
  • Posts: 37
  • Hero Points: 0
Re: Tagging has trouble with "void ::Foo()"
« Reply #7 on: April 19, 2009, 01:17:23 pm »
Give me a little credit.  ;)  Empirically the results of the workaround are much better than not using the workaround.
...
Also note that the workaround does not make the code "invalid" for the SlickEdit tagger, it just causes the tagger to think the return type is a namespace.

I have the impression that your workaround is more or less a lucky strike. Although it may currently be a partial solution, it relies on the current implementation of the SlickEdit tagger for C++ which can change with every version. Maybe a future version of SlickEdit can be configured to use output generated by a specific compiler with a specific switch for tagging.

However you are right in one point. The SlickEdit tagger really has some limitations when namespaces are involved and they should be removed. I boiled down your example code in Repro2.zip to the essence. Now it is much shorter and shows even more!
Code: [Select]
namespace SomeNamespace { // COMMENT OUT THIS LINE TO MAKE IT WORK
  template <typename T>
  struct SomeTemplateClass
  {
    T* operator->();
    T& GetTheT();
    int n1;
  };

  template <typename T>
  struct SomeDerivedTemplateClass : SomeTemplateClass<T>
  {
    int n2;
  };
}; //namespace SomeNamespace // COMMENT OUT THIS LINE TO MAKE IT WORK

using namespace SomeNamespace;

struct SomeGlobalClass
{
  void Babble();
};

void TestAutoListMembersOfSlickEdit14_0_0_7()
{
  SomeTemplateClass<SomeGlobalClass> obj1;
  obj1.n1;                  // OK
  obj1.GetTheT().Babble();  // OK
  obj1->Babble();           // OK

  SomeDerivedTemplateClass<SomeGlobalClass> obj2;
  obj2.n2;                  // OK
  // If SomeDerivedTemplateClass is defined in SomeNamespace
  obj2.GetTheT().GetTheT(); // GetTheT() is listed again and again. // ### ERROR
  obj2->                    // Babble() is NOT listed.              // ### ERROR
  // If SomeDerivedTemplateClass is defined in the GLOBAL namespace
  obj2.GetTheT().Babble();  // OK
  obj2->Babble();           // OK
}

There is a very good quote from Blaise Pascal on page two of some (or probably all) books of Addison-Wesley's "The C++ In-Depth Series".
It says: "I have made this letter longer than usual, because I lack the time to make it short."
I think it applies to writing as well as to programming.
In this case I spent MY time to get to the bottom. Actually I would have expected this work from YOU after you struggled with this case for about a year (as you write). Your initial complaining about some macro loving programmers was completely misleading. (I also dislike macros generally by the way.)

But nevertheless it's true that the SlickEdit tagger also has some limitations regarding macros. See also "Tagging/Context problem with #define?"
« Last Edit: April 19, 2009, 01:21:08 pm by jb »

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #8 on: April 19, 2009, 06:19:22 pm »
I have the impression that your workaround is more or less a lucky strike. Although it may currently be a partial solution, it relies on the current implementation of the SlickEdit tagger for C++ which can change with every version.
...
There is a very good quote from Blaise Pascal on page two of some (or probably all) books of Addison-Wesley's "The C++ In-Depth Series".
It says: "I have made this letter longer than usual, because I lack the time to make it short."
I think it applies to writing as well as to programming.
In this case I spent MY time to get to the bottom. Actually I would have expected this work from YOU after you struggled with this case for about a year (as you write).
Thank you for the many criticisms, I will think about them.

Can we please get on topic?
I'm looking for suggestions for how I can solve or work around the problems.

jb

  • Community Member
  • Posts: 37
  • Hero Points: 0
Re: Tagging has trouble with "void ::Foo()"
« Reply #9 on: April 19, 2009, 08:06:55 pm »
Can we please get on topic?
Yes, we can, but I hope that my meta remarks help to get more efficient discussions in the future.

I'm looking for suggestions for how I can solve or work around the problems.
Well, I would summarize the situation as follows:
1. The SlickEdit 14.0.0.7 tagger for C++ has some limitations with namespaces, #include directives, #define directives (macros) and probably all other preprocessor directives.
   Some problems arise from combinations of these limitations.
   I consider these limitations as MINOR.
2. Now you can:
   a) Just accept these limitations and do nothing.
   b) Adjust your coding conventions to minimize the impact of these limitations on your projects, that is,
      avoid "#define" directives whenever possible and do not place "using namespace" statements in header files
      (both is already recommended since more than 10 years by Bjarne Stroustrup in the 3rd edition of his C++ book).
   c) Try to convince the SlickEdit company to improve its C++ tagger.

I think it's best to burn the candle at both ends (b+c).
Maybe the chances for c) are not that bad because it seems to me that Visual Studio 2008 has surpassed (at least partly) the C++ tagging capabilities of SlickEdit. So there is a certain pressure for SlickEdit to catch up.
« Last Edit: April 19, 2009, 08:17:24 pm by jb »

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #10 on: April 19, 2009, 10:18:56 pm »
...The SlickEdit 14.0.0.7 tagger for C++ has some limitations with namespaces...
...I consider these limitations as MINOR.
I'm glad that the namespace limitations impact you in minor ways.
However, they impact me in major ways.


2. Now you can:
   a) Just accept these limitations and do nothing.
   b) Adjust your coding conventions to minimize the impact of these limitations on your projects, that is,
      avoid "#define" directives whenever possible and do not place "using namespace" statements in header files
      (both is already recommended since more than 10 years by Bjarne Stroustrup in the 3rd edition of his C++ book).
   c) Try to convince the SlickEdit company to improve its C++ tagger.
I think it's best to burn the candle at both ends (b+c).
Maybe the chances for c) are not that bad because it seems to me that Visual Studio 2008 has surpassed (at least partly) the C++ tagging capabilities of SlickEdit. So there is a certain pressure for SlickEdit to catch up.
Those are three good suggestions.
  • If (a) were acceptable, I wouldn't have posted.
  • I already explained why (b) is not possible, but the coding change that would be required would be to convince 500+ developers to remove all usage of namespaces from 80,000+ files.
  • For (c) I distilled three sample workspaces that reproduce 6 separate symptoms, and sent them to Support.

I'm looking for help with:

   d) Find a workaround in the meantime to minimize the effect of the problems SlickEdit experiences with namespaces in C++.

I think I'll give up on this topic and wait for a response from Support.
Maybe I'll post again about it in a few weeks.
« Last Edit: April 19, 2009, 10:49:04 pm by chrisant »

Lee

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 1177
  • Hero Points: 112
Re: Tagging has trouble with "void ::Foo()"
« Reply #11 on: April 20, 2009, 01:39:27 pm »
I can see that the function prototypes are fully-qualified in the namespace declaration, and the functions definitions are tagged using only the partial qualified signature.  I'm pretty sure that is where the confusion lies, the tagging engine doesn't do a name lookup like the compiler would to generate the fully qualified name for that function.  So, the tag file contains two versions with slightly different signatures.  I'm trying to think of how you would workaround this, but I'm not sure.  I'm hoping Dennis might know.

I am curious as the purpose of the empty namespace declaration after the namespace was already defined by the #include.  What purpose does that serve?

chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #12 on: April 20, 2009, 05:12:43 pm »
I am curious as the purpose of the empty namespace declaration after the namespace was already defined by the #include.  What purpose does that serve?
I don't know for sure, but I think it is a forward declare, so things can be put into the namespace explicitly ("void ThisNamespace::Foo()") without being surrounded by a namespace scope ("namespace ThisNamespace { void Foo(); }").  It's in the BEGIN_OUR_NAMESPACE macro that is broadly used, so for accuracy I included it in the sample workspace in case it made a difference to the editor.

Dennis

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 2445
  • Hero Points: 351
Re: Tagging has trouble with "void ::Foo()"
« Reply #13 on: April 23, 2009, 04:30:29 pm »
OK, I'm going to first address the "easy" issues, then get into the specific namespace issues when there is more time available.

First:  Tagging of "void ::Foo();"  Yes, this was an anomoly produced by chrisant's workaround, but as it turns out, of course, it is also valid C++.  I am currently working on a fix for this issue which is targetted for 14.0.1.  There is a caveat, it will work fine for functions that return void or builtin types, but it has no chance at a function whose return type is a typedef or some other symbol which appears to be like any other class name:

Code: [Select]
void ::Foo() {                  // works
}
unsigned int ::Foo() {       // works
}
ClassName ::Foo() {         // Foo will be (incorrectly) considered as a
}                                  // member of ClassName with an implicit int return type

Yes the handling of "ClassName ::Foo();" is incorrect.  I'm filing a separate defect about that which we will investigate later.  It may be one of those where it's just not worth fixing.

Second:  To address the funky workaround.  If instead of #define'ing "SomeNamespace" to the empty string, if you instead pointed them to "std", you could take advantage of an ugly workaround I was forced to put into SlickEdit, precisely because some folks thought that putting "using namespace std;" in their header files was a pretty good idea.  Changing your usercpp.h #define in this manner would shuffle all those declarations into the "std" namespace which is always searched by default, and thus does not require a using statement in the mainstream code.

I'm looking at the Repro.zip test cases next.


chrisant

  • Senior Community Member
  • Posts: 1413
  • Hero Points: 131
Re: Tagging has trouble with "void ::Foo()"
« Reply #14 on: April 23, 2009, 06:20:39 pm »
Second:  To address the funky workaround.  If instead of #define'ing "SomeNamespace" to the empty string, if you instead pointed them to "std", you could take advantage of an ugly workaround I was forced to put into SlickEdit, precisely because some folks thought that putting "using namespace std;" in their header files was a pretty good idea.  Changing your usercpp.h #define in this manner would shuffle all those declarations into the "std" namespace which is always searched by default, and thus does not require a using statement in the mainstream code.
Thanks, Dennis, that's a great idea.  It is much cleaner than using an empty string, and it does indeed improve finding the symbol during Ctrl+. so it is a very welcome improvement (hp++ of course).

It doesn't improve enough things to be a long term workaround.
For completeness, here is how the 'std' workaround affects the Repro zip workspaces as far as I can tell:

Repro1:
 ;D It does allow Ctrl+. to jump to both the prototype and the function (of the 6 issues, this is the most important to me, but completion is a close second -- and I'm guessing the issues behind symbol coloring and completion are related and either both will work or both will not).
 ;D Dialog does not pop up when using Ctrl+. on Quux
 :( Symbol coloring is still red.
 :( Completion still does not work, probably because symbol completion doesn't take advantage of the usercpp.h defines (and I wouldn't expect it to use the tagging hints).

Repro3:  (which includes Repro2 in it)
 :( Symbol coloring is still red.
 :( Completion still does not work.

(Perhaps I should trademark "Emotibullets"?  No, perhaps not...)



I just want to add, I am highly impressed by the complexity of SE's tag handling.  The editor is comprehending types and inheritance with templates, even for some funky template usage.  It's just getting a little confused about namespaces and nested namespaces.  The namespace issue may be hard because of ambiguities/etc, but the template support had to have been some meaty code to write.  There is some really powerful stuff going on under the covers in the editor.
« Last Edit: April 23, 2009, 06:31:49 pm by chrisant »