Author Topic: New language support - how it really works ?  (Read 8706 times)

GrzegorzWolszczak

  • Junior Community Member
  • Posts: 3
  • Hero Points: 0
New language support - how it really works ?
« on: September 19, 2013, 02:36:21 PM »
Hi All.
I try to understand how adding new language support works in SlickEdit
I'v read http://blog.slickedit.com/2008/05/tutorial-adding-language-support-to-slickedit so I should be fine. But I'm not.

All I want to do is to enable tagging of my new languge. For the purpose of education I created new abstract language (called 'ala'), with source code extensions files (called 'ala' as well) that has only one keyword: 'function'.

I've created two source files (in attachments).
First (defs.ala) contains some function definition:
Code: [Select]
function func_1(arg)
{}

Second file (some_code.ala) just calls the function:
Code: [Select]
func_1(10);

I've created ala_proc_search(...) function in ala.e macro file.
I've created new project, added 'source' files to it and expected that Slick will tag this files correctly so when I search for references Slick will correctly show me that file defs.ala has func_1 definition and others func_1 references are just calls.
Or... the other way around... when I go into some_code.ala file and point my cursor on the func_1() and pick ''Go to definition of func_1" , I expected SlickEdit to direct me to defs.ala file with this definition.
Instead I get messge "Tag 'func_1' not found".
Why is that ???

I've checked that my ala_proc_search() function in ala.e file and verfied that it correctly parses symbol and split its name and type , where name is "func_1", and type is "function".
I call
Code: [Select]
tag_tree_compose_tag(symbol_name,'',ala_symbol_type,0,'','').
and assign its result to proc_name arg as I should.
By my code still doesn't work.
What am I missing ?

Could anybody help me please ?

I've attached all the files needed to run the example
My ala_proc_search code looks like this (debug lines cut off):
Code: [Select]
#define REGEX_PATTERN_1 '[ \t]*{function}[ \t]*{[a-zA-Z0-9_$]#} *'
int ala_proc_search(_str &proc_name, int find_first)
{
    typeless status=0;
    if ( find_first )
    {
        status=search('(function)','@rihwxsc');
    }
    else
    {
        status=repeat_search();
    }
    _str line='';
    _str symbol_name='';
    _str ala_symbol_type='';
    for (;;)
    {
        if ( status )
        {
            return(status);
        }
        get_line(line);
        if ( pos(REGEX_PATTERN_1,line,0,'R') )
        {
            ala_symbol_type = substr(line,pos('S0'),pos("0"));
            symbol_name = substr(line,pos('S1'),pos("1"));
            symbol_name=tag_tree_compose_tag(symbol_name,'',ala_symbol_type,0,'','');
            if ( proc_name=='' )
            {
                proc_name=symbol_name;
            }
            else if ( proc_name :== symbol_name )
            {
                status=repeat_search();
                continue;
            }
            return(0);
        }
        else
        {
            status=repeat_search();
        }
    }
}


I searched in SlickEdit documentation, on the web, I looked into other files in marco/ directory (most of the files are very complex and I'm only the beginner in Slick macro writting), but I still don't understand what am I missing.

If you can please help me. I am close to lose hope that I will make it work it at all.

P.S. hostfix for slickedit version 17.0.3 - change the number in hostfix.xml to mach your version of Slickedit
P.S.2 file ala.e has some debug code but it should be straitforward
« Last Edit: September 19, 2013, 04:31:20 PM by GrzegorzWolszczak »

Graeme

  • Senior Community Member
  • Posts: 2793
  • Hero Points: 347
Re: New language support - how it really works ?
« Reply #1 on: September 22, 2013, 01:46:32 PM »
Hi

I've been having a look at your code.  I've run out of time right now but here's what I found so far.
I added some code to debug_log to get some output.  "say" produces a log window - you can use Ctrl-H in that window to get some help.
void debug_log(_str a_line )
{
   say(a_line);
 }

and I found that with just this in some_code.ala
   func_1(10);
saving the file triggers tagging and your debug code outputs this
1.procname ''
2.findfirst '1'
2.1 search status '-3022'
1.procname ''
2.findfirst '1'
2.1 search status '-3022'

If I change the code to
function func_2(arg)
{
   func_1(10);
}
and save the file, I now get this
good
1.procname ''
2.findfirst '1'
2.1 search status '0'
3.line 'function func_2(arg)'
4. combined symbol name 'func_2(function)'
5.1. return 0
1.procname ''
2.findfirst '0'
2.1 repeating search...
2.2 search status '-3022'

showing that it parsed func_2 correctly.  If I use slickedit's find_symbol dialog, it shows func_2 but not func_1.  func_2 is being found via "context tagging"  (the current buffer) rather than the tag file.

If I look in tools -> tag files, it shows the two files as being part of the tag file.
If I now do "rebuild tag file", there are no calls to ala_proc_search so I suspect there is something wrong with _ala_MaybeBuildTagFile

When I use Ctrl-dot (go to definition) with the cursor on the func_1 call, the _ala_MaybeBuildTagFile function gets called twice and outputs "good" and "1" each time, indicating both the recycle call failed and ext_MaybeBuildTagFile failed - however a tag file is getting produced somehow.  Maybe you should look more closely at the corresponding function in the tutorial - especially check case sensitivity etc.

Code: [Select]
int _ala_MaybeBuildTagFile(int &tfindex)
{
   _str langid = 'ala';
   _str tagfilename = '';
   if (ext_MaybeRecycleTagFile(tfindex, tagfilename, langid, langid)) {
      say("hello");
      return(0);
   }
   say("good");
   int status = ext_MaybeBuildTagFile(tfindex, langid, langid, "ALA Builtins");
   say(status);
   return 0;
}

« Last Edit: September 23, 2013, 12:37:31 AM by Graeme »

Graeme

  • Senior Community Member
  • Posts: 2793
  • Hero Points: 347
Re: New language support - how it really works ?
« Reply #2 on: September 23, 2013, 11:09:45 AM »
Disregard my last post.

I changed one line of your code to this and now tagging works
            symbol_name=tag_tree_compose_tag(symbol_name,'',"proc",0,'','');

You need to set the third argument to "proc" when you call tag_tree_compose_tag  - just as the code in logo.e does.

GrzegorzWolszczak

  • Junior Community Member
  • Posts: 3
  • Hero Points: 0
Re: New language support - how it really works ?
« Reply #3 on: September 23, 2013, 04:43:25 PM »
Greame,

First, thanks very much for your help.
Second, your first post was also valuable. It seems that I incorecttly set lang_id in _ala_MaybeBuildTagFile function. It was 'ala' but I think it should be the set to the same value as the second argument of _CreateLanguage function, which in this case was "ALA" (uppercased).

On the other hand. It is kind of dissapointing that we must discover all those things by trial-and-error instead of just reading this in documentation. The tutorial (http://blog.slickedit.com/2008/05/tutorial-adding-language-support-to-slickedit) does not state it very clearly.

Whats more, the most 'interesting' functions like [ext]_list_locals, int _[ext]_find_matching_word(boolean quiet) , _[lang]_get_expression_info and so on... are left unimplemented as 'exercise for the reader'. As if the author did not 'really' want reader to learn about this.
But maybe this is my feel of dissapoinment speaking. I was 'fighinting' with this code for some time now and acutally thanks to your post I've moved forward.

Graeme

  • Senior Community Member
  • Posts: 2793
  • Hero Points: 347
Re: New language support - how it really works ?
« Reply #4 on: September 24, 2013, 01:04:20 PM »
Hi

I may have confused you by suggesting you check the "case sensitivity" in that maybe-tag function.  As far as I can see, the argument corresponding to the "basename" parameter is supposed to be the language-id, not the mode name  - so I think you were right the first time in using ala (your language id) and not ALA (your mode name).

Did you notice that ext_MaybeBuildTagFile goes looking in the slickedit installation "builtins" folder for a file called ala.tagdoc that it uses to get documentation for the "builtins".
In case you didn't know, in the slickedit installation "docs" folder there's a file called SlickCMacroBestPractices.pdf which has some useful info on Slick C.  Also, Slickedit comes with a debugger for Slick C that allows you to breakpoint, single-step etc.  That wouldn't have helped in this case though.  I think you'll find it worthwhile learning Slick C.  It took me a while to get used to the fact that Slick C can de-reference integer constants which it treats as windows-ids.  Also, I find it's often useful to record a macro and then edit the generated code.


GrzegorzWolszczak

  • Junior Community Member
  • Posts: 3
  • Hero Points: 0
Re: New language support - how it really works ?
« Reply #5 on: September 26, 2013, 01:05:56 PM »
Greame,

 I've modified/corrected and simplified my 'ala language' example (attached). E.g. for debugs i use function _SccDisplayOutput from "vc.e" file. Sometimes it is more convenient than 'say'

I noticed something strange though. In defload() function I now display what is the lexer name for language.
Code: [Select]
Lexer name for lang id 'ALA' is :'ALA'But when I try to find this lexer in Tools->Options->Langugage->User Defined Langugages -> Ala ->Color Coding
there is no such lexer in "Lexer name:" combo box.
I need to manually import ala.vlx in order to add this to the available names.
Is it a bug ?

I'm starting (slowly) to understand 'in general' how the _proc_search work and does, but in tags.e , there is piece of documentation saying the that even 'more important' function is vs"ext"_list_tags.
Do you happen to know how the example implementation should look  for lets say 'ala langugage trivial example' ?

There is reference in the documentation to see file "sample.c" but it is nowhere to find in the SlickEdit instalation directory. Where to find this file ?
Are there any 'exmple' files (on the internet maybe) that could work as learning tutorial on how to implement "ext"_list_tags ?
« Last Edit: September 26, 2013, 02:18:37 PM by GrzegorzWolszczak »

Graeme

  • Senior Community Member
  • Posts: 2793
  • Hero Points: 347
Re: New language support - how it really works ?
« Reply #6 on: September 26, 2013, 03:43:08 PM »
Hi

The comments in tags.e refer to "sample.c" but it's a typo - it should be "simple.c"  - this is in the installation "samples" folder.  There aren't any examples of a slick C version of xxx-list-tags as far as I can see  - so they're probably all in a dll I guess.  list-tags wouldn't be trivial to implement  - you would need to be able to completely parse your ala source code I think, but I guess it would allow you to insert tags in the database for all identifiers in your source code. tags.e says it should call tag_insert_tag to add a tag to the tagging database. 

tags.e says you don't need both proc-search and list-tags so I suspect you shouldn't worry about implementing list-tags for now.  I'll have a look at that lexer problem you mentioned in a day or so.

orenbenkiki

  • Junior Community Member
  • Posts: 5
  • Hero Points: 1
Re: New language support - how it really works ?
« Reply #7 on: September 15, 2016, 07:06:15 PM »
I noticed something strange though. In defload() function I now display what is the lexer name for language.
Code: [Select]
Lexer name for lang id 'ALA' is :'ALA'But when I try to find this lexer in Tools->Options->Langugage->User Defined Langugages -> Ala ->Color Coding
there is no such lexer in "Lexer name:" combo box.
I need to manually import ala.vlx in order to add this to the available names.
Is it a bug ?

I have the same problem in my user-defined language, even though I invoke cload to load the configuration file. So... is there a special trick to loading the lexer, or must my users go through the process of doing it manually? (Since nobody answered the original question in 3 years I'm not too optimistic, but one can hope...).