Author Topic: Tagging does not understand Python 3 variables with types  (Read 3230 times)

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Tagging does not understand Python 3 variables with types
« on: June 16, 2019, 02:14:54 AM »
The context tagging in SlickEdit does not seem to understand
Python 3 syntax for declaring the type for a variable. For example:


GLOBAL_VAR: str = "foo-bar"

class Foo():
    CLASS_VAR: ClassVar[int] = 123


In both cases the context tagger will use the type instead
of the variable name when declaring the symbol. For example,
for GLOBAL_VAR, it will declare the symbol with the name "str"
instead of GLOBAL_VAR. For CLASS_VAR, it will be "ClassVar"
instead of CLASS_VAR. Basically, it seems that the tagging
parser ignores everything that comes before ":" and treats
everything after ":" as the variable name instead.

I tried this with SlickEdit 2016 and 2018 - the results are the same.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6826
  • Hero Points: 526
Re: Tagging does not understand Python 3 variables with types
« Reply #1 on: June 17, 2019, 01:35:53 PM »
We will look into this

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6826
  • Hero Points: 526
Re: Tagging does not understand Python 3 variables with types
« Reply #2 on: June 18, 2019, 01:04:05 AM »
What OS are you running SlickEdit on?

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #3 on: June 18, 2019, 01:07:31 AM »
> What OS are you running SlickEdit on?

Windows.

SlickEdit Support

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 384
  • Hero Points: 29
Re: Tagging does not understand Python 3 variables with types
« Reply #4 on: June 18, 2019, 07:52:50 PM »
PM sent.

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #5 on: June 19, 2019, 12:23:12 AM »
> PM sent.

Yes, I received a separate email with the instructions for downloading
vsapi.dll. Unfortunately, it does not seem to work - after replacing
this DLL, SlickEdit fails to start with a whole bunch of errors
about missing symbols. It seems like a version mismatch of some
kind (yes, I tried it in 23.0.2 64-bit). I compared the DLL sizes,
my vsapi.dll is 12MB, yours is 18MB, which seems like too much
of a change if this were the same version.

Maybe I should just wait for an official hotfix patch :)

The good news is that this is easily reproducible. Just create
a Python file with this content:


GLOBAL_VAR: str = "foo-bar"

class Foo():
    CLASS_VAR: ClassVar[int] = 123


And look in the Defs tab on the left. In my version it looks
like this:


str="foo-bar"

Foo
  ClassVar=123
    int


What it really should look like is this:


GLOBAL_VAR="foo-bar"

Foo
  CLASS_VAR=123

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #6 on: June 19, 2019, 02:15:14 AM »
Quote
Unfortunately, it does not seem to work - after replacing
this DLL, SlickEdit fails to start with a whole bunch of errors
about missing symbols.

On the second thought, it occurred to me: why is that the instructions
are asking me to replace vsapi.dll but the file is called cparse-2019-06-08.dll?
With that, I replaced cparse.dll instead. The file sizes are similar and SlickEdit
seems to be happy to start. It even recognizes types in global/class variables
now. So, yay, thank you! Time to create an official hot fix ;)

While we are on this topic, there are other places where types can be specified
in Python 3. For example, a typical function declaration would look
like that:


def do_something(foo: str, bar: int) -> List[bool]:
    pass


The current context tagging logic in SlickEdit seems to understand
this. At least I can navigate between functions without issues.
The only "peculiarity" that I see is that in the Defs tab on the left
functions are shown with the type separated by a space.
So, the above function would look like this:


do_something(foo str, bar int)


I do not know if this is the intended behavior or not. The only
consequence that I can see of this is that autocomplete for
parameters at the function call does not work. More specifically,
if I have a variables called "foo" and "bar", SlickEdit automatically
substitutes them when calling do_something() - but only if the function
is declared without types. With types, this does not work - SlickEdit
seems to be looking for variables called "foo str" and "bar int"
respectively.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6826
  • Hero Points: 526
Re: Tagging does not understand Python 3 variables with types
« Reply #7 on: June 19, 2019, 02:18:02 PM »
It probably would be better for the parser to strip the function argument types like it does for variable/member types since the type information can't be utilized for context tagging right now (it just messes it up). We will try to make this change. It's possible but a lot more work for the context tagging to support and use this type information. Definitely doable though.

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #8 on: June 19, 2019, 04:11:11 PM »
Quote
It probably would be better for the parser to strip the function argument types like it does for variable/member types since the type information can't be utilized for context tagging right now (it just messes it up).

Yes, stripping this information is probably the right (and the easiest) thing to do.

Quote
We will try to make this change.

Thank you!

Quote
It's possible but a lot more work for the context tagging to support and use this type information. Definitely doable though.

There does not seem to be any use (at least in the editor) for the types of function arguments. The editor is not going to try to match the types of variables with the types of arguments in the call - it is quite hard, actually. There is a static type checker for Python 3, mypy, that does this but, IMHO, this functionality does not really belong to an editor. The type of the return value is a different story. The editor COULD, in theory, look at the declared return type and use that in subsequent operations when calculating the choices for the autocomplete. For example, if you know that a function returns a string and the return value is stored in a variable called result, then when the user types result., the editor could know that this is a string and show the list of methods for a string.

SlickEdit Support

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 384
  • Hero Points: 29
Re: Tagging does not understand Python 3 variables with types
« Reply #9 on: June 19, 2019, 07:32:24 PM »
PM sent.

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #10 on: June 20, 2019, 05:23:09 PM »
Quote
PM sent.

Unfortunately, this build does not seem to work.
With this DLL the parser seems "to eat" all arguments.
For example, the do_something() example above
becomes this:

do_something( , )

Notice that there are no arguments, just the commas. Functions
without type information in the arguments are parsed fine though.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6826
  • Hero Points: 526
Re: Tagging does not understand Python 3 variables with types
« Reply #11 on: June 20, 2019, 06:03:36 PM »
Can you post a very small sample file. While I did find an issue, it's a more complicated case.

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #12 on: June 20, 2019, 06:39:24 PM »
Quote
Can you post a very small sample file.

Here is a sample file:


from typing import ClassVar, Dict, List, Set


GLOBAL_STR: str = "foo-bar"


GLOBAL_DICT: Dict[str, List[int]] = {
    "one": [1, 2, 3],
    "two": [2, 3, 4],
}


GLOBAL_MULTI_LINEDICT: Dict[
    str,
    List[int]
] = {
    "one": [1, 2, 3],
    "two": [2, 3, 4],
}


def global_func(x: str, y: Set[str], z: Dict[int, Set[str]]) -> bool:
    pass


def global_multi_line_func(
    x: str,
    y: Set[str],
    z: Dict[int, Set[str]],
) -> bool:
    pass


def global_multi_multi_line_func(
    x: str,
    y: Set[str],
    z: Dict[
        int,
        Set[str]
    ],
) -> bool:
    pass


class Foo():
    CLASS_STR: ClassVar[str] = "foo-bar"

    CLASS_DICT: ClassVar[Dict[str, List[int]]] = {
        "one": [1, 2, 3],
        "two": [2, 3, 4],
    }

    CLASS_MULTI_LINE_DICT: ClassVar[
        Dict[
            str,
            List[int]
        ]
    ] = {
        "one": [1, 2, 3],
        "two": [2, 3, 4],
    }

    def class_func(self, x: str, y: Set[str], z: Dict[int, Set[str]]) -> bool:
        pass

    def class_multi_line_func(
        self,
        x: str,
        y: Set[str],
        z: Dict[int, Set[str]],
    ) -> bool:
        pass

    def class_multi_multi_line_func(
        self,
        x: str,
        y: Set[str],
        z: Dict[
            int,
            Set[str]
        ],
    ) -> bool:
        pass


Quote
While I did find an issue, it's a more complicated case.

I think the problem is with function definitions spanning multiple lines.
Please see the attached screenshot - one line functions are parsed fine.
It is the multi-line functions that are not parsed correctly.

SlickEdit Support

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 384
  • Hero Points: 29
Re: Tagging does not understand Python 3 variables with types
« Reply #13 on: June 20, 2019, 09:27:20 PM »
PM sent.

commanderprs

  • Community Member
  • Posts: 10
  • Hero Points: 0
Re: Tagging does not understand Python 3 variables with types
« Reply #14 on: June 20, 2019, 10:23:17 PM »
Quote
PM sent.

The latest cparse.dll works great! Thank you! Time to create a hotfix ;)