Author Topic: Bad handling of really large file  (Read 2215 times)

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Bad handling of really large file
« on: July 20, 2016, 06:59:08 PM »
I have a log file:
07/20/2016  11:37 AM    10,487,407,894 log9.txt


I opened it in Slick.
Slick is hung.


PROCMON shows that Slick is reading 8KB chunks from log9.txt, but then writing in 14,960 byte chunks to the slk temp file.


It it progressing at about 21MB/s (this is on an SSD).





11:41:31.3036255 AM   vs.exe   6316   WriteFile   C:\Users\josepo03\AppData\Local\Temp\$slk.6316   SUCCESS   Offset: 6,514,825,680, Length: 14,960, Priority: Normal
11:41:31.3036608 AM   vs.exe   6316   ReadFile   C:\src\sdk3\log9.txt   SUCCESS   Offset: 3,511,664,640, Length: 8,192
11:41:31.3037036 AM   vs.exe   6316   WriteFile   C:\Users\josepo03\AppData\Local\Temp\$slk.6316   SUCCESS   Offset: 6,514,840,640, Length: 14,960, Priority: Normal
11:41:31.3037358 AM   vs.exe   6316   ReadFile   C:\src\sdk3\log9.txt   SUCCESS   Offset: 3,511,672,832, Length: 8,192
11:41:31.3037782 AM   vs.exe   6316   WriteFile   C:\Users\josepo03\AppData\Local\Temp\$slk.6316   SUCCESS   Offset: 6,514,855,600, Length: 14,960, Priority: Normal
11:41:31.3039263 AM   vs.exe   6316   ReadFile   C:\src\sdk3\log9.txt   SUCCESS   Offset: 3,511,681,024, Length: 8,192




It seems to be making more than one pass ....


When I first loaded this file is was much smaller.
I did a "Readload with encoding utf-8" (It defaulted to sbcs/dbcs).


After restarting slick, and opening this file with encoding sbcs/dbcs, it now doesn't behave so badly.




Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Bad handling of really large file
« Reply #1 on: July 20, 2016, 11:03:20 PM »
If you use SBCS/DBCS mode, the file will open in an instant. From the command line: e +ftext "filename". From invocation: vs +ftext "filename". If this is good enough, you will get great performance.

The issue with UTF-8 files is that it has to scan for long lines to avoid bad display performance. It has to read the entire file for this and the block size is small. We might be able to play games to use a bigger block size (or memory mapped I/O). Not much data is spilled so that's not a big deal.

I'd like SlickEdit to open huge Utf-8 as fast as DBCS/SBCS files. I think this it's possible (much harder) and this would be much more worth while than just messing with the block size or using memory mapped I/O.  Maybe I'll have time to pull this one off.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Bad handling of really large file
« Reply #2 on: July 25, 2016, 08:01:28 PM »
The next beta will be able to open huge Utf-8 files in an instant (like DBCS/SBCS files).