Author Topic: automated project file adding  (Read 57694 times)

mikesart

  • Community Member
  • Posts: 56
  • Hero Points: 11
automated project file adding
« on: April 12, 2008, 01:23:43 AM »
I work on several projects where folks are adding/removing files via source control and I finally got around to trying to automate the project properties files dialog. The macro adds files from on a list of directories based on the project names. Not sure if anyone else will find it useful, but I'm sticking it up here just in case. (It's vaguely related to this thread but all handled via SlickEdit Macros: http://community.slickedit.com/index.php?topic=1616.0). Should also be quite a bit faster as it only adds or removes modified files.

To use, I modify the dirlist, filespec, exclude list, hit F12 (reload macro), then type in ssync and am done. Normally I'm not modifying the dir spec so I'm just typing ssync.
 -Mike

Code: [Select]
struct DIRINFO
{
   _str opts;
   _str dir;
   _str filespec;
};

_command void ssync() name_info(','VSARG2_REQUIRES_PROJECT_SUPPORT)
{
   int i;
   DIRINFO dir_list[];
   _str exclude_list = "";
   _str file_spec_default_list = "*.c;*.cc;*.cpp;*.cp;*.cxx;*.c++;*.h;*.hh;*.hpp;*.hxx;*.inl;*.e;*.sh;*.bat;*.cmd;*.php";

   dir_list._makeempty();

   // Always add main SE macros and build files.
   dinfo_add(dir_list, "-t", "c:/slm/slickedit2008");
   dinfo_add(dir_list, "+t", "c:/dev/smac/bin");

   _str projName = strip_filename(_project_name, 'PE');
   switch(projName)
   {
   case "SlickC":
      dinfo_add(dir_list, "+t", "c:/slm/slickedit2008");
      break;

   case "pixo9lrb":
      dinfo_add(dir_list, "+t", "c:/dev/smac/pixo9");
      dinfo_add(dir_list, "+t", "c:/dev/smac/dx9cap");
      dinfo_add(dir_list, "-t", "c:/Program Files (x86)/Microsoft DirectX SDK (November 2007)/Include", "d3d10*.h");
      dinfo_add(dir_list, "-t", "c:/Program Files (x86)/Microsoft DirectX SDK (November 2007)/Include", "d3dx10*.inl");
      dinfo_add(dir_list, "-t", "c:/Program Files (x86)/Microsoft DirectX SDK (November 2007)/Include", "DXGIType.h");

      exclude_list = "c:/dev/smac/pixo9/bin/compiler_0.5/";
      break;

   case "pixo9xlrb":
      dinfo_add(dir_list, "+t", "c:/dev/smac/pixo9x");
      dinfo_add(dir_list, "+t", "c:/dev/smac/dx9cap");
      dinfo_add(dir_list, "-t", "c:/Program Files (x86)/Microsoft DirectX SDK (November 2007)/Include", "d3d10*.h");
      dinfo_add(dir_list, "-t", "c:/Program Files (x86)/Microsoft DirectX SDK (November 2007)/Include", "d3dx10*.inl");
      dinfo_add(dir_list, "-t", "c:/Program Files (x86)/Microsoft DirectX SDK (November 2007)/Include", "DXGIType.h");

      exclude_list = "c:/dev/smac/pixo9x/bin/compiler_0.5/";
      break;

   case "d3dref10_1":
      dinfo_add(dir_list, "+t", "c:/dev/smac/d3dref10_1");
      break;

   case "Alpine":
      dinfo_add(dir_list, "+t", "c:/slm/alpine/alpine-src");
      break;

   default:
      _str msg = "ssync(mikesart.e): No dir_list specified for " projName ".";
      _message_box(msg, "", MB_OK | MB_ICONEXCLAMATION);
      return;
   }

   // Get list of current project files.
   int projectfiles_list:[];
   ssync_getprojectfilelist(_project_name, projectfiles_list);

   // Create our temporary view for insert_file_list().
   int filelist_view_id;
   int orig_view_id = _create_temp_view(filelist_view_id);

   // Split out default file spec list into an array.
   _str file_specs[];
   split(file_spec_default_list, ';', file_specs);

   // Do an insert_file_list() for every dir_list + file_spec entry.
   for(i = 0; i < dir_list._length(); i++)
   {
      int j;
      DIRINFO dirInfo = dir_list[i];

      if(dirInfo.filespec != "")
      {
         _str cmd = dirInfo.opts " " maybe_quote_filename(dirInfo.dir "/" dirInfo.filespec);

         insert_file_list("-v +a +p " cmd);
      }
      else
      {
         for(j = 0; j < file_specs._length(); j++)
         {
            _str cmd = dirInfo.opts " " maybe_quote_filename(dirInfo.dir "/" file_specs[j]);

            insert_file_list("-v +a +p " cmd);
         }
      }
   }

   // Go through and kill any directories or files that match the exclude list.
   if(exclude_list != "")
   {
      _str excludes[];

      split(exclude_list, ";", excludes);

      for(i = 0; i < excludes._length(); i++)
      {
         _str exclude_str = stranslate(excludes[i], '\', '/');

         top();
         up();

         while(!search('^[ \t]+' _escape_re_chars(exclude_str), '@r' _fpos_case))
         {
            delete_line();
         }
      }
   }

   // Remove duplicate files.
   int file_list_hash:[];

   file_list_hash._makeempty();
   top();
   up();
   while(!down())
   {
      _str filename;

      get_line(filename);
      filename = strip(filename, 'B');

      if(projectfiles_list._indexin(filename))
      {
         // If this file is current in the project, set the projectfiles_list
         //  entry to 1 (so we don't remove it from the project) and then
         //  remove this entry from the temp buffer add list.
         projectfiles_list:[filename] = 1;

         delete_line();
         up();
      }
      else if(file_list_hash._indexin(filename) &&
         file_list_hash:[filename] != p_line)
      {
         // This is a new file and it's a duplicate so wack it.
         delete_line();
         up();
      }
      else
      {
         // New file: record the line number we first saw it at.
         file_list_hash:[filename] = p_line;
      }
   }

   // Create a list of files to remove from the project. Every hash entry
   //  in projectfiles_list[] that is a 0 should be removed.
   int value;
   _str filename;
   _str filelist_remove = "";
   int files_removed = 0;
   foreach(filename => value in projectfiles_list)
   {
      if(value == 0)
      {
         files_removed++;
         filelist_remove = filelist_remove " " maybe_quote_filename(filename);
      }
   }
   if(filelist_remove != "")
   {
      filelist_remove = strip(filelist_remove, 'B');

      int status = project_remove_filelist(_project_name, filelist_remove);
      if(status != 0)
      {
         _str msg = "Warning: Unable to remove files from project. " :+ get_message(status);
         _message_box(msg, "", MB_OK | MB_ICONEXCLAMATION);
      }
   }

   // Add the new list of files to our project.
   if(file_list_hash._length())
   {
      tag_add_viewlist(_GetWorkspaceTagsFilename(), filelist_view_id);
   }

   int project_filecount = projectfiles_list._length() - files_removed + file_list_hash._length();

   message("Ssync completed. files_added:" file_list_hash._length() :+
           " files_removed:" files_removed :+
           " total_files:" project_filecount);

   activate_window(orig_view_id);
}

static void dinfo_add(DIRINFO (&dir_list)[], _str opts, _str dir, _str filespec = "")
{
   int len = dir_list._length();

   dir_list[len].opts      = opts;
   dir_list[len].dir       = dir;
   dir_list[len].filespec  = filespec;
}

//=========================================================================
// ssync_getprojectfilelist(): Get list of project files.
//=========================================================================
static void ssync_getprojectfilelist(_str ProjectName, int (&projectfiles_list):[])
{
   _str filelist = "";
   int file_view_id = 0;
   int orig_view_id = p_window_id;

   GetProjectFiles(ProjectName, file_view_id);
   p_window_id = file_view_id;

   top();
   up();

   while(!down())
   {
      _str filename;

      get_line(filename);
      filename = strip(filename, 'B');

      if(filename != "")
      {
         projectfiles_list:[filename] = 0;
      }
   }

   p_window_id = orig_view_id;
   _delete_temp_view(file_view_id);
}

mikesart

  • Community Member
  • Posts: 56
  • Hero Points: 11
Re: automated project file adding
« Reply #1 on: April 12, 2008, 01:25:59 AM »
Oh, one more note. OSs that use forward slashes may need to modify the stranslate line:

Code: [Select]
  _str exclude_str = stranslate(excludes[i], '\', '/');
Haven't tested it anywhere but Windows, but I believe that's the only part which should potentially be an issue.

chrisant

  • Senior Community Member
  • Posts: 1410
  • Hero Points: 131
Re: automated project file adding
« Reply #2 on: April 12, 2008, 01:41:54 AM »
Nice, thanks for sharing this.  I have similar needs, and you've just saved me many hours, now I don't need to write that myself!  8)  HP++

mikesart

  • Community Member
  • Posts: 56
  • Hero Points: 11
Re: automated project file adding
« Reply #3 on: April 14, 2008, 05:11:01 PM »
Ah, thanks Chrisant. You've had a number of great & useful posts so it'd be cool if I accidentally did something that saved you some time. :)

I ran into a few questions while doing this that I was hoping some folks could shed some light on.

In project.e, insert_file_list() is being used with a few options that I couldn't find documentation on. +W (multiple file specs?), +o (OptimizeStats?), and +L. The +W could be useful for this macro so I'm wondering if these are not-ready-for-primetime deliberately-not-documented options or has it just not happened yet?

While writing _create_temp_view(), I found it extremely helpful to debug by using the "edit('+bi ' p_buf_id);" command to show the temporary buffer contents. Much better than the vsapi getline() + say() method I started with. Example below. I'm wondering if this is how other folks do this type of thing or is there a better way?

Code: [Select]
_command void blah() name_info(',')
{
   int filelist_view_id;
   int orig_view_id = _create_temp_view(filelist_view_id);

   insert_file_list("c:/*.*");

   edit('+bi ' p_buf_id);

   activate_window(orig_view_id);
   _delete_temp_view(filelist_view_id);
}

While debugging the edit() thing above, 'list_buffers -h' became my new friend. While tracking down my leaked Untitled buffers, it appears that the message list window has a couple of Untitled buffers open with just '(show all)' or '' as the content. Just wondering what the recommended behavior on this type of thing is? (Slickedit v13.0.0.0)

And this is more an observation that I couldn't find documented anywhere but found by the extremely scientific method of banging on my keyboard; It appears that ctrl+k clears the vsapi.dll window. (This would be MS Windows only I believe). This made say() a lot more useful to me.

Thanks!

chrisant

  • Senior Community Member
  • Posts: 1410
  • Hero Points: 131
Re: automated project file adding
« Reply #4 on: April 14, 2008, 05:52:38 PM »
Ah!  I hadn't found list_buffers -h yet, that's great.
That allowed me to confirm that vc.e leaks a buffer every time it captures output from source control commands.
Will post a separate thread on that, though (and report to support, of course).

hs2

  • Senior Community Member
  • Posts: 2761
  • Hero Points: 292
Re: automated project file adding
« Reply #5 on: April 14, 2008, 08:10:49 PM »
Hi, Dennis told some vsapi secrets here.
@mikesart: Good idea to 'edit' the temp buffer to check the contents and thanks for this ingenious hint !
HS2

chrisant

  • Senior Community Member
  • Posts: 1410
  • Hero Points: 131
Re: automated project file adding
« Reply #6 on: April 21, 2008, 06:46:27 AM »
Thanks, mikesart, for that macro.  I'm using it now, with a number of tweaks (attached).

Most notable:
  • Fixed case sensitivity in the hash array; systems with case insensitive filenames needed a tweak.
  • Converted the project dir list definitions to be a data table (array) rather than code; this will help if/when this macro is enhanced to allow runtime user configuration of the project dir lists.
  • Expanded the default set of file extensions, and added (expandable) shorthand notation for multiple default sets of file extensions (<sources>, <headers>, <slick>, etc).
  • All dir list entries support file spec lists (rather than requiring a separate dir list entry per explicit wildcard).
  • Expands env vars in directories.
  • Added some performance notes; will follow up on them later.

Your project dir lists should all be intact inside the #define MIKESART sections.  Mine are in the #define CHRISANT section.  Other users can replace those with their own list of project dir list definitions.  Hopefully I haven't broken anything for platforms with case-sensitive file systems.  :-[

@SlickTeam: The project support in SlickEdit is great.  However, setting up the file list for a project could be improved quite a bit (I gather that's already on your wish list).  Currently we have to choose between (1) entering lots of individual wildcard entries (and exclusions, oh my) and getting poor performance for large projects (due to scanning the file system once per session), or (2) manually updating the project file list (have to know/remember the paths and long lists of extensions, and exclusions, and iterate through them manually) but getting good performance for large projects (loads a cached file list rather than scanning the file system).  This macro is a simplistic compromise between (1) and (2), but the capability in this macro (or something better :)) should ideally be built into SlickEdit.  This macro still has a drawback in that it runs in the foreground, so for large projects it is still expensive to run, but at least we can control when it runs.  I'd be happy to discuss more about project definitions for large projects (20k+ files, 10-30 wildcards, exclusions, supporting tag files, etc) further in the forums or in private emails if/when you want to enhance the project file list support (you have my email address).  In the meantime, this macro is pretty good workaround (albeit with part of the project definition living outside the actual project file), so it's not like we're dying without project enhancements.  ;)

Thanks,
Chris
« Last Edit: April 21, 2008, 06:48:26 AM by chrisant »

Kung Foo

  • Community Member
  • Posts: 29
  • Hero Points: 0
Re: automated project file adding
« Reply #7 on: June 11, 2008, 10:33:12 AM »
Nice piece of a script!

I tried to use it, but I get error "Expecting '}'" when I try to load the macro by pressing F12. The cursor is moved just before 'h' in foreach:
Code: [Select]
   foreach(filename => value in projectfiles_list)

I suspect that its the directory config table, but I do not see any error in it:
Code: [Select]
static PROJINFO projects[] =
{
    {
       'SlickEdit',
       {
          { '+t', 'c:\mydir' }
       }
    }
};

Any idea on what this means?

I'm using SlickEdit Version 12.0.3.0 in Windows XP.

Kung Foo

  • Community Member
  • Posts: 29
  • Hero Points: 0
Re: automated project file adding
« Reply #8 on: June 11, 2008, 01:28:22 PM »

Hmm, I tried to find documentation on the foreach statement used in this script, but I did not find such!
Is it supported in Slick-C (any more)?

Lisa

  • Senior Community Member
  • Posts: 238
  • Hero Points: 24
  • User-friendly geek-speak translator extraordinaire
Re: automated project file adding
« Reply #9 on: June 11, 2008, 02:13:49 PM »
The foreach statement is not supported in 12.0.3, but was added in SlickEdit 2008 (v13) along with several more Slick-C enhancements. Here is the documentation on the statement:


The foreach statement works with arrays, hash tables, strings (same as Bourne shell), structs (iterates over the fields of the struct), and classes (if instance of sc.lang.llterable, otherwise like structs). The syntax of foreach is:
Code: [Select]
foreach ( [ k => ] v in a ) {
statements;
}

Example:
Code: [Select]
void printStats(int (&statistics):[]) 
{
   foreach (auto name => auto count in statistics) {
      say("testForeach: "name"="count);
   }
   int i,j=0;
   foreach (i in range(10, 20, 2)) {
      say("printStats: sequence["j++"]="i);
   }
}

There is an optional key which is useful for hash tables. The value can be omitted (key=> . in ht):
Code: [Select]
foreach( key => value in ht ) {
      statements;
   }

Both value and index can be auto-declared using the auto keyword. If value is auto-declared, its type will be inferred from the type of the collection. The implementation uses _nextel().


These docs will be in the upcoming 13.0.1 release.

Kung Foo

  • Community Member
  • Posts: 29
  • Hero Points: 0
Re: automated project file adding
« Reply #10 on: June 12, 2008, 07:42:49 AM »

Thanks Lisa. I was not aware of that "foreach" statement situation.

Seems that the script has some problems with some directories and/or file names.
I have some quite long directory paths, and the script does not seem to cope with these. Some of the leaf directories are shown as files(!), and some of the files are shown in folders "j:\..\.." and "j:\macro\" (neither of which do not actually exist).

Does the script work OK for others?

PS: I rewrote the foreach loop to a normal for-loop, since I do not have the new version of Slickedit:
Code: [Select]
   for (filename._makeempty();;) {
      projectfiles_list._nextel(filename);
      if (filename._isempty()) break;
      if(projectfiles_list:[filename] == 0)
      {
         files_removed++;
         //$ PERF: (chrisant) String growth has exponential cost here unless
         // Slick-C is using geometric growth under the covers.  However, the
         // number of files removed should typically be small, so this may be
         // acceptable.
//say('remove: "'filename'"');
         filelist_remove = filelist_remove " " maybe_quote_filename(filename);
      }
   }

chrisant

  • Senior Community Member
  • Posts: 1410
  • Hero Points: 131
Re: automated project file adding
« Reply #11 on: June 12, 2008, 04:17:30 PM »
Could you give some specific examples, Kung?
And can you show your table of paths?

The macro is working for a 20,000+ file workspace that I have, with directories up to about 180 characters deep.

Kung Foo

  • Community Member
  • Posts: 29
  • Hero Points: 0
Re: automated project file adding
« Reply #12 on: June 13, 2008, 06:46:36 AM »

Sure, I tried with a simple test setup, and the updated script where the "foreach" is replaced with "for" equivalent (see attachment).

I attached the slickedit project file hierarchy view pictures after first update (first ssync call), and after second update (without any changes to the "project").

As you can see, the first update works OK. But when I make the second update, it all goes terribly wrong. E.g. the "level_6_qwertyuiop" directory is placed to the root, and the level 5 and 6 file names are wrong.

Please tell me it's the modifications I made to the script - I'm not much of a Slick-C coder  ;D
Does the script version I made work for you?

chrisant

  • Senior Community Member
  • Posts: 1410
  • Hero Points: 131
Re: automated project file adding
« Reply #13 on: June 13, 2008, 07:41:31 PM »
I cannot reproduce that on Slick 13.0.0 r29 using your copy of the macro.
First update picks up the files properly.
Second update results in no changes.

I don't have Slick 12.0.3 handy right now so I can't take a look at the moment.

I've attached my latest version of the macro and added an $Id$ tag at the top (now at revision #3).
- Added support for relative paths.
- Added optional debug output.
- Uses for() instead of foreach() for compatibility.

I plan to move the data table into a separate .txt file (or maybe .xml file), so that it becomes possible to just load a new version of the macro without needing to copy/paste your project definitions.
If someone would like to help debug what's going wrong on Slick 12.0.3 that would be nice.  Or I will probably get around to it in the next couple of weeks (day job very busy, don't have that kind of time to poke around debugging mysterious stuff at the moment, sorry :().

Kung Foo

  • Community Member
  • Posts: 29
  • Hero Points: 0
Re: automated project file adding
« Reply #14 on: June 14, 2008, 07:24:15 AM »
Thanks chrisant, I think this helps to point out the root cause.
Seems that there must be something wrong with the SlickEdit version 12.0.3, or my setup of it (which should be somewhat standard though).
I just noticed that there's a hotfix pack available for SlickEdit 12.0.3, which I have not yet tried to install. I'll try that monday when I get back to work..

Thanks for updating the script to at least compile also with 12.0.3.
And it sounds like a nice idea that the configuration would be moved to a separate file.