beautifying obfuscated code

[storkinsj]

Hi,
    I am enjoying automating slickedit and am hoping to continue on my journey that I began this week to take an entire code-base and make it conform to a team standard (which we are still defining).
    I think in my mind, the beautify command is the right thing to use and I have figured out how to get it to run on nested project folders. The problem I see is that in the interest of making it behave correctly and not frustrate us, it is cautious. My goal is a bit different than editting right now so it will probably need more tweaking than I am able to figure out.

     I would like to point slickedit at a bunch of obfuscated c code or javascript code and have it create newlines when necessary and make it look as though it were logically indented correctly.  In the case of javascript, this is amazingly helpful because we get compressed javascript "libraries" that get pulled into web pages. When debugging... it's easy to decide which line to put the breakpoint on- there is only one line for a 400k or so file LOL!  Deobfuscating my web site's javascript is a really good way to accomplish library debugging.
     My main mission however is making a large body of c/c++ conform to coding standards. If slickedit is not the right tool to do that I'd rather know sooner than later. I see that Patrick figured out how to quickly create a filter that shells out to a command line and uses as CLI tool to accomplish this. Obviously I would eliminate VS from that workflow if I knew of such a tool. I'd much rather use VS than a command line tool; this way I can make the code conform to Beautify settings I specify... and later while editing maintain the standard almost automatically.

     I have figured out that adaptive formatting should be off as that would make the code appear like it already does. Beyond that I"m open to suggestions!

Thank you.

[Graeme]

Not sure if this is useful but the following macro beautifies all files in a project.  You could adapt it to beautify all open files.

Code: [Select]

_command void xbeautify_project(boolean ask = true, boolean no_preview = false, boolean autosave = true) name_info(',')
{
   _str files_to_beautify [];

   //_GetWorkspaceFiles(_workspace_filename, files_to_beautify);
   _getProjectFiles( _workspace_filename, _project_get_filename(), files_to_beautify, 1);

   if (ask && !no_preview) {
      activate_preview();
   }

   int k;
   for (k = 0; k < files_to_beautify._length(); ++k) {
      if (ask) {

         if (!no_preview) {
            struct VS_TAG_BROWSE_INFO cm;
            tag_browse_info_init(cm);
            cm.member_name = files_to_beautify[k];
            cm.file_name = files_to_beautify[k];
            cm.line_no = 1;
            cb_refresh_output_tab(cm, true, false, false);
            _UpdateTagWindowDelayed(cm, 0);
         }

         _str res = _message_box("Beautify " :+ files_to_beautify[k], "Beautify project", MB_YESNOCANCEL|IDYESTOALL);
         if (res == IDCANCEL) return;
         if (res == IDNO) continue;
         if (res == IDYESTOALL) ask = false;
      }

      if (edit("+B " :+ files_to_beautify[k]) == 0) {
         beautify();
         if (autosave) save();
      }
      else
      {
         edit(files_to_beautify[k]);
         beautify();
         if (autosave) save();
         quit();
      }
   }
}


[Clark]

If you can configure SlickEdit's beautifier settings the way you need, SlickEdit will definitely be able to do the job. Worst case, you might need a macro like the one above. SlickEdit's file manager is another way to filter specific files. It's a little crude but it does the job. I typically use SlickEdit's file manager for things like this. I'm even too lazy to write a macro for this. I recently ran the 'edit-with "%f"  "-#check-line-endings U"' command through our source code using the SlickEdit file manager. Some older versions of Visual Studio trashed some of our code.

[storkinsj]

 hi folks,
      thanks for replying but if you read my question you'll see that I already figured out how to go through all the files in the folder and in fact I already posted that answer elsewhere on the forum before asking this question.

 This is completely about getting the settings correct and whether the editor can handle  deobfuscation.  can I get the editor to be a little less cautious and add line feeds etc. when necessary… Especially with regards to C++ code.

[Graeme]

Well your original post is vague, rambling and too long.  In fact you said this

My main mission however is making a large body of c/c++ conform to coding standards.

I still can't really tell what you're looking for.  Why don't you try it out and see what it does.  Slickedit can beautify while typing.  See the C++ "general" options.

[storkinsj]

I'll quote some of the important parts:

-I would like to get code to conform to coding standards.
-The problem I see is that in the interest of making it behave correctly and not frustrate us, slickedit is (too) cautious.
-I would like to have it create newlines when necessary and make it look as though it were logically indented correctly

I can say in addition that when I eliminate all newlines and extra spaces, Slickedit leaves the code as "one single line".

So to me that is cautious. I mentioned why I think it does this, but I'd like to figure out what settings will make it simply take the code and make it adhere from an indentation perspective with the code logic. It is quite possible that this is not possible.

[Graeme]

As far as I can see you didn't say why you think it's cautious.

For javascript see this
https://reverseengineering.stackexchange.com/questions/1436/analyzing-highly-obfuscated-javascript

For C++, can you post an example of before and after and what the problem is, and also explain how your C++ code gets to be all on one line.

[storkinsj]

Hi Graeme,
   All of my posts mention new lines as an issue- especially the last one.

I am attaching a sample. Beautify (by default) does not add any newlines to the attached example.

   I just dug through all of the beautify options, of which there are plenty. I did find a section labeled "blank lines" and just a couple of options that can add a newline. I will try adding at least one line for every "blank line" setting. Blank lines are not the same as new lines but it's a good start.

   The reason for the longer initial question is to see if anyone else has done a project like this- so far it sounds like no.

[storkinsj]

BTW thanks for the very interesting article on deobfuscating javascript! 
In most cases I am not experiencing "true obfuscation". More or less ALL white space is removed to optimize download / web performance.

[Graeme]

Hi

Slickedit beautify doesn't like your patternize file, the comments at the start disappear amongst  other things, on my system anyway.  I can see versions of this file on the internet but they have normal formatting with newlines.  How did your version get to have no newlines?

[patrick]

You can't pull C/C++ into a single line like you do with Javascript because of preprocessor directives and comments.  Both of those process to the end of the line, which eats up the rest of the source file.  I assume Javascript minifiers are careful to either strip line comments or turn them into block comments if comments are being left in.

Probably the best bet source for obfuscated C/C++ code to test with is the Obfuscated C contests.  As you say, there's plenty of minified Javascript examples. 

We use obfuscated code like that as a torture test, but it is the minority case in our test suite. One of the problems you can run into is that some of the rules look at whether some constructs are on the same line in the original source to decide how to apply a setting, which isn't helpful for this type of file.  So I'd recommend creating a separate profile for cleaning up files that are obfuscated/minified vs. regular code that's just gotten messy.   In this profile, would go to the Newlines heading and turn off any "leave X on the same line" or "Allow one line X" settings, and enable any settings that add newlines to things like empty blocks.  Enabling some of the rules under "Blank Lines" can also get some spacing between things like type and var decls which can help.

[jporkkahtc]

http://universalindent.sourceforge.net/
https://github.com/danblakemore/universal-indent-gui

A command line tool that aggregates several other code beautifiers/stylers and provide a common UI to control them.

It would be nice if it had a slickedit plugin :-)

Looks like it might not be maintained anymore though, but a neat idea.

[storkinsj]

DOH! Patrick you are right on the money. What I provided is not even a program. Now that I see I compressed the entire program into a single preprocessor definition I will give slickedit a chance without making that mistake.  I will check out the obfuscated C site- sounds like you have which gives me great hope lol!

@jporkkahtc that is exactly what I want but I'll retry my experiment. Then I will certainly look into your tool.

 The macros I've seen from Patrick in other sections seem like they could easily load the entire buffer into a response file... pass it to the tool... and pass it back into your file buffer. The only thing is that of course it would need to be configured externally. Also, it would not be able to correct as you type.

Thank you everyone.