New code completion remarks/issues

[ollydbg]

Ok, tiny fixed with ReadClsNames in parserthread.
for broken codes below:

#include <iostream>

using namespace std;
typedef class qq
{
   int x;
   int y;

}xx
int main()
{
   cout << "Hello world!" << endl;
   return 0;
}

Note:This code can't be compiled in G++.

[blueshake]

Exactly,but I refer to visual assist,its parser do what the patch do.

[ollydbg]

Exactly,but I refer to visual assist,its parser do what the patch do.

I have applied your patch in my local copy, it works. . Thanks for your effort!

[blueshake]

It seems

Code Select Expand

Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args); can not work for the test codes below.

Code Select Expand

#include <iostream>
#include <iostream>
using namespace std;
std::vector<string> ssss;
#define  xxxxxxxxx 12e
typedef class qq
{
    int x;
    int y;
}eeee;
typedef unsigned int wwwwwwwww;
www|
int main()
{

    cout << "Hello world!" << endl;
    return 0;
}


In the "|" position , the suggestion list flash and can not show.If I comment the statement

Code Select Expand

typedef unsigned int wwwwwwwww;,it works again.

[ollydbg]

@blueshake
That's something related in the nativeparser.cpp, and the code you cited. I will exam that.

[blueshake]

@blueshake
That's something related in the nativeparser.cpp, and the code you cited. I will exam that.

Nice to hear that.Thanks.

[ollydbg]

I can only understand a little about the FindAIMatches() function.

For example, your test code:

Code Select Expand


typedef unsigned int wwwww;

wwww|


You will see in the symbol tree, the "wwwww" is a global "typedef". which is quite different if your code is like below:

Code Select Expand


typedef class qq
{
    int x;
    int y;
}eeee;

then, "eeee" in the symbol tree is of type "class".


The piece of code in FindAIMatches()

Code Select Expand


if (local_result.size() == 1)
    {
        int id = *local_result.begin();
        Token* token = tree->at(id);

        if (token->m_TokenKind == tkTypedef)
        {
            std::queue<ParserComponent> type_components;
            BreakUpComponents(parser, token->m_ActualType, type_components);

            while(!components.empty())
            {
                ParserComponent comp = components.front();
                components.pop();
                type_components.push(comp);
            }

    #if DEBUG_CC_AI
            if (s_DebugSmartSense)
            #if wxCHECK_VERSION(2, 9, 0)
                Manager::Get()->GetLogManager()->DebugLog(F(_T("Replacing %s to %s"), token->m_Name.wx_str(), token->m_ActualType.wx_str()));
            #else
                Manager::Get()->GetLogManager()->DebugLog(F(_T("Replacing %s to %s"), token->m_Name.c_str(), token->m_ActualType.c_str()));
            #endif
    #endif
            return FindAIMatches(parser, type_components, result, parentTokenIdx, noPartialMatch, caseSensitive, use_inheritance, kindMask, search_scope);
        }

    }


this means, when you only get one match in the current context, then for example, you are entering

Code Select Expand

www

This time, the only matched string is "wwwww" will be replace with "unsigned int" and the total line will be reparsed. (which means the FindAIMatches will be called iteratively).



But I think at this test code, this typedef replacement is useless. I can't find a test case that this replacement is meaningful.

Can someone explain more about this? Thanks.

[MortenMacFly]

Guys, I am a bit lost and trying to catch up with all the posts. Am I rioght, that these two patches have to be applied together?

Patch 1:

Code Select Expand


Index: tokenizer.cpp
===================================================================
--- tokenizer.cpp (revision 5834)
+++ tokenizer.cpp (working copy)
@@ -216,17 +216,22 @@

bool Tokenizer::SkipToCharBreak()
{
-  if (PreviousChar() != '\\')
-      return true;
-  else
-  {
-      // check for "\\"
-      if (   ((m_TokenIndex - 2) >= 0)
-          && ((m_TokenIndex - 2) <= m_BufferLen)
-          && (m_Buffer.GetChar(m_TokenIndex - 2) == '\\') )
-          return true;
-  }
-  return false;
+    if (PreviousChar() != '\\')
+        return true;
+    else
+    {
+        // check for "\\"
+        unsigned int numBackslash = 2;
+        while(   ((m_TokenIndex - numBackslash) >= 0)
+            && ((m_TokenIndex - numBackslash) <= m_BufferLen)
+            && (m_Buffer.GetChar(m_TokenIndex - numBackslash) == '\\') )
+            ++numBackslash;
+        if(numBackslash%2==1) // number of back slash(include current char ) is even
+            return true;     // eg: "\""
+        else
+            return false;    // eg: "\\""
+    }
+    return false;
}


Patch 2:

Code Select Expand

Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- src/plugins/codecompletion/parser/parserthread.cpp (revision 5825)
+++ src/plugins/codecompletion/parser/parserthread.cpp (working copy)
@@ -1768,7 +1768,7 @@
     Manager::Get()->GetLogManager()->DebugLog(F(_("HandleTypedef() : Adding typedef: name='%s', ancestor='%s'"), components.front().c_str(), ancestor.c_str()));
#endif
//    Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);
-    Token* tdef = DoAddToken(tkClass, components.front(), lineNr, 0, 0, args);
+    Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);
     if (tdef)
     {
         if (!is_function_pointer)
@@ -1796,7 +1796,7 @@
                 m_Tokenizer.UngetToken();
                 break;
             }
-        else if (wxIsalpha(current.GetChar(0)))
+        else if (wxIsalpha(current.GetChar(0)) && (m_Tokenizer.PeekToken() == ParserConsts::semicolon || m_Tokenizer.PeekToken() == ParserConsts::comma))
         {
#if PARSERTHREAD_DEBUG_OUTPUT
             Manager::Get()->GetLogManager()->DebugLog(F(_T("ReadClsNames() : Adding variable '%s' as '%s' to '%s'"), current.c_str(), m_Str.c_str(), (m_pLastParent?m_pLastParent->m_Name.c_str():_T("<no-parent>"))));
@@ -1811,7 +1811,10 @@
             }

         }
-        else // unexpected
+        else// unexpected
+        {
+            m_Tokenizer.UngetToken();
             break;
+        }
     }
}


???

I think the function name "SkipToCharBreak" is confusion, because it doesn't really do a skip.
So, I suggest using another name like :  IsRealCharBreak or IsCharBreak or IsEscapeChar

That's right and probably was a typo. I'll change that certainly.

[ollydbg]

@MortenMacFly
Yes, the two patches can both be applied.

But, blueshake's patch of changing the code :

Code Select Expand


-    Token* tdef = DoAddToken(tkClass, components.front(), lineNr, 0, 0, args);
+    Token* tdef = DoAddToken(tkTypedef, components.front(), lineNr, 0, 0, args);


should be reconsidered, because a small bug he states in this post:
http://forums.next.codeblocks.org/index.php/topic,11187.msg76788.html#msg76788

[blueshake]

hello,another bad news.
It seems some bugs hide in the codecompletion.In the file codecompletion.cpp,in this place:

Code Select Expand

           // Remove duplicate items
           for (size_t i=0; i<items.Count()-1; i++)
               if (items.Item(i)==items.Item(i+1))
                   items.RemoveAt(i);
Manager::ge|

when you type Mana,yes,the suggestion show,then you type ::,yes work again,but when you type ge,the issue come out,the suggestion list flash again,and not show.what is going on here,the suggestion list is try to show,but cancle by something.

Note:
I just use the latest cc,change nothing.
Edit:

Code Select Expand

Manager::Get()->|
And in the "|" position, the cc not work.

[ollydbg]

hello,another bad news.
It seems some bugs hide in the codecompletion.In the file codecompletion.cpp,in this place:

Code Select Expand

           // Remove duplicate items
           for (size_t i=0; i<items.Count()-1; i++)
               if (items.Item(i)==items.Item(i+1))
                   items.RemoveAt(i);
Manager::ge|

when you type Mana,yes,the suggestion show,then you type ::,yes work again,but when you type ge,the issue come out,the suggestion list flash again,and not show.what is going on here,the suggestion list is try to show,but cancle by something.

Note:
I just use the latest cc,change nothing.
Edit:

Code Select Expand

Manager::Get()->|
And in the "|" position, the cc not work.

I can confirm both the two bugs.
Is it possible to create a simple test case. So that I can debug it. Currently, Opening the codeblocks's whole source code is not necessary.

Also: I think it's time to create some standard test code for codecompletion now. Any comments?

Also 1: @morten, in the current CC code, there are many code using the preprocessor guard.like:

Code Select Expand


#if PARSERTHREAD_DEBUG_OUTPUT
           Manager::Get()->GetLogManager()->DebugLog(F(_T("HandlePreprocessorBlocks() : Special case \"#if 0\" not skipped.")));
#endif


Is there any way we can just clean up these code, and use something like this:

Code Select Expand


#if PARSERTHREAD_DEBUG_OUTPUT
#define DebugOutput(a,b,c)  Manager::Get()->GetLogManager()->DebugLog(a,b,c)
#else
#define DebugOutput(a,b,c)  
#endif


So, the the source code, we can only use this:

Code Select Expand


DebugOutput(F(_T("HandlePreprocessorBlocks() : Special case \"#if 0\" not skipped.")));

This will make the source code more clean.


Also 2:

There are many variable names which are not following the source style guard. like:

Code Select Expand


   Parser parser(this);
   parser.ParseBufferForFunctions(control->GetTextRange(0, pos));

   wxArrayString funcs;
   TokensTree* tmptree = parser.GetTempTokens();

   // look for implementation functions that enclose our current line number
   for(size_t i = 0; i < tmptree->size();i++)


the tmptree should be tmpTree.

Also, there are many other variables should be renamed. such as:

Code Select Expand


// find current function's namespace so we can include local scope's tokens
   // we ' ll get the function's token (all matches) and add its parent namespace
   TokenIdxSet scope_result;
   TokenIdxSet proc_result;
   if (FindCurrentFunctionToken(editor, proc_result) != 0)
   {
       for (TokenIdxSet::iterator it = proc_result.begin(); it != proc_result.end(); ++it)
       {
           Token* token = parser->GetTokens()->at(*it);
           if (!token)
               continue;
           scope_result.insert(token->m_ParentIndex);
#if DEBUG_CC_AI
           if (s_DebugSmartSense)
           {
               Token* parent = parser->GetTokens()->at(token->m_ParentIndex);
               Manager::Get()->GetLogManager()->DebugLog(_T("Adding search namespace: ") + (parent ? parent->m_Name : _T("Global namespace")));
           }
#endif
       }
   }


These name should be "scopeResult" and so on...

[ollydbg]

@blueshake:

look at the Get function prototype, it is a static function.

Code Select Expand


class DLLIMPORT Manager
{
    static Manager* Get();
}

but, in the symbol tree debug dialog, you will see that the

type is Manager *
actual type is Manager

That's may be the cause of bug...

[blueshake]

@blueshake:

look at the Get function prototype, it is a static function.

Code Select Expand


class DLLIMPORT Manager
{
   static Manager* Get();
}

but, in the symbol tree debug dialog, you will see that the

type is Manager *
actual type is Manager

That's may be the cause of bug...

But it worked before.I updated the local copy,and it don't work.
look at these codes.The static are ignored.

Code Select Expand

       else if (token==ParserConsts::kw_static ||
           token==ParserConsts::kw_virtual ||
           token==ParserConsts::kw_inline)
       {
           // do nothing (skip it)
       }

[blueshake]

anther issue.

Code Select Expand

std::string ss;
ss. not work


Code Select Expand


string ss;
ss.work
parser error:

Code Select Expand

string ss("abc");
ss.
it should be a variable,but ss is parsed as function.

[attachment deleted by admin]

[MortenMacFly]

Just a tiny remark: For testing if the current modification (a.k.a patch) does not break something else please make sure that after applying the patch do the following:
1.) Open Code::Blocks having the debug log enabled.
2.) Open CodeBlocks.cbp (project files of core and core plugins)
3.) Wait, until the parser has completed.
3.) Check the debug log for the number of tokens, it should look like the following:
Parsing stage done (1570 total parsed files, 92637 tokens in 1 minute(s), 8.922 seconds).
The tokens should be round about 90000 - 100000 (depending on the patch level of C::B).

Furthermore: I am still happy to collect sample projects for testing CC I already have some...