Pointer/Array not recognized

[daniloz]

Hi All,

I don't know exactly what's the technical name of what I'm doing, but look at the following code:

Code Select Expand


float *pFloat;
float **pFloat2;
float (*pFloatPar)[3];

int main()
{
pFloat;
pFloat2;
pFloat
}

I've declared pFloatPar as a double array in a way that the compiler knows about the second dimension size and then I can write pFloatPar[2][1].

However, the actual code completion doesn't recognize the declaration (see the attached screenshot). My assumption is that CC cannot correctly parse the parenthesis...

[ollydbg]

I have put your code in our parsertest project, and the result looks below:

Code Select Expand


--------------M-a-i-n--L-o-g--------------

000001. ParserDummy() : ParserBase() : Instantiation of Parser object.
000002. InitTokenizer() : m_Filename='test.h', m_FileSize=108.
000003. Init() : m_Filename='test.h'
000004. test.h
000005. Parse() : Parsing 'test.h'
000006. DoParse() : Loop:m_Str='', token='float'
000007. DoParse() : Loop:m_Str='float ', token='*'
000008. DoParse() : Loop:m_Str='float ', token='pFloat'
000009. DoAddToken() : Created token='pFloat', file_idx=1, line=1, ticket=
000010. GetActualTokenType() : Searching within m_Str='float'
000011. GetActualTokenType() : Compensated m_Str='float'
000012. GetActualTokenType() : Found 'float'
000013. DoAddToken() : Prepending ''
000014. DoAddToken() : Added/updated token 'pFloat' (0), kind 'variable', type 'float*', actual 'float'. Parent is  (-1)
000015. DoParse() : Loop:m_Str='float', token=';'
000016. DoParse() : Loop:m_Str='', token='float'
000017. DoParse() : Loop:m_Str='float ', token='*'
000018. DoParse() : Loop:m_Str='float ', token='*'
000019. DoParse() : Loop:m_Str='float ', token='pFloat2'
000020. DoAddToken() : Created token='pFloat2', file_idx=1, line=2, ticket=
000021. GetActualTokenType() : Searching within m_Str='float'
000022. GetActualTokenType() : Compensated m_Str='float'
000023. GetActualTokenType() : Found 'float'
000024. DoAddToken() : Prepending ''
000025. DoAddToken() : Added/updated token 'pFloat2' (1), kind 'variable', type 'float**', actual 'float'. Parent is  (-1)
000026. DoParse() : Loop:m_Str='float', token=';'
000027. DoParse() : Loop:m_Str='', token='float'
000028. ReadParentheses(): (* pFloatPar), line=3
000029. DoParse() : Loop:m_Str='', token=';'
000030. DoParse() : Loop:m_Str='', token='int'
000031. DoParse() : Loop:m_Str='int ', token='main'
000032. ReadParentheses(): (), line=5
000033. HandleFunction() : Adding function 'main': m_Str='int '
000034. HandleFunction() : name='main', args='()', peek='{'
000035. HandleFunction() : !(Ctor/Dtor) 'main', m_Str='int ', localParent='<none>'
000036. HandleFunction() : Adding function 'main', ': m_Str='int ', enc_ns='nil'.
000037. HandleFunction() : Add token name='main', args='()', return type='int '
000038. GetBaseArgs() : args='()'.
000039. GetBaseArgs() : baseArgs='()'.
000040. DoAddToken() : Created token='main', file_idx=1, line=5, ticket=
000041. GetActualTokenType() : Searching within m_Str='int'
000042. GetActualTokenType() : Compensated m_Str='int'
000043. GetActualTokenType() : Found 'int'
000044. DoAddToken() : Prepending ''
000045. DoAddToken() : Added/updated token 'main' (2), kind 'function', type 'int', actual 'int'. Parent is  (-1)
000046. ParserDummy() : ~ParserBase() : Destruction of Parser object.


--------------T-r-e-e--L-o-g--------------

000047. float* pFloat [1,0]
000048. float** pFloat2 [2,0]
000049. main() [5,5]


--------------L-i-s-t--L-o-g--------------

000050. variable float* pFloat [1,0]
000051. variable float** pFloat2 [2,0]
000052. function int main() [5,5]



Yes. The parser just skip the "[3]".

The proposed fix is try to peek the token after the right parenthesis, and if it is a "[", then we can guess that is is a second level pointer.

[daniloz]

Yes. The parser just skip the "[3]".

Even if the parser does so, the result should be float *pFloatPar; and the parser should treat it as a pointer to a float. This would, of course, be wrong, but it seems the parser is just dropping this symbol completely.

The proposed fix is try to peek the token after the right parenthesis, and if it is a "[", then we can guess that is is a second level pointer.

That sounds good to me, however I'm not that familiar with the CC plugin to try a patch, sorry... ;-)

[ollydbg]

The proposed fix is try to peek the token after the right parenthesis, and if it is a "[", then we can guess that is is a second level pointer.

That sounds good to me, however I'm not that familiar with the CC plugin to try a patch, sorry... ;-)

I think I will take some time to fix it. 

[daniloz]

I think I will take some time to fix it. 

That'd be great... Thanks! 

[ollydbg]

This is a dirty hack on this

Code Select Expand


Index: E:/code/cb/cb_trunk/src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
--- E:/code/cb/cb_trunk/src/plugins/codecompletion/parser/parserthread.cpp (revision 7561)
+++ E:/code/cb/cb_trunk/src/plugins/codecompletion/parser/parserthread.cpp (working copy)
@@ -506,11 +506,11 @@
     m_LastToken.Clear();
     m_LastUnnamedTokenName.Clear();

-    // Notice: clears the queue "m_EncounteredTypeNamespaces"
+    // Notice: clear the queue "m_EncounteredTypeNamespaces"
     while (!m_EncounteredTypeNamespaces.empty())
         m_EncounteredTypeNamespaces.pop();

-    // Notice: clears the queue "m_EncounteredNamespaces"
+    // Notice: clear the queue "m_EncounteredNamespaces"
     while (!m_EncounteredNamespaces.empty())
         m_EncounteredNamespaces.pop();

@@ -942,7 +942,8 @@
                         int pos = peek.find(ParserConsts::ptr);
                         if (pos != wxNOT_FOUND)
                         {
-                            if (m_Tokenizer.PeekToken().GetChar(0) == ParserConsts::opbracket_chr)
+                            peek = m_Tokenizer.PeekToken();
+                            if (peek.GetChar(0) == ParserConsts::opbracket_chr)
                             {
                                 arg.Trim(true).RemoveLast();
                                 //wxString token = arg.Mid(pos+1,)
@@ -953,6 +954,18 @@
                                     HandleFunction(arg);
                                 }
                             }
+                            else if (peek.GetChar(0) == ParserConsts::semicolon_chr)
+                            {
+                                // we meet such mode:
+                                // AAA (*BBB)[ccc];
+                                // The "[ccc]" was skipped by Tokenizer, so we meet the semicolon
+                                m_Str << token<<_T("*");
+                                //strip the "*" from arg.
+                                arg.Trim(true).RemoveLast();
+                                arg.Remove(0, pos+1); //remove the heading "*"
+                                if (m_Options.handleVars)
+                                    DoAddToken(tkVariable, arg, m_Tokenizer.GetLineNumber());
+                            }
                         }
                         else // wxString arg = m_Tokenizer.GetToken(); // eat args ()
                             m_Str = token + arg;
@@ -2646,6 +2659,7 @@
         else if (tmp==ParserConsts::gt)
         {
             --nestLvl;
+            m_TemplateArgument.Trim();
             m_TemplateArgument << tmp;
         }
         else if (tmp==ParserConsts::semicolon)
@@ -2658,7 +2672,7 @@
         else if (tmp.IsEmpty())
             break;
         else
-            m_TemplateArgument << tmp;
+            m_TemplateArgument << tmp <<_T(" ");
         if (nestLvl <= 0)
             break;
     }



Well, the log is below:

--------------M-a-i-n--L-o-g--------------

000001. ParserDummy() : ParserBase() : Instantiation of Parser object.
000002. InitTokenizer() : m_Filename='test.h', m_FileSize=108.
000003. Init() : m_Filename='test.h'
000004. test.h
000005. Parse() : Parsing 'test.h'
000006. DoParse() : Loop:m_Str='', token='float'
000007. DoParse() : Loop:m_Str='float ', token='*'
000008. DoParse() : Loop:m_Str='float ', token='pFloat'
000009. DoAddToken() : Created token='pFloat', file_idx=1, line=1, ticket=
000010. GetActualTokenType() : Searching within m_Str='float'
000011. GetActualTokenType() : Compensated m_Str='float'
000012. GetActualTokenType() : Found 'float'
000013. DoAddToken() : Prepending ''
000014. DoAddToken() : Added/updated token 'pFloat' (0), kind 'variable', type 'float*', actual 'float'. Parent is  (-1)
000015. DoParse() : Loop:m_Str='float', token=';'
000016. DoParse() : Loop:m_Str='', token='float'
000017. DoParse() : Loop:m_Str='float ', token='*'
000018. DoParse() : Loop:m_Str='float ', token='*'
000019. DoParse() : Loop:m_Str='float ', token='pFloat2'
000020. DoAddToken() : Created token='pFloat2', file_idx=1, line=2, ticket=
000021. GetActualTokenType() : Searching within m_Str='float'
000022. GetActualTokenType() : Compensated m_Str='float'
000023. GetActualTokenType() : Found 'float'
000024. DoAddToken() : Prepending ''
000025. DoAddToken() : Added/updated token 'pFloat2' (1), kind 'variable', type 'float**', actual 'float'. Parent is  (-1)
000026. DoParse() : Loop:m_Str='float', token=';'
000027. DoParse() : Loop:m_Str='', token='float'
000028. ReadParentheses(): (* pFloatPar), line=3
000029. DoAddToken() : Created token=' pFloatPar', file_idx=1, line=3, ticket=
000030. GetActualTokenType() : Searching within m_Str='float*'
000031. GetActualTokenType() : Compensated m_Str='float*'
000032. GetActualTokenType() : Found 'float'
000033. DoAddToken() : Prepending ''
000034. DoAddToken() : Added/updated token ' pFloatPar' (2), kind 'variable', type 'float*', actual 'float'. Parent is  (-1)
000035. DoParse() : Loop:m_Str='float*', token=';'
000036. DoParse() : Loop:m_Str='', token='int'
000037. DoParse() : Loop:m_Str='int ', token='main'
000038. ReadParentheses(): (), line=5
000039. HandleFunction() : Adding function 'main': m_Str='int '
000040. HandleFunction() : name='main', args='()', peek='{'
000041. HandleFunction() : !(Ctor/Dtor) 'main', m_Str='int ', localParent='<none>'
000042. HandleFunction() : Adding function 'main', ': m_Str='int ', enc_ns='nil'.
000043. HandleFunction() : Add token name='main', args='()', return type='int '
000044. GetBaseArgs() : args='()'.
000045. GetBaseArgs() : baseArgs='()'.
000046. DoAddToken() : Created token='main', file_idx=1, line=5, ticket=
000047. GetActualTokenType() : Searching within m_Str='int'
000048. GetActualTokenType() : Compensated m_Str='int'
000049. GetActualTokenType() : Found 'int'
000050. DoAddToken() : Prepending ''
000051. DoAddToken() : Added/updated token 'main' (3), kind 'function', type 'int', actual 'int'. Parent is  (-1)
000052. ParserDummy() : ~ParserBase() : Destruction of Parser object.


--------------T-r-e-e--L-o-g--------------

000053. float* pFloat   [1,0]
000054. float** pFloat2   [2,0]
000055. float*  pFloatPar   [3,0]
000056. main()   [5,5]


--------------L-i-s-t--L-o-g--------------

000057. variable float* pFloat   [1,0]
000058. variable float** pFloat2   [2,0]
000059. variable float*  pFloatPar   [3,0]
000060. function int main()   [5,5]

[MortenMacFly]

...I got another one (a vector) not working properly for me:

Code Select Expand


#include <vector>

class AClass
{
public:
   AClass() {;}
  ~AClass() {;}

  void MyMethod() {;}
};

class AnotherClass
{
public:
   AnotherClass() {;}
  ~AnotherClass() {;}

  inline const std::vector<AClass*> GetAClassVec()
  {
    return mvpAClass;
  }

private:
   std::vector<AClass*> mvpAClass;
};

int main()
{
  AnotherClass a;
  a. // does not show GetAClassVec
}


[ollydbg]

Look, it seems the function does not parse correctly. See the screen shot below

[ollydbg]

OK, we arrived in this if clause.

Code Select Expand

                else if (peek==ParserConsts::dcolon)
                {
                    if (m_Str.IsEmpty())
                        m_EncounteredTypeNamespaces.push(token); // it's a type's namespace
                    else
                        m_EncounteredNamespaces.push(token);
                    m_Tokenizer.GetToken(); // eat ::
                }
Here, I just parse the third line:

Code Select Expand


class AnotherClass
{
  inline const std::vector<AClass*> GetAClassVec() {;}
};


Now:
token = "std"
peek = "::"
m_Str = "const "

The logic is:
We put the "std" as the m_EncounteredNamespaces,
but the correct way is: "std" should be put in m_EncounteredTypeNamespaces.

One possible way is: we should skip the "const" (do not put "const" into m_Str, as m_Str is like a returned type stack), so that when we peek a "::", we see the m_Str is empty, and "std" will put into m_EncounteredTypeNamespaces.

OK, can you understand my explanation???

[MortenMacFly]

We put the "std" as the m_EncounteredNamespaces,
but the correct way is: "std" should be put in m_EncounteredTypeNamespaces.

OK, can you understand my explanation???

Yes, but I think removing const wouldn't be wise because you wanna know, don't you.

What about doing something like (untested, blind meta-code):

Code Select Expand

else if (peek==ParserConsts::dcolon)
{
   if (   m_Str.IsEmpty()
       || m_Str.Trim(true).Trim(false).IsSameAs(ParserConsts::const)
       || m_Str.Trim(true).Trim(false).IsSameAs(ParserConsts::volatile) ) // what else?!
       m_EncounteredTypeNamespaces.push(token); // it's a type's namespace
   else
       m_EncounteredNamespaces.push(token);
   m_Tokenizer.GetToken(); // eat ::
}

[MortenMacFly]

What about doing something like (untested, blind meta-code):

(...answering myself...
Raises immediately the question, what about such things then:

Code Select Expand


inline const mynamespace::anothernamespace::std::vector<AClass*> GetAClassVec() {;}

???

[ollydbg]

What about doing something like (untested, blind meta-code):

(...answering myself...
Raises immediately the question, what about such things then:

Code Select Expand


inline const mynamespace::anothernamespace::std::vector<AClass*> GetAClassVec() {;}


It works OK, because the next time we enter the if clause, the have the same m_Str == const.

Code Select Expand

                else if (peek==ParserConsts::dcolon)
                {
                    if (   m_Str.IsEmpty()
                        || m_Str.Trim(true).Trim(false).IsSameAs(ParserConsts::kw_const)
                        || m_Str.Trim(true).Trim(false).IsSameAs(ParserConsts::kw_volatile) ) // what else?!
                        m_EncounteredTypeNamespaces.push(token); // it's a type's namespace
                    else
                        m_EncounteredNamespaces.push(token);
                    m_Tokenizer.GetToken(); // eat ::
                }