A very important question. As it was already explained in this thread, this is a well-known and quite usual phenomenon (called childhood amnesia), caused by re-wiring of the brain at the critically important stage of development. The timing is slightly different in different people, and, I feel, in respect to different kinds of brain activity — for example, you cannot forget how to swim or ride a bicycle, if you learnt these skills at the age covered by amnesia. Also, it appears that children do not unlearn how to read or do arithmetic — but they can eventually forget how did they learn to read.
In general, I think non-one should worry about their childhood amnesia — these were natural changes in one’s brain, and they were to one’s benefit.
I collected hundreds of testimonies from people about their very first memories of learning mathematics — and discovered, that it seems that majority of people simply do not remember anything at all about their earliest encounters with school mathematics — including, it appears, many teachers of mathematics. Unfortunately, I had a day job to do and had no time to run a proper statistical analysis. But I think, this is something that should be taken into account in professional education of future school teachers of mathematics.