"Mom, I won't go back this year. If everything goes well, I will go back to visit you early next year."

"I'm not playing around outside. I'm not dropping out of school. Don't be nervous. I know what I'm doing."

"There is no need to send me any money. I am not starting a business. I just need to concentrate on solving some problems during this period. It will be the end of this year. Don't worry, it will be the end of this year. If it doesn't succeed, I will listen to you whatever you say."

"Okay, okay, remember to ask dad to go out and exercise more. Take care of your health!"

"Go back, go back. I must go back during the Chinese New Year."

After hanging up the phone, Meng Fanqi leaned against the wall and let out a long sigh.

It was really hard to explain these things to his parents. If it weren't for the fear that Dean Fu would help, he would not have planned to suspend his studies.

Now that my parents have learned about the current situation, they must be very worried. No matter what I say during the last few phone calls, my parents can't let go.

Meng Fanqi could only rely on the deadline at the end of the year to appease them, promising that as long as he could not achieve outstanding results this year, he would listen to them in everything.

Looking at Don Juan in the room who was gloating about his misfortune and adjusting the ginseng, Meng Fanqi said angrily, "Why are you laughing?"

Don Juan instantly made a serious expression, but no matter how he looked at it, he still had a sense of yin and yang, which made people very unhappy.

--------------------------

Later that day, Meng Fanqi received Alex's reply. Looking at the time, Alex should have replied to the email as soon as he got up, which can be said to be quite diligent.

This time point is close to the beginning of this year's competition, but Alex himself has no plans to continue to form a team to participate in this competition.

It's just that although I'm not in the Jianghu, the Jianghu is full of legends about me.

The 2013 Microsoft Research team, the National University of Singapore team, and the Decaf, UC Berkeley and other teams led by Jia Yangqing were all deeply influenced by Alex’s thoughts last year.

According to the method introduced by Alex, Meng Fanqi quickly completed the debugging and successfully migrated batch normalization, Adam optimizer and other operations from his own experimental environment to the cuda-conv framework, and made some adjustments and adaptations.

Not long after, it was already running on CIFAR-10.

Next is the core competitiveness of this competition, the implementation of the residual network on this framework.

"This year's IMAGE challenge actually doesn't have many new ideas. Everyone basically focuses on replicating Alex's algorithm."

Of course, Meng Fanqi is not prepared to re-implement Alex's competition algorithm last year based on Alex's framework like other teams. Many early algorithms have a lot of redundancy in the amount of parameters and calculations.

"Google's Google and Oxford's VGG were originally expected to be developed next year, but after I participate in the competition this year, these two important networks in the history of AI will probably be stillborn, right?"

Not to mention now, as of the end of 2014, redundant designs still exist in the algorithms of Google and Oxford University.

It wasn’t until 2015 that that man won Image-2015 with Residual Network Res, creating the godhead of AI models in the deep learning era.

Residual thinking is Meng Fanqi's best choice at this moment. On the one hand, it will become a milestone in the era of deep learning, because its core idea is how to make deep networks 'deeper'.

On the other hand, its implementation and structure are simple, abandoning a large number of designs that seem reasonable and valuable to humans, but are actually of little use.

Repeating the simple and easy-to-use structure repeatedly also greatly reduced the amount of development engineering required by Meng Fanqi.

Alex has an eight-layer structure, each with its own independent design. At this time, the eight layers are already a revolutionary deep network.

Deeper networks are particularly difficult to train, so it was not until 2014 that Google and Oxford advanced the depth to 22 and 19 layers respectively.

The idea of ​​​​Res completely and fundamentally solves the problem that the network cannot be trained smoothly when the network becomes deeper. It makes the training of 50, 100, 150, and even 1000-layer networks possible.

"From 8 last year, to 100 this year, and even 1,000 floors can be trained. From the perspective of 2013, this dose of strong medicine should be enough."

Not only is it a groundbreaking breakthrough in depth, but the performance of Res is also amazing. It is the first structure with a Top-5 error rate lower than human capabilities, with a single model reaching about 4.6%.

If we combine Res trained in several different ways and average their outputs to predict, the TOP-5 error rate is even as low as 3.7%.

"Actually, it's a weird thing to be too accurate on IMAGE." Considering later research, a large proportion of the labels in this data set are actually wrong, and the closer the accuracy is to 100%, the more absurd it is.

feeling.

Res in Res refers to residual, which is more seriously called residual. To put it more clearly, it is a kind of short circuit or jump link.

To put it more simply, assuming that the original operation is f(x), Res changes the calculation process from f(x) to f(x) x.

This was also one of the first discussions Meng Fanqi had with Dean Fu.

This kind of branch link without any parameters directly adds the content before the operation to the content after the operation. This is the secret why the deep network can suddenly converge at a depth of thousands of layers.

Almost all AI articles after this are difficult to avoid this idea. Kaiming himself received 150,000 to 60,000 article citations in six or seven years with this article alone.

It has to be said that this is an astronomical number. You must know that even if one of the three world-class journals of Nature Cell Science is a frequent visitor and a famous Chinese and American double scientific academician, the total number of citations of all articles so far adds up to more than 50,000.

That’s a lot.

Of course, it is biased to evaluate the quality of an article and the author solely based on the number of times a paper has been cited.

Many basic disciplines have high standards and few people read them, and few people write about them. Naturally, they cannot be cited.

But for AI, a subject with strong practicality and experimental nature, if you compare it internally, the gold content of the quotes is still quite high.

Everyone will only vote with code, and valuable things will continue to be iterated. Technologies that are conducive to improving the performance of their own models on tasks will continue to be cited.

There has always been a saying in the academic world that those who engage in theory and those who engage in experimentation look down upon each other. This is not unfounded.

After Kaiming established his historical position with several masterpieces, there are always people who like to say that "Res's ideas have been mentioned long ago", "Things written decades ago are nothing new", "This article is also

It’s just that it has a lot of traffic, it’s the same as a traffic star.

The sour smell almost overflows the screen.

It’s not that Meng Fanqi hasn’t read some theoretical articles, but he personally feels that AI in the new era, talk is cheap, show me the code. You said something that existed decades ago, why not write a Res to break through the human level?

Is it difficult to write the core code of Res? It’s not difficult.

Based on Alex's framework, he can basically reproduce all the structures in the Res paper in just a few hours.
Chapter completed!