本文作者為香港浸會大學 賀鑫。
英文標題 | AutoML：A survey of State-of-the-art
作 者 | Xin He, Kaiyong Zhao, Xiaowen Chu
單 位 | Hong Kong Baptist University(香港浸會大學)
論文鏈接 | https://arxiv.org/abs/1908.00709
深度學習已經運用到多個領域，為人們生活帶來極大便利。然而，為特定任務構造一個高質量的深度學習系統不僅需要耗費大量時間和資源，而且很大程度上需要專業的領域知識。
因此，為了讓深度學習技術以更加簡單的方式應用到更多的領域，自動機器學習(AutoML)逐漸成為人們關注的重點。
本文首先從端到端系統的角度總結了自動機器學習在各個流程中的研究成果(如下圖)，然后著重對最近廣泛研究的神經結構搜索(Neural Architecture Search, NAS)進行了總結，最后討論了一些未來的研究方向。
?
?
一、數據準備
眾所周知，數據對于深度學習任務而言至關重要，因此一個好的AutoML系統應該能夠自動提高數據質量和數量，我們將數據準備劃分成兩個部分：數據收集和數據清洗。
?
?
1、數據收集
現如今不斷有公開數據集涌現出來，例如MNIST，CIFAR10，ImageNet等等。我們也可以通過一些公開的網站獲取各種數據集，例如Kaggle， Google Dataset Search以及Elsevier Data Search等等。但是對于一些特殊的任務，尤其是醫療或者涉及到個人隱私的任務，由于數據很難獲取，所以通常很難找到一個合適的數據集或者數據集很小。解決這一問題主要有兩種思路：數據生成和數據搜索。
1)數據生成
圖像：
Cubuk, EkinD., et al. "Autoaugment: Learning augmentation policies fromdata." arXiv preprint arXiv:1805.09501 (2018).
語音：
Park, DanielS., et al. "Specaugment: A simple data augmentation method for automaticspeech recognition." arXiv preprint arXiv:1904.08779(2019).
文本：
Xie, Ziang,et al. "Data noising as smoothing in neural network languagemodels." arXiv preprint arXiv:1703.02573 (2017).
Yu, Adams Wei,et al. "Qanet: Combining local convolution with global self-attention forreading comprehension." arXiv preprint arXiv:1804.09541 (2018).
GAN：
Karras, Tero, Samuli Laine, and Timo Aila. "A style-basedgenerator architecture for generative adversarial networks." Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
模擬器：
Brockman, Greg, et al. "Openai gym." arXiv preprintarXiv:1606.01540 (2016).
2)數據搜索
Roh, Yuji, Geon Heo, and Steven Euijong Whang. "A survey ondata collection for machine learning: a big data-ai integrationperspective." arXiv preprint arXiv:1811.03402(2018).
Yarowsky, David. "Unsupervised word sense disambiguationrivaling supervised methods." 33rd annual meeting of the associationfor computational linguistics. 1995.
Zhou, Yan, and Sally Goldman. "Democraticco-learning." 16th IEEE International Conference on Tools withArtificial Intelligence. IEEE, 2004.
2、數據清洗
Krishnan,Sanjay, and Eugene Wu. "Alphaclean: Automatic generation of data cleaningpipelines." arXiv preprint arXiv:1904.11827 (2019).
Chu, Xu, etal. "Katara: A data cleaning system powered by knowledge bases andcrowdsourcing." Proceedings of the 2015 ACM SIGMOD InternationalConference on Management of Data. ACM, 2015.
Krishnan,Sanjay, et al. "Activeclean: An interactive data cleaning framework formodern machine learning." Proceedings of the 2016 InternationalConference on Management of Data. ACM, 2016.
Krishnan,Sanjay, et al. "SampleClean: Fast and Reliable Analytics on DirtyData." IEEE Data Eng. Bull. 38.3 (2015): 59-75.
二、特征工程
特征工程可分為三個部分：
1、特征選擇
2、特征構造
H. Vafaie and K. De Jong, “Evolutionary feature spacetransformation,” in Feature Extraction, Construction and Selection. Springer,1998, pp. 307–323
J. Gama, “Functional trees,” Machine Learning, vol. 55, no. 3, pp.219–250, 2004.
D. Roth and K. Small, “Interactive feature space construction usingsemantic information,” in Proceedings of the Thirteenth Conference onComputational Natural Language Learning. Association for Computational Linguistics,2009, pp. 66–74.
3、特征提取
Q. Meng, D. Catchpoole, D. Skillicom, and P. J. Kennedy, “Relationalautoencoder for feature extraction,” in 2017 International Joint Conference onNeural Networks (IJCNN). IEEE, 2017, pp. 364–371.
O. Irsoy and E. Alpayd?n, “Unsupervised feature extraction withautoencoder trees,” Neurocomputing, vol. 258, pp. 63–73, 2017.
?
?
三、模型生成
模型生成的方式主要有兩種：一是基于傳統的機器學習方法生成模型，例如SVM，decision tree等，已經開源的庫有Auto-sklearn和TPOT等。另一種是是神經網絡結構搜索(NAS)。我們會從兩個方面對NAS進行總結，一是NAS的網絡結構，二是搜索策略。
?
?
1、網絡結構
1)整體結構(entire structure)：
該類方法是生成一個完整的網絡結構。其存在明顯的缺點，如網絡結構搜索空間過大，生成的網絡結構缺乏可遷移性和靈活性。
B. Zoph and Q. V. Le, “Neural architecture search with reinforcementlearning.” [Online]. Available:http://arxiv.org/abs/1611.01578
H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficientneural architecture search via parameter sharing,” vol. ICML. [Online].Available: http://arxiv.org/abs/1802.03268
2)基于單元結構(cell-based structure)：
為解決整體結構網絡搜索存在的問題提出了基于單元結構設計的方法。如下圖所示，搜索到單元結構后需要疊加若干個單元結構便可得到最終的網絡結構。不難發現，搜索空間從整個網絡縮減到了更小的單元結構，而且我們可以通過增減單元結構的數量來改變網絡結構。但是這種方法同樣存在一個很明顯的問題，即單元結構的數量和連接方式不確定，現如今的方法大都是依靠人類經驗設定。
?
?
H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficientneural architecture search via parameter sharing,” vol. ICML. [Online].Available: http://arxiv.org/abs/1802.03268
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learningtransferable architectures for scalable image recognition.” [Online].Available: http://arxiv.org/abs/1707.07012
Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu, “Practicalblock-wise neural network architecture generation.” [Online]. Available:http://arxiv.org/abs/1708.05552
B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neuralnetwork architectures using reinforcement learning,” vol. ICLR. [Online].Available: http://arxiv.org/abs/1611.02167
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q.Le, and A. Kurakin, “Large-scale evolution of image classifiers.” [Online].Available: http://arxiv.org/abs/1703.01041
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolutionfor image classifier architecture search.” [Online]. Available:http://arxiv.org/abs/1802.01548
3)層次結構(hierarchical structure)
不同于上面將單元結構按照鏈式的方法進行連接，層次結構是將前一步驟生成的單元結構作為下一步單元結構的基本組成部件，通過迭代的思想得到最終的網絡結構。如下圖所示，(a)中左邊3個是最基本的操作，右邊是基于這些基本操作生成的某一個單元結構;(b)中左邊展示了上一步驟中生成的若干個單元結構，通過按照某種策略將這些單元結構進行組合得到了更高階的單元結構。
H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu,“Hierarchical representations for efficient architecture search,” in ICLR, p.13
?
?
4)基于網絡態射結構(network morphism-based structure)：
一般的網絡設計方法是首先設計出一個網絡結構，然后訓練它并在驗證集上查看它的性能表現，如果表現較差，則重新設計一個網絡。可以很明顯地發現這種設計方法會做很多無用功，因此耗費大量時間。而基于網絡態射結構方法能夠在原有的網絡結構基礎上做修改，所以其在很大程度上能保留原網絡的優點，而且其特殊的變換方式能夠保證新的網絡結構還原成原網絡，也就是說它的表現至少不會差于原網絡。
?
?
T. Chen, I. Goodfellow, and J. Shlens, “Net2net: Acceleratinglearning via knowledge transfer,” arXiv preprint arXiv:1511.05641, 2015.
T. Elsken, J. H. Metzen, and F. Hutter, “Efficient multi-objectiveneural architecture search via lamarckian evolution.” [Online]. Available:http://arxiv.org/abs/1804.09081
H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang, “Efficientarchitecture search by network transformation,” in Thirty-Second AAAIConference on Artificial Intelligence, 2018
2、搜索策略
1)網格搜索
H. H. Hoos, Automated Algorithm Configuration and Parameter Tuning,2011
I. Czogiel, K. Luebke, and C. Weihs, Response surface methodologyfor optimizing hyper parameters. Universitatsbibliothek Dortmund, 2006.
C.-W. Hsu, C.-C. Chang, C.-J. Lin et al., “A practical guide tosupport vector classification,” 2003.
J. Y. Hesterman, L. Caucci, M. A. Kupinski, H. H. Barrett, and L. R.Furenlid, “Maximum-likelihood estimation with a contracting-grid searchalgorithm,” IEEE transactions on nuclear science, vol. 57, no. 3, pp.1077–1084, 2010.
2)隨機搜索
J. Bergstra and Y. Bengio, “Random search for hyper-parameteroptimization,” p. 25.
H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio,“An empirical evaluation of deep architectures on problems with many factors ofvariation,” in Proceedings of the 24th international conference on Machinelearning. ACM, 2007, pp. 473–480.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar,“Hyperband: A novel bandit-based approach to hyperparameter optimization.”[Online]. Available: http://arxiv.org/abs/1603.06560
3)強化學習
B. Zoph and Q. V. Le, “Neural architecture search with reinforcementlearning.” [Online]. Available: http://arxiv.org/abs/1611.01578
B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neuralnetwork architectures using reinforcement learning,” vol. ICLR. [Online].Available: http://arxiv.org/abs/1611.02167
H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficientneural architecture search via parameter sharing,” vol. ICML. [Online].Available: http://arxiv.org/abs/1802.03268
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learningtransferable architectures for scalable image recognition.” [Online].Available: http://arxiv.org/abs/1707.07012
Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu, “Practical block-wiseneural network architecture generation.” [Online]. Available:http://arxiv.org/abs/1708.05552
4)進化算法
L. Xie and A. Yuille, “Genetic CNN,” vol. ICCV. [Online]. Available:http://arxiv.org/abs/1703.01513
M. Suganuma, S. Shirakawa, and T. Nagao, “A genetic programmingapproach to designing convolutional neural network architectures.” [Online].Available: http://arxiv.org/abs/1704.00764
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q.Le, and A. Kurakin, “Large-scale evolution of image classifiers.” [Online].Available: http://arxiv.org/abs/1703.01041
K. O. Stanley and R. Miikkulainen, “Evolving neural networks throughaugmenting topologies,” vol. 10, no. 2, pp. 99–127. [Online]. Available:http://www.mitpressjournals.org/doi/10.1162/106365602320169811
T. Elsken, J. H. Metzen, and F. Hutter, “Efficient multi-objectiveneural architecture search via lamarckian evolution.” [Online]. Available:http://arxiv.org/abs/1804.09081
5)貝葉斯算法
J. Gonzalez, “Gpyopt: A bayesian optimization framework in python,”http://github.com/SheffieldML/GPyOpt, 2016
J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesianoptimization of machine learning algorithms,” in Advances in neural informationprocessing systems, 2012, pp. 2951–2959.
S. Falkner, A. Klein, and F. Hutter, “BOHB: Robust and efficienthyperparameter optimization at scale,” p. 10.
F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-basedoptimization for general algorithm configuration,” in Learning and IntelligentOptimization, C. A. C. Coello, Ed. Springer Berlin Heidelberg, vol. 6683, pp.507–523. [Online]. Available:http://link.springer.com/10.1007/978-3-642-25566-3 40
J. Bergstra, D. Yamins, and D. D. Cox, “Making a science of modelsearch: Hyperparameter optimization in hundreds of dimensions for visionarchitectures,” p. 9.
A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter, “Fastbayesian optimization of machine learning hyperparameters on large datasets.”[Online]. Available: http://arxiv.org/abs/1605.07079
6)梯度下降算法
H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiablearchitecture search.” [Online]. Available: http://arxiv.org/abs/1806.09055
S. Saxena and J. Verbeek, “Convolutional neural fabrics,” inAdvances in Neural Information Processing Systems, 2016, pp. 4053–4061.
K. Ahmed and L. Torresani, “Connectivity learning in multi-branchnetworks,” arXiv preprint arXiv:1709.09582, 2017.
R. Shin, C. Packer, and D. Song, “Differentiable neural networkarchitecture search,” 2018.
D. Maclaurin, D. Duvenaud, and R. Adams, “Gradient-basedhyperparameter optimization through reversible learning,” in InternationalConference on Machine Learning, 2015, pp. 2113–2122.
F. Pedregosa, “Hyperparameter optimization with approximategradient,” arXiv preprint arXiv:1602.02355, 2016.
S. H. Han Cai, Ligeng Zhu, “PROXYLESSNAS: DIRECT NEURAL ARCHITECTURESEARCH ON TARGET TASK AND HARDWARE,” 2019
G. D. H. Andrew Hundt, Varun Jain, “sharpDARTS: Faster and MoreAccurate Differentiable Architecture Search,” Tech. Rep. [Online]. Available:https://arxiv.org/pdf/1903.09900.pdf
四、模型評估
模型結構設計好后我們需要對模型進行評估，最簡單的方法是將模型訓練至收斂，然后根據其在驗證集上的結果判斷其好壞。但是這種方法需要大量時間和計算資源。因此有不少加速模型評估過程的算法被提出，總結如下：
?
?
1、低保真度評估
Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter, “Fastbayesian optimization of machine learning hyperparameters on large datasets.”[Online]. Available: http://arxiv.org/abs/1605.07079
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learningtransferable architectures for scalable image recognition.” [Online]. Available:http://arxiv.org/abs/1707.07012
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolutionfor image classi?er architecture search.” [Online]. Available:http://arxiv.org/abs/1802.01548
A. Zela, A. Klein, S. Falkner, and F. Hutter, “Towards automateddeep learning: Ef?cient joint neural architecture and hyperparameter search.” [Online].Available: http://arxiv.org/abs/1807.06906
Y.-q. Hu, Y. Yu, W.-w. Tu, Q. Yang, Y. Chen, and W. Dai,“Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion,” p. 8, 2019.
2、遷移學習
C. Wong, N. Houlsby, Y. Lu, and A. Gesmundo, “Transfer learning withneural automl,” in Advances in Neural Information Processing Systems, 2018, pp.8356–8365.
T. Wei, C. Wang, Y. Rui, and C. W. Chen, “Network morphism,” inInternational Conference on Machine Learning, 2016, pp. 564–572.
T. Chen, I. Goodfellow, and J. Shlens, “Net2net: Acceleratinglearning via knowledge transfer,” arXiv preprint arXiv:1511.05641, 2015.
3、基于代理(surrogate-based)
K. Eggensperger, F. Hutter, H. H. Hoos, and K. Leyton-Brown,“Surrogate benchmarks for hyperparameter optimization.” in MetaSel@ ECAI, 2014,pp. 24–31.
C. Wang, Q. Duan, W. Gong, A. Ye, Z. Di, and C. Miao, “An evaluationof adaptive surrogate modeling based optimization with two benchmark problems,”Environmental Modelling & Software, vol. 60, pp. 167–179,2014.
K. Eggensperger, F. Hutter, H. Hoos, and K. Leyton-Brown, “Ef?cientbenchmarking of hyperparameter optimizers via surrogates,” in Twenty-Ninth AAAIConference on Arti?cial Intelligence, 2015.
K. K. Vu, C. D’Ambrosio, Y. Hamadi, and L. Liberti, “Surrogate-basedmethods for black-box optimization,” International Transactions in OperationalResearch, vol. 24, no. 3, pp. 393–424, 2017.
C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L.Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecturesearch.” [Online]. Available: http://arxiv.org/abs/1712.00559
4、早停(early-stopping)
A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter, “Learningcurve prediction with bayesian neural networks,” 2016.
B. Deng, J. Yan, and D. Lin, “Peephole: Predicting networkperformance before training,” arXiv preprint arXiv:1712.03351, 2017.
T. Domhan, J. T.Springenberg, and F. Hutter, “Speeding up automatic hyperparameter optimizationof deep neural networks by extrapolation of learning curves,” in Twenty-FourthInternational Joint Conference on Arti?cial Intelligence, 2015.
M. Mahsereci, L. Balles, C. Lassner, and P. Hennig, “Early stoppingwithout a validation set,” arXiv preprint arXiv:1703.09580, 2017.
五、NAS算法性能總結
?
?
下圖總結了不同NAS算法在CIFAR10上的搜索網絡所花費的時間以及準確率。可以看到相比于基于強化學習和進化算法的方法，基于梯度下降和隨機搜索的方法能夠使用更少的時間搜索得到表現優異的網絡模型。
?
?
六、總結
通過對AutoML最新研究進展的總結我們發現還有如下問題值得思考和解決：
1、完整pipeline系統
現如今有不少開源AutoML庫，如TPOT，Auto-sklearn都只是涉及整個pipeline的某一個或多個過程，但是還沒有真正實現整個過程全自動，因此如何將上述所有流程整合到一個系統內實現完全自動化是未來需要不斷研究的方向。
2、可解釋性
深度學習網絡的一個缺點便是它的可解釋性差，AutoML在搜索網絡過程中同樣存在這個問題。目前還缺乏一個嚴謹的科學證明來解釋
為什么某些操作表現更好，例如
就基于單元結構設計的網絡而言，很難解釋為什么通過疊加單元結構就能得到表現不錯的網絡結構。另外為何ENAS提出的權值共享能夠work同樣值得思考。
3、可復現性
大多數的AutoML研究工作都只是報告了其研究成果，很少會開源其完整代碼，有的只是提供了最終搜索得到的網絡結構而沒有提供搜索過程的代碼。另外較多論文提出的方法難以復現，一方面是因為他們在實際搜索過程中使用了很多技巧，而這些都沒有在論文中詳細描述，另一方面是網絡結構的搜索存在一定的概率性質。因此如何確保AutoML技術的可復現性也是未來的一個方向。
4、靈活的編碼方式
通過總結NAS方法我們可以發現，所有方法的搜索空間都是在人類經驗的基礎上設計的，所以最終得到的網絡結構始終無法跳出人類設計的框架。例如現如今的NAS無法憑空生成一種新的類似于卷積的基本操作，也無法生成像Transformer那樣復雜的網絡結構。因此如何定義一種泛化性更強，更靈活的網絡結構編碼方式也是未來一個值得研究的問題。
5、終身學習(lifelong learn)
大多數的AutoML都需要針對特定數據集和任務設計網絡結構，而對于新的數據則缺乏泛化性。而人類在學習了一部分貓狗的照片后，當出現未曾見過的貓狗依然能夠識別出來。因此一個健壯的AutoML系統應當能夠終身學習，即既能保持對舊數據的記憶能力，又能學習新的數據。
