Cybernetic Vermin posted:

google along with a lot of other actors have realized that a lot of successful ml simply involves ridiculous amounts of semi-skilled man-hours tweaking parameters and topologies randomly until the thing seems to do the thing, so it is worth it to try to capture more hands early on.

there's some glimmers of theoretical work happening on what works and what doesn't, notably lin-tegmark(-rolnick) https://arxiv.org/abs/1608.08225v4 , which does the very basic needful demonstrating that e.g. encoding multiplication requires exponentially wide hidden layers, which should reasonably be interpreted as "cannot be learned with current methods".

i never find the time, but i am personally quite sure that it is a pretty doable task to draw from minsky's perceptrons book (which famously killed off neural networks last time they were popular) the needed machinery, combined with reasonable assumptions on what is the limits of learning (e.g. finite precision) to demonstrate that some practical-sounding problems cannot be taught to deep neural networks without encoding as much information as a full table realization of the function requires. which should again be interpreted as "cannot be learned with current methods". i don't think such a paper would change the enthusiasm that much, but it is one of those directions that ought to be investigated sooner or later.