Problem: Due to the rise of cloud computing costs, many companies are switching to cheaper options: more efficient ML or cheaper edge devices.
Subscribe here to get access to the first 500 ideas from our blog. For just one coffee a month, you can have access to more than $500 billion dollars of ideas. What's not to love?
Solution: I recently came across this great article which attempted to broaden the ML market and cloud computing market then proposed solutions for solving some of its open problems.
As Chip Huyen described at the end of his article,
It’s helpful to think of how your models run on different hardware backends so that you can improve their performance. Austin Huang posted on our MLOps Discord that he often sees 2x speedup by just using simple off-the-shelf tools (quantization tools, Torchscript, ONNX, TVM) without much effort.
Here’s a great list of tips that can help you speed up PyTorch models on GPUs without even using compilers.
When your model is ready for deployment, it makes sense to try out different compilers to see which one gives you the best performance boost. You can run the experiments in parallel. A small boost for one inference request can accumulate into big returns over millions or billions of inference requests.
Even though there has been huge progress in compilers for machine learning, there’s still a lot of work to be done before we can abstract compilers completely from general ML practitioners. Think about traditional compilers like GCC. You write your code in C or C++, and GCC automatically lowers your code into machine code. Most C programmers don’t even care what the intermediate representations GCC generates.
In the future, ML compilers can be the same way. You use a framework to create an ML model in the form of a computation graph, and your ML compiler can generate machine-native code for whatever hardware you run on. You won’t even need to worry about intermediate representations.
Tools like TVM are steps towards making that future possible.
Could a billion-dollar business be created with the whole purpose of implementing these (and other) techniques to lower the compute costs of major businesses?
Monetization: Sales of this ML service.
Contributed by: Michael Bervell (Billion Dollar Startup Ideas)