In our last article, we showed you how from increasing the resource yield to strengthening your value chain, the diligent execution of optimization within the design process can mean the difference between thriving and declining. The next logical question is, how do you pick the right optimization tool?
There are numerous open-source optimization algorithms and many choices of commercial tools that range from free to being very expensive. How you pick the right optimization tool is important and can be complicated. In this article, we offer a step-by-step checklist in the hopes of making the process less complex than it seems.
For the purpose of this article, we are assuming that you are aware of the three essential elements of an optimization problem: objective(s), constraint(s), and variables. If not, please refer to our article “A Brief History of Optimization.” We are also assuming that you are picking a tool for a wide range of design problems.
Figure 1 shows the steps and procedure of selecting the right design optimization tools for you.
Figure 1: six steps to evaluating an optimizer
Step 1: Check if you have equations for all functions
Will you always have equations for your objectives and constraints? If yes, then you may be satisfied with most of the open-source software that are downloadable from the internet. If your problem is linear, meaning all your equations are linear functions, you are probably the luckiest engineer as you can solve these problems with mathematical certainty. But because so many engineering problems are nonlinear and complex you might sometimes have to resort to commercial software tools even for problems where you have a set of equations.
Step 2: Determine whether your simulation is quick to run
As for most engineers, you don’t always have equations for all the functions. You use finite element analysis (FEA), computational fluid dynamics (CFD), or other computationally intensive models. One exception is that sometimes your simulation or analysis is very fast to run, e.g., you can run 1000 simulations in seconds. In this special case, you can use open-source and/or free software for metaheuristic algorithms such as genetic algorithms (GA), particle swarm optimization (PSO) etc. MATLAB has a toolbox for this purpose too. Our focus is then optimization based on relatively expensive simulations.
Step 3: Compare algorithms for simulation-based optimization algorithms
There are many design optimization tools in the market, but not every tool is created equal. Each tool comes with its own set of algorithms. Different algorithms for different optimization problems, including global optimization, multi-objective optimization, optimization with expensive constraints, and probabilistic optimization. Most people are concerned with global optimization, which is because other types of optimization algorithms are constructed on the basis of global optimization algorithms. If you have to use one specific type of optimization algorithm, such as optimization with expensive constraints, then this could be your criterion to filter out tools that do not have this type of algorithms. But most often, you end up comparing algorithms in tools.
Let’s take the global optimization algorithms as an example. There are two ways to compare an algorithm. To prepare for the comparison, decide on a standard testbed first. The research community has developed a basket of benchmark mathematical functions that can be used to compare algorithms. The PDOL-SFU testbed is one such resource, and another resource you can use offers plots of these functions and can be found on Dr Surjanovic’s webpage.
· First way of comparing algorithms: Efficiency and accuracy are two criteria for algorithm comparison. You can fix the number of simulations, which represents the optimization cost, and see which algorithm reaches a better optimum with the same number of simulations. Note that neither CPU time nor wall time is a good indicator for cost as it is influenced by computing power, the number of other processes running at the same time, and whether parallel computing is used.
· Second way of comparing algorithms: The other approach for comparison is to compare the number of simulations when reaching a pre-set optimum value. The first approach measures the effectiveness of the algorithm and the latter measures the efficiency of the algorithm. A complementary comparison is to plot the convergence curve, i.e., objective value vs the number of simulations. This plot will show how fast an algorithm approaches to the converged optimum and if the algorithm gets stuck in some local optimum.
Please see Figure 2 as an example of a convergence plot (insert a convergence plot).
Figure 2 An example of convergence plot for minimizing a hypothetical function f1.
If you don’t have time to compare the algorithm, you can always ask your vendor for an algorithm test report. One thing to watch for is what benchmark problems the vendor picked, if these benchmark problems represent an unbiased selection of problems, and if these chosen problems are representative of your design challenges.
Step 4: Check software usability
While algorithms are the core of an optimization tool, the usability of these tools is also important as it affects the productivity of your team. Different users may have different preferences on usability. There are a few important ones listed below:
Algorithm picking and parameter tuning: Most software tools ask users to pick a “right” algorithm from dozens of algorithms. After an algorithm is chosen, the user then needs to specify or tune a couple of algorithm parameters. For simulation-based optimization, this is unreasonable because if the user does not know the properties of the simulation function, it is impossible for the user, even an optimization expert, to pick the “right” algorithm, let alone the “right” parameters. If a tool does not require a user to pick an algorithm, or tuning the parameters, this would be a great usability feature.
Learning curve: A good tool should present you with easily understood information about your design, not an arduous report that requires becoming an optimization expert to understand. A tool with a deep learning curve means more time and cost will be repetitively spent on learning the tool and refreshing the learning — using such a tool is wasting time. An easy-to-learn tool usually has simple and clearly laid out user interface that gives simple and clear output, making it easy to set up and run an optimization problem. A rough guideline is that an easy-to-learn tool should not take more than half a day for an inexperienced user to learn to use the tool to solve optimization problems.
Insight: One other criterion is, does the tool offer the user actionable insight to the design challenges? Such insight may include the following:
· Which variable has the biggest impact on my objective?
· In which interval does a variable always reach good solutions?
· If the optimal is not usable, where can you find the next best solution?
· Which variables are correlated?
· Which constraints are not necessary?
A tool may not be able give you all these insights, but you can compare what could be provided by a tool if such insight is useful to you.
Outputs: A good optimization tool should give you a succinct report of the optimization results and a detailed list of all the trial points the algorithm has searched. It should also support different design scenarios and design changes as well as keep a record of all these results for design studies. Note that you are looking for an optimization tool to support engineering design, not just a math tool.
Step 5: Ensure compatibility with your CAD/CAE tools
While almost all of the tools can integrate with some simulation tools, you need to examine the integration of the optimizer with your specific CAD/CAE tools. CAD tools are also included in this discussion as nowadays CAD tools often come with a simulation module. Some optimization tools offer direct plugin for popular CAD tools, through which their CAE modules are called for design evaluation. The vendor may not have a dedicated plug-in with your CAE tools. In this case, you can ask the vendor to provide a template or an example of how their tool is integrated with your CAE tool. You’d make sure the integration is easy and repeatable.
Step 6: Evaluate the support provided
Lots of people tend to ignore this last step as they become overwhelmed with the previous steps. The key is not to chase big names but to evaluate the people who actually support you, which is almost irrelevant to the size of the company. Although big names may give you more confidence, they may have lots of customers and you are just one of many. On the other hand, small companies can be more agile, attentive, and responsive to your problem. The good news in the area of design optimization is that there is no dominant vendor yet as this is still an emerging field.
If you have questions or need more information, please feel free to contact me.