CodeBuild Service role
Link
Posts
Run ML pipeline on AWS ECS
In our project we have a ML pipeline which is suppressed to be run just a few times a year. Our stakeholders requested pipeline to be not expensive and simple to execute., that's why we decided to utilise AWS ACS. In out pipeline with have only 2 steps. The first is data preprocessing, second is inference. Both are computational heavy and require GPUs. This was an obstacle for us, because we wanted to use ECS without managing underlying EC2 infrastructure. Fortunately for us, we had opportunity to ask for advice AWS Support. Guys were kind to explain us that ECS service has a feature called Capacity Providers. It is possible to specify EC2 as capacity provider and select instances with GPU to be provided on demand, like on the picture below we we selected EC2 Capacity Provider for our cluster. We can select desired AMI and instance type. It is quite important to specify desired capacity. As we do not have streaming tasks and our customer can wait until cluster is provi...