274 lines
7.9 KiB
HTML
274 lines
7.9 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en" dir="ltr">
|
|
|
|
<head>
|
|
<meta name="generator" content="Hugo 0.95.0" />
|
|
<meta charset="UTF-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<meta name="description" content="Now that I had successfully run my model without any runtime errors, the next step this week was finding some GPU compute so I can train my model on much more powerful hardware to accelerate the training.
|
|
My first idea was to use cloud computing. There are machine learning specific cloud technologies, but I didn’t want to use these as I didn’t want my code to be dependent on the specific ways cloud platforms want the code in.">
|
|
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Week 7" />
|
|
<meta property="og:description" content="Now that I had successfully run my model without any runtime errors, the next step this week was finding some GPU compute so I can train my model on much more powerful hardware to accelerate the training.
|
|
My first idea was to use cloud computing. There are machine learning specific cloud technologies, but I didn’t want to use these as I didn’t want my code to be dependent on the specific ways cloud platforms want the code in." />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://mmp.oscar.blue/posts/week-7/" /><meta property="article:section" content="posts" />
|
|
<meta property="article:published_time" content="2022-03-20T12:40:18+01:00" />
|
|
<meta property="article:modified_time" content="2022-03-20T12:40:18+01:00" />
|
|
|
|
<title>Week 7 | MMP | Oscar Pocock</title>
|
|
<link rel="manifest" href="/manifest.json">
|
|
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
|
<link rel="stylesheet" href="/book.min.68be0b7a9674f2a612ce0e9b2e9447ff4b7ac96546e06b642bfd3ded0ca490ef.css" integrity="sha256-aL4LepZ08qYSzg6bLpRH/0t6yWVG4GtkK/097QykkO8=">
|
|
<script defer src="/en.search.min.6bd18a0f048eff9ca273fd5ac40fe7159f145a30edf28a0847cab90265bd241a.js" integrity="sha256-a9GKDwSO/5yic/1axA/nFZ8UWjDt8ooIR8q5AmW9JBo="></script>
|
|
|
|
<script defer src="/sw.min.6f6f90fcb8eb1c49ec389838e6b801d0de19430b8e516902f8d75c3c8bd98739.js" integrity="sha256-b2+Q/LjrHEnsOJg45rgB0N4ZQwuOUWkC+NdcPIvZhzk="></script>
|
|
<!--
|
|
Made with Book Theme
|
|
https://github.com/alex-shpak/hugo-book
|
|
-->
|
|
|
|
|
|
</head>
|
|
|
|
<body dir="ltr">
|
|
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
|
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
|
<main class="container flex">
|
|
<aside class="book-menu">
|
|
<div class="book-menu-content">
|
|
|
|
<nav>
|
|
<h2 class="book-brand">
|
|
<a href="/"><span>MMP | Oscar Pocock</span>
|
|
</a>
|
|
</h2>
|
|
|
|
|
|
<div class="book-search">
|
|
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
|
<div class="book-search-spinner hidden"></div>
|
|
<ul id="book-search-results"></ul>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li class="book-section-flat" >
|
|
|
|
|
|
|
|
|
|
|
|
<a href="https://mmp.oscar.blue/docs/developer/" class="">Developer Documentation</a>
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
</ul>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li class="book-section-flat" >
|
|
|
|
|
|
|
|
|
|
|
|
<a href="https://mmp.oscar.blue/docs/user/" class="">User Documentation</a>
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
</ul>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
<li>
|
|
<a href="/posts/" >
|
|
Blog
|
|
</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="https://teaching.dcs.aber.ac.uk/mmp" target="_blank" rel="noopener">
|
|
Aberystwyth MMP Site
|
|
</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="https://git.oscar.blue" target="_blank" rel="noopener">
|
|
Source Code
|
|
</a>
|
|
</li>
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</nav>
|
|
|
|
|
|
|
|
|
|
<script>(function(){var e=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(){localStorage.setItem("menu.scrollTop",e.scrollTop)}),e.scrollTop=localStorage.getItem("menu.scrollTop")})()</script>
|
|
|
|
|
|
|
|
</div>
|
|
</aside>
|
|
|
|
<div class="book-page">
|
|
<header class="book-header">
|
|
|
|
<div class="flex align-center justify-between">
|
|
<label for="menu-control">
|
|
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
|
</label>
|
|
|
|
<strong>Week 7</strong>
|
|
|
|
<label for="toc-control">
|
|
|
|
<img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" />
|
|
|
|
</label>
|
|
</div>
|
|
|
|
|
|
|
|
<aside class="hidden clearfix">
|
|
|
|
|
|
<nav id="TableOfContents"></nav>
|
|
|
|
|
|
|
|
</aside>
|
|
|
|
|
|
</header>
|
|
|
|
|
|
|
|
<article class="markdown">
|
|
<h1>
|
|
<a href="/posts/week-7/">Week 7</a>
|
|
</h1>
|
|
|
|
<h5>March 20, 2022</h5>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p>Now that I had successfully run my model without any runtime errors, the next step this week was finding some GPU compute so I can train my model on much more powerful hardware to accelerate the training.</p>
|
|
<p>My first idea was to use cloud computing. There are machine learning specific cloud technologies, but I didn’t want to use these as I didn’t want my code to be dependent on the specific ways cloud platforms want the code in. Instead, I wanted to get a general VM with an attached GPU where I could run my workloads manually. I had already written docker images that contained all the depencies of my code that I could deploy to these VMs to ensure a reproducible and portable environment.</p>
|
|
<p>First place I looked was
|
|
<a href="https://www.linode.com/products/gpu/">Linode</a>. Although, after I contacted their support about it they said I needed at least $100 of transactions on my account in order to request access to their GPU instances. They also noted I could make a deposit of $100 to start using them straight away. I wasn’t sure if my model was going to use up $100 to train yet so I didn’t want to risk it.</p>
|
|
<p>I then looked to
|
|
<a href="https://azure.microsoft.com">Microsoft’s Azure</a>. I had used Azure during my industrial year and had previously passed the fundamentals and associate administrator exams for the platform so felt fairly confident in using it for my project. I ran into some issues I couldn’t quite explain at the start of the week. For some reason no services were available for me to use. I couldn’t use any VMs, create any networks or drives etc… Turns out I noticed that my UK account was defaulting to trying to create resources in the US which I didn’t have access to. So I had to manually set my location to the UK in order to create any resources.</p>
|
|
<p>While I was trying to work out the issue with Azure, I looked at using
|
|
<a href="https://cloud.google.com">GCP</a>. GCP automatically sets new accounts to have a quota of 0 for GPUs. Meaning you can’t attach one to any VM. You can increase the quota which requires getting in contact with customer support. Within 10 minutes I got a response and my quota was increases by 1. I wrote
|
|
<a href="https://www.terraform.io/">Terraform</a> IaC (Infrastructure as Code) to automatically spin up and spin down cloud resources quickly.</p>
|
|
<p>At this point I realised my code wasn’t very portable as it included hard-coded absolute paths among other things. I refactored a lot of my code in order to run it on any machine. I was also granted access to the university’s GPU compute servers which allowed me to train my model without paying for cloud fees. To make sure my refactoring worked and that the GPU was actually being utilised during training, I rain the training script on the uni’s GPU compute server for 20 epochs. It successfully finished in ~13 minutes.</p>
|
|
</article>
|
|
|
|
|
|
|
|
<footer class="book-footer">
|
|
|
|
<div class="flex flex-wrap justify-between">
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
</footer>
|
|
|
|
|
|
|
|
<div class="book-comments">
|
|
|
|
</div>
|
|
|
|
|
|
|
|
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
|
</div>
|
|
|
|
|
|
<aside class="book-toc">
|
|
<div class="book-toc-content">
|
|
|
|
|
|
<nav id="TableOfContents"></nav>
|
|
|
|
|
|
|
|
</div>
|
|
</aside>
|
|
|
|
</main>
|
|
|
|
|
|
</body>
|
|
|
|
</html>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|