Research Computing and its customers embrace cloud
Researcher and IT interest in cloud technologies is growing significantly. Amid that interest, Research Computing now supports more than 30 projects across the three major cloud vendors: Amazon Web Services, Google Cloud Platform and Microsoft Azure.
For the most part, single researchers or labs are leading these projects, and they’re interested in harnessing the unmatched scale and scientific computational features that very likely can only be delivered in the cloud. In some cases, Research Computing is supporting research that might not have happened otherwise.
While supporting a growing stable of internal and researcher projects in the cloud, Research Computing has concentrated program development efforts in four areas: driving awareness through training, scaling its business processes, funding small researcher projects directly, and starting larger, proof-of-concept projects with internal, Research Computing staff members.
Events highlight cloud computing
Research Computing’s awareness program has been very popular. Among the training events that it has held for researchers in 2019, Research Computing and the Kenan Institute of Private Enterprise conducted a Machine Learning Day in April 2019. The event was open to the UNC-Chapel Hill campus community and held at the Kenan-Flagler Business School. Research Computing’s Sandeep Sarangi and Ashley Brown of the Kenan Institute of Private Enterprise organized the event and Microsoft Azure sponsored it.
Research Computing is planning similar events with Google Cloud Platform and Amazon Web Services. These events enable researchers to both experience new technologies and collaborate with peers who have similar interests. Like the Kenan Institute event, most sessions are vendor sponsored, leaving Research Computing with only the expense of providing lunch.
Researchers recognize cloud value
The scale and features of the cloud are certainly attractive to nearly all computational researchers. However, if Research Computing cannot reliably and transparently show researchers and the University administration what it is spending, most projects will never get off the ground — for good reason. In 2019, Research Computing’s Tim McGuire tested several software packages with an eye toward improving Research Computing’s business process and gaining a better understanding of the division’s bills in the big picture of economies of scale.
“We want to give our constituents — no matter how small — granular visibility into their cloud spending and automated alerting, when necessary,” said McGuire, Director, Campus Infrastructure Services.
Cloud credits benefit users
Competent billing and usage management are key factors in Research Computing’s ability to directly fund researchers and provide even more value to the community. In 2019, Research Computing was able to provide more than $50,000 to individual researchers and project teams in the form of cloud credits.
In one particularly successful startup project, Research Computing teamed up with Steven King, a professor of emerging technology at Carolina’s Hussman School of Journalism and Media, and the KFNext initiative at the Kenan-Flagler Business School. King’s team provided expertise and technical staff while Research Computing contributed an initial, renewable $5,000 Amazon Web Services credit to provide an easy machine learning platform for faculty to analyze admissions data.
Researcher funding is a great way to give individuals and labs a way to learn and utilize cloud technologies. Research Computing, though, also wanted to expand its own internal services to serve more researchers in better ways.
Cloud vendors sponsor projects
“So we began major projects sponsored by the cloud vendors themselves to deliver documented, repeatable reference environments in the cloud,” McGuire said.
In late summer 2019, Karen McCollough, Rob Zelt and McGuire completed the Secure Research Work space project, which can provide almost limitless capacity in a secure, private environment for teams to perform computational research on sensitive data in Microsoft Azure.
“We are now working on automating and operationalizing the platform which will enable us to quickly build these secure environments for any project that has a need,” Zelt said.
In another project, Research Computing’s Jeff Roach, Mike Waldron, Jenny Williams, Karen Milberger and McGuire began working with Google in summer 2019 to expand Research Computing’s Longleaf high-performance computing cluster.
The cluster has nearly 14,000 vCPU cores available for use by any UNC-Chapel Hill faculty member. The popular service sometimes has virtual queues to get in. Researcher jobs can sometimes wait days to run.
“The idea is that when queues are long or a researcher needs more computing than we have on hand, we create a relief valve that might move data and run those jobs in Google Cloud Platform (GCP),” said Liam Greenwood, Research Computing’s IT Manager for Advanced Computing Infrastructure. “In our initial testing, we submitted a large job to the campus Longleaf cluster. Then Longleaf was able to spin up over 16,000 deeply discounted vCPU cores in a GCP datacenter in South Carolina, effectively doubling our raw capacity in under 30 minutes.”
A couple hours later when the job completed, Longleaf turned off all the cloud computers, effectively stopping the meter from running and saving customers money.
“Both of these projects,” said John McGee, Interim Director, Research Computing, “represent a significant leap forward in our ability to provide value to the computational research and data science communities at UNC-Chapel Hill.”