AmazonEC2: Difference between revisions

From OpenSeesWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(21 intermediate revisions by the same user not shown)
Line 1: Line 1:
OpenSees can run Sequentially and in Parallel on the resources provided by Amazon EC2. Amazon EC2 is the largest and most widely used cloud service provider. Once you are set up with Amazon and have installed some software from MIT, the running of the applications is trivial. For example, 6 commands from the terminal and you could be running a parallel application on a remote Aamzon EC2 cluster of a few thousand nodes for a few dollars an hour. And it might not cost you anything, i.e. it could be FREE! as you get [http://aws.amazon.com/education/awseducate/ free hours] through the education program if you are a student or educator.
OpenSees can run '''Sequentially''' and in '''Parallel''' on the resources provided by [http://aws.amazon.com/ec2/ Amazon EC2]. '''Amazon EC2''' is the largest and most widely used cloud service provider. Once you are set up with Amazon and have installed some software from MIT, the running of the applications is trivial. For example, 7 commands from the terminal and you could be running a parallel application on a remote Amazon EC2 cluster of a few thousand nodes for a few dollars an hour. And it might not cost you anything, i.e. it could be '''FREE!''' Students and Educators can get [http://aws.amazon.com/education/awseducate/ free hours] through the education program and and for those in Industry get a limited amount of resources [http://aws.amazon.com/free/ free for the first 12 months].
and those in industry get a limted amount of resources [http://aws.amazon.com/free/ for the first 12 months].


The following are the 6 commands you would issue from your terminal window to run the file Example.tcl on a 100 node Amazon Cluster
The following are the 7 commands you would issue from a terminal/dos prompt while you are located in the directory containing the main script to run, Example.tcl, using OpenSeesMP on a 100 node Amazon Cluster and obtain the results that the application generates in the results folder back to the results folder in your current directory. Utilizing Amazon bidding process, see ([http://aws.amazon.com/ec2/purchasing-options/spot-instances/ Amazon Spot pricing], you could run such a 100 node machine for as little as $1.03 an hour ($0.0103 a node at time of writing) !
and obtain the results that the scipts created in the results folder back. Amazon bidding (see Amazon Spot pricing) could allow you to run such a 100 node machine for as little as $1.1 an hour!


<pre>
<pre>
starcluser start smallcluster
starcluster start smallcluster
starcluster put Example.tcl /root
starcluster put smallcluster ./ /root
starcluster sshmaster smallcluster
starcluster sshmaster smallcluster
mpiexec -n 100 OpenSees Example.tcl
mpiexec -n 100 OpenSeesMP Example.tcl;
starcluster get /root/results /results
exit
starcluster get smallcluster /root/results ./
starcluster terminate smallcluster
starcluster terminate smallcluster
</pre>
</pre>


The first command starts the cluster. The second command will copy the file Eample.tcl to the remote machine. The third command will cause you  to log on to your remote clusters 'master' node. The next command is the command you would issue on the remote machine to run a 100 processor job. The next command retrieves the contents of the /root/results directory. And lastly and an important command (assuming you don't want to use all your free allocation and go into bankruptcy) is the command to terminate the remote cluster.
 
For those doing this through a DOS terminal on Windows the starcluster put and get are slightly different:
 
<pre>
starcluster put smallcluster .\ /root
 
starcluster get smallcluster /root/results .\
 
</pre>
 
 
The first command starts the cluster. The second command will copy the file Example.tcl to the remote machine. The third command will cause you  to log on to your remote clusters 'master' node. The fourth command is the command you would issue on the remote machine to run a 100 processor job. The fifth command retrieves the contents of the /root/results directory. And lastly and an important command (assuming you don't want to use all your free allocation or max out your credit card!) is the command to terminate the remote cluster.
 
The following [http://opensees.berkeley.edu/OpenSees/code/SmallMP.zip zip file] contains an Example to try out.
 
And here is a [http://opensees.berkeley.edu/OpenSees/videos/AmazonEC2.mp4 video], at the end of which I demonstrate the process.


The initial set-up to enable this requires that:
The initial set-up to enable this requires that:
* You log in with a credit card and obtain an [http://aws.amazon.com/ec2/ Amazon EC2 account].  
* You log in with a credit card and obtain an [http://aws.amazon.com/ec2/ Amazon EC2 account] using their detailed [http://star.mit.edu/cluster/docs/latest/installation.html install instructions].
* You correctly install the [http://star.mit.edu/cluster/ StarCluster Software from MIT] on your computer.  
* You correctly install the [http://star.mit.edu/cluster/ StarCluster Software from MIT] on your computer.  
* Lastly you use the AMI that we have created for you in the StarCluster config file. The Amazon Ec2 AMI to specify is ami-e9ff1182 and is available in the US East Region.  
* Lastly you use the AMI that we have created for you in the StarCluster config file. The Amazon EC2 AMI to specify is '''ami-97c6cdfd''' and is available in the US East Region.  


The change to the config file is simply to swap out there ami and replace with ours. The NODE_IMAGE_ID lines should read:
The change to the StarCluster config file is simply to swap out their ami and replace with ours. The NODE_IMAGE_ID lines should read:


<pre>
<pre>
NODE_IMAGE_ID = ami-e9ff1182
NODE_IMAGE_ID = ami-97c6cdfd
</pre>
</pre>
   
   
It is assumed you know how to run OpenSees. But if you have got this far on the website you obviously can!
NOTES:
* The Amazon AMI is a 64bit Linux machine image.
* As you get more familiar with Amazon you will start using their Amazon storage service, Amazon S3, to store data (think remote hard drives in the sky). The data stored can be mounted to appear as part of the file system. This will allow you to preserve the data remotely without downloading it and allow you to use other AMIs that contain other applications to process the data.
* It is assumed you know how to run OpenSees and use the terminal window. But if you have got this far on the website you obviously can!
 
 
 
== Windows Specific ==
 
For those having issues using StarCluster on Windows machines please read the
[http://opensees.berkeley.edu/community/viewtopic.php?f=7&t=63276 thread on the message board]

Latest revision as of 17:34, 22 September 2017

OpenSees can run Sequentially and in Parallel on the resources provided by Amazon EC2. Amazon EC2 is the largest and most widely used cloud service provider. Once you are set up with Amazon and have installed some software from MIT, the running of the applications is trivial. For example, 7 commands from the terminal and you could be running a parallel application on a remote Amazon EC2 cluster of a few thousand nodes for a few dollars an hour. And it might not cost you anything, i.e. it could be FREE! Students and Educators can get free hours through the education program and and for those in Industry get a limited amount of resources free for the first 12 months.

The following are the 7 commands you would issue from a terminal/dos prompt while you are located in the directory containing the main script to run, Example.tcl, using OpenSeesMP on a 100 node Amazon Cluster and obtain the results that the application generates in the results folder back to the results folder in your current directory. Utilizing Amazon bidding process, see (Amazon Spot pricing, you could run such a 100 node machine for as little as $1.03 an hour ($0.0103 a node at time of writing) !

starcluster start smallcluster
starcluster put smallcluster ./ /root
starcluster sshmaster smallcluster
mpiexec -n 100 OpenSeesMP Example.tcl; 
exit
starcluster get smallcluster /root/results ./
starcluster terminate smallcluster


For those doing this through a DOS terminal on Windows the starcluster put and get are slightly different:

starcluster put smallcluster .\ /root

starcluster get smallcluster /root/results .\


The first command starts the cluster. The second command will copy the file Example.tcl to the remote machine. The third command will cause you to log on to your remote clusters 'master' node. The fourth command is the command you would issue on the remote machine to run a 100 processor job. The fifth command retrieves the contents of the /root/results directory. And lastly and an important command (assuming you don't want to use all your free allocation or max out your credit card!) is the command to terminate the remote cluster.

The following zip file contains an Example to try out.

And here is a video, at the end of which I demonstrate the process.

The initial set-up to enable this requires that:

  • You log in with a credit card and obtain an Amazon EC2 account using their detailed install instructions.
  • You correctly install the StarCluster Software from MIT on your computer.
  • Lastly you use the AMI that we have created for you in the StarCluster config file. The Amazon EC2 AMI to specify is ami-97c6cdfd and is available in the US East Region.

The change to the StarCluster config file is simply to swap out their ami and replace with ours. The NODE_IMAGE_ID lines should read:

NODE_IMAGE_ID = ami-97c6cdfd

NOTES:

  • The Amazon AMI is a 64bit Linux machine image.
  • As you get more familiar with Amazon you will start using their Amazon storage service, Amazon S3, to store data (think remote hard drives in the sky). The data stored can be mounted to appear as part of the file system. This will allow you to preserve the data remotely without downloading it and allow you to use other AMIs that contain other applications to process the data.
  • It is assumed you know how to run OpenSees and use the terminal window. But if you have got this far on the website you obviously can!


Windows Specific

For those having issues using StarCluster on Windows machines please read the thread on the message board