Remote Login and File Transfer ============================== A Little History ---------------- **The 1970's** Before the days of Personal Computers (PCs), computers were giant room sized machines called Main Frames as we discussed in the `Introduction`_ to this course. These computers were large machines running in an isolated and temperature controlled place (usually called The Machine Room ) and users logged on to the Main Frame. These computers supported many users, logged in simultaneously. **The early 1980's** Then came the PC. With a personal computer, the programs and files, etc. were all physically located in the box on the user's desk, where everything is self contained. Typically, only one user would be logged on to the computer at any time. **The mid 1980's** About the same time as the rise of the PC, the Internet (but not The Web which uses the Internet) became an important part of the worldwide cyberstructure allowing computers to transmit packets of data addressed to a unique computer somewhere else on the internet. Still The Web as you know it know did not exist. In the late 1980's we had three basic forms of using the Internet: + **Email** + **Telnet** This is a way of logging into a remote computer, as if you were sitting at a terminal connected to that computer. Once logged in, you got a shell prompt on that machice, just like you get when you open an xterm on your computer. You could issue unix commands and move around the file tree from the prompt. + **FTP** (File Transfer Protocol). This was a separate program from Telnet which allows users to transfer files to/from a remote computer. In the days before the World Wide Web (WWW), this was the way we got things from the internet. There were archives of open files that one could access by using FTP protocol to login as "`Anonymous`" with password "`your@email.address`". Once in, you could browse and download files---usually through a 9600 baud modem... **1990** Then came the World Wide Web, a system that linked files on different computers through " Browsers that could interpret files written in "HyperText Markup Language" (HTML). These files could contain "Links" to files, or other HTML pages. This system was developed at the European Particle Accelerator, `CERN `_. Recently the CERN Courier, the monthly news publication from CERN, published an article marking the 20th anniversary of the Web. It featured this document: .. image:: imgs/CCwww1_04_09.jpg which is a copy of Tim Berners-Lee's original memo to his boss outlining an idea for a system of servers, browsers, and interactions. Notice the handwritten note his boss wrote at the top: "*Vague but exciting...*", scribbled by Berner-Lee's boss. The idea went on to transform the world and lead to The Information Age of global connectivity. The complete article for the CERN Courier is `here `_ if you are interested in reading it. While we can now cruise the Web and find a truly amazing amount of information in a variety of forms, there are still times when it is necessary to login to a remote machine and perform tasks from a non- graphical (no mouse or menus) prompt. | Get Connected with SSH ====================== In the scientific world you usually remote login to other computers because + the other computer has special resources that are not available on your computer. For example, the remote machine might be one of those supercomputers you looked at in `Homework 1 `_. It is super-fast and has a ridiculously large disk. + your files are there. In this case you might use (S)FTP to get them + Often certain computers at an institution or lab have certain software applications on them. One computer might have data visualization packages, another might have the statistical software. Another may be a computing cluster for running programs. Yet another could be the storage server connected to a large disk. Thus we need to learn the basics of remote login. SSH: Secure Shell ----------------- Telnet, which I mentioned above, is a program that you run at the command line of an xterm shell window. It opens a connection to a remote machine which allows access via telnet, then presents the user a login prompt (asks for your username and password) on the remote machine. If the user can provide the correct password, a shell prompt awaiting commands was returned. The **problem** is... Telnet transactions are sent over the internet as **plain text**, i.e. as simple ASCII characters. These packets bounce from one router to the next to the next and so on, until they finally arrive at their destination. Anywhere along the way, an unscrupulous person could record packets traveling through nearby routers and read all the data. It is very simple to look for the characters " Username: " and " Password: " in the stream, then read the characters following this (up to the RETURN). The hacker would then have the person's username and password. .. note:: In about 2001 I discovered a breach of Pacific's network by finding a hacker logged into one of the machines here. After killing his login and looking around a bit, I found that he had set up a program to log router traffic which left a file which contained the usernames and passwords of all the people on the south campus who accessed our school database via telnet on a router in the Psych building. Because of this big flaw in Telnet, not many computers allow telnet access these days. The replacement for Telnet, is **Secure Shell** (SSH). Basically SSH is the same as Telnet--offering the user a shell prompt on the remote machine--however it encrypts all the data that is transmitted between the machines. Even if a hacker recorded the entire transmitted session, it would only look like giberish. The transmission can only be decoded by `public key decryption `_, which is quite strong if the keys are sufficiently large. Using SSH --------- First, we need a machine that we can ssh into. I've made an account for us to use on **physics-nix.stk.pacific.edu**. The *username* (*for all of us*) is `phystu` The *password* for this account is given under the *Resources* course tool menu item on the Sakai PHYS 27 site (look for *Resources* on the left hand tool menu). Open the Resourses Folder and look for the document on the SSH Accounts. The reason for putting this here is that I don't want to post the password to this account on the internet as a webpage. I only want those who have PHYS 27/193 access to Sakai to be able to view it. Using what you learned above, **ssh to physics-nix.stk.pacific.edu**. Remember that your USERNAME on this machine will be **phystu** (we will be sharing the account) and the domain name is given above (`physics-nix.stk.pacific.edu`). Also, .. warning:: You *must* be on campus to login to `physics-nix`. If you live off campus (outside the Pacific firewall), you will need to contact me about getting VPN (Virtual Private Network) access. Or you can just do the work of this chapter and the next by bringing your laptop on campus. | ----------------------------------------- .. topic:: Exercise Open a terminal shell, and type: `ssh phystu@physics-nix.stk.pacific.edu` | Once you are logged into **physics-nix.stk.pacific.edu** you will get a prompt that looks like this: :: phystu@physics-nix[~]> I've preloaded the `.bashrc` file for you. Do a directory listing `ls` to see what files are there (not much). ---------------------------------------------- | Emacs on a remote machine ------------------------- Now that you have a shell prompt on the remote machine ( **physics-nix**), you can issue commands there. For example, the ``which`` commmand tells you if a command or program is available, and the location of the program file. Type: `which gnuplot` to see the location of the **gnuplot** executable file on this machine. Suppose we wanted to edit a file on **physics-nix**. You could type **emacs**, and in a little while an Emacs window will pop up on your screen. The reason it takes a little while is that the full X-Windowed data is being sent through the network. **physics-nix** must get information about your graphics capability, the location of other windows, the status of your mouse, etc., then send the information for an Emacs window though the internet to the machine you are currently working from (the laptop or desktop on which you are now working). *AND--all this gets encrypted*. This is a fairly bulky process involving a lot of network traffic. Sometimes it's necessary--say, if you are using gnuplot; you have to be able to look at the graph. However, if you can save on bandwidth, your connection to the remote computer will be faster. This is why it's useful to be able to use emacs in "*Text Mode*" as we discussed when we learned in the `Emacs: GUI v. Text `_. You can start emacs in text mode on **physics-nix** by typing `emacs -nw` Remember this command? The **-nw** stands for "No Window". | Now you can run programs and edit files on the remote machine. What more do you need! You could be in a cyber cafe in Paris, *ssh'ed* into a computer here in the US, and--as long as you are familiar with moving around in a shell at the command prompt--have pretty much complete access and control of the computer here in the US. In my research, I use supercomputers around the world, like `Hopper `_ shown here at `NERSC `_ Once I am granted an account on one of these machines (usually by a grant writing procedure, where I get access for a certain amount of time), I can login via *ssh* from anywhere, just as you have done in this tutorial. I then use *emacs* to create and modify files, such as simulation programs which I then run on that particular computer, making use of the fact that *it is INCREDIBLY powerful and has a HUGE amount of disk space*. Once these programs have finished running, I can analyze the output on the supercomputer, or even better, reduce and *transfer* the data file to my local computer for analysis here (more on *file transfer* shortly). | -------------------------- .. topic:: Exercise Before you log out from **physics-nix**, use emacs to create a file called "**YOURNAME_nix.txt**". You will need this file below. where YOURNAME is (duh), *your name*. In the file, put the text :: This file was created by USERNAME on physics-nix.stk.pacific.edu. Date: today's date I'm feeling: --how are you feeling today?-- --------------------------- | Getting Back Out ---------------- When you are finished with your work on the remote computer, you logout by typing: `logout` oddly enough. Equivalently, you can type `exit` or even just: `CTRL-d` (this is my fave). All of these do the same thing: exit your session on the remote machine and log you out. | **Logout from** *physics-nix.stk.pacific.edu*, if you haven't already. | This should leave you at the prompt in your terminal shell on your laptop/computer. The reason I set up your `.bashrc` file to display: `sci[~]>` in the propmt is so that you will know that you entering commands back on your local laptop. For the same reason, I put `phystu@physics-nix[~]>` in the .bashrc file for user *phystu* on `physics-nix`. It helps keep track of which machine you are logged into when typing commands at the prompt. | SFTP and SCP ============ Sometimes you don't want a prompt on the remote machine to run commands,, but rather you need to *get* (or *put*) some files from (or to) the remote machine. That's when you call on **sftp**: *Secure File Transfer Protocol*. The syntax is very much like **ssh**. SFTP ---- SFTP stands for *Secure File Transfer Protocol*. | ------------------------- At the prompt on your local machine (your laptop, say), type: `sftp phystu@physics-nix.stk.pacific.edu` ------------------------- In a short while, **physics-nix.stk.pacific.edu** will ask you for the *password* for the phystu account (this is the same one as you just used to ssh to **physics-nix** above. Finally, you will get a prompt that looks like this: :: sftp> You can find all the commands that **sftp** accepts by typing **?** . :: sftp> ? Available commands: cd path Change remote directory to 'path' lcd path Change local directory to 'path' chgrp grp path Change group of file 'path' to 'grp' chmod mode path Change permissions of file 'path' to 'mode' chown own path Change owner of file 'path' to 'own' help Display this help text get remote-path [local-path] Download file lls [ls-options [path]] Display local directory listing ln oldpath newpath Symlink remote file lmkdir path Create local directory lpwd Print local working directory ls [path] Display remote directory listing lumask umask Set local umask to 'umask' mkdir path Create remote directory put local-path [remote-path] Upload file pwd Display remote working directory exit Quit sftp quit Quit sftp rename oldpath newpath Rename remote file rmdir path Remove remote directory rm path Delete remote file symlink oldpath newpath Symlink remote file version Show SFTP version !command Execute 'command' in local shell ! Escape to local shell ? Synonym for help You can intuit most of these. The most commonly used commands are: + **cd path** change directory on the remote machine + **lcd path** change directory on the local machine + **get filename** download a file from the remote to local machine + **put filename** upload a file from the local to remote machine + **mget filen\*** download many files (* matches any characters ) + **mput filen\*** upload many files (* matches any characters ) + **quit** or **exit** exit sftp | ---------------------------------------- .. topic:: Exercise Make sure you have opened an ``sftp`` session on ``physics-nix`` as described `above `_. Now, at the ``sftp>`` prompt, type **get USERNAME_nix.txt** You should see something like this: :: sftp> get jhetrick_nix.txt Fetching /home/jhetrick/jhetrick_nix.txt to /jhetrick.txt /home/jhetrick//jhetrick_nix.txt 100% 444 0.4KB/s 00:00 Now **quit** the *SFTP* session, by typing.... ``quit``. | Once you have your ``sci[~]>`` prompt back, indicating you are in the shell on your local computer, do an `ls` to see that you indeed downloaded the file **USERNAME.txt** *from* **physics-nix** to your laptop. Is it there? View the contents of the file (remember `how `_?) and check that its contents say that it was created by you on **physics-nix**. ---------------------------- | This is how you get file to and from remote computers. To upload files, you do pretty much the same thing, except that you use the **put** command. | ------------------------------ .. topic:: Exercise On **your local computer** (i.e. your laptop), create a file called "**USERNAME_local.txt**". Add the text "*This file was created on my super-deluxe Dell-o-tron.*" (or something similar that describes your laptop/computer) to the file, so that you know that this was the locally created file. Upload this file to **physics-nix**. ------------------------------- | I'll be able to see this file on physics-nix so I'll be able to tell if you have done it correctly. This is part of `Homework 10 `_. | SCP --- Another way to transfer files is to use *Secure CP*. This tool is a mashup of SFTP and the usual unix CP command. `Recall `_ how we copy a file from one place to another, by using the **cp** command: :: cp file1 file2 cp file1 dir/ cp file1 dir/newname to: + make a *copy* of `file1` called `file2`, in the same directory + make a *copy* of `file1` in the directory `dir/` + make a *copy* of `file1` called `newname` in directory `dir/` In general, the ``cp`` command takes the form: **cp source target** meaning that the first thing (the *source*) is copied to the second thing (*the target*). | **SCP** allows us to use essentially the same one-liner command, but include reference to files and directories on remote machines. The syntax goes like this: ``sci[]> scp file1 janedoe@remote.mach.ine:dir1/dir2/file2`` This command will + open a connection to the machine called `remote.mach.ine` and login as user **janedoe** and ask for her password on the remote machine. This is what is meant by the text: **janedoe@remote.mach.ine:** + put a copy of `file1` in the *sub-subdirectory* `dir1/dir2` beneath *janedoe's* HOME directory on the remote machine, called *file2*. .. note:: if your username on the local machine AND the remote machine is the same, you don't have to include the username (janedoe, in this example). You could just give the remote machine name. In that case, you would type **scp file1 remote.mach.ine:** to put ``file1`` in your HOME directory on ``remote.mach.ine`` ). | ----------------------------- .. topic:: Exercise To try this out for yourself, open a terminal shell on your laptop, and make a test file (*emacs, "blah blah", save->* ``mynametest.txt``). Now scp this file to *physics-nix*. If the file you just made is in the current directory, you would do this: ``scp mynametest.txt phystu@physics-nix.serv.pacific.edu:`` Rememeber the password for *phystu* on physics-nix; you'll be asked for it. This should put a copy of the file ``mynametest.txt``, in the home directory of user: *phystu*, on the remote machine: **physics-nix.serv.pacific.edu**. ------------------------------- | Now, logon to *physics-nix.serv.pacific.edu* using **ssh**: ``ssh phystu@physics-nix.serv.pacific.edu`` | At the prompt on *physics-nix*, do a directory listing and verify that the file ``mynametest.txt`` is there. Edit it with emacs (*remotely, using text-mode emacs* as discussed above), and add a sentence to the end: "*This file was successfully copied to physics-nix with scp*" | Now, logout from physics-nix (type exit, or CTRL-d). You should have the ``sci[~]>`` prompt back in your local terminal shell. Delete the *local* ``mynametest.txt`` file (*rm mynametest.txt*). We are going to grab back the copy we just put on physics-nix. Do an ``ls`` to verify it is gone. Now, get a copy from physics-nix: ``scp phystu@physics-nix.serv.pacific.edu:mynametest.txt .`` | Notice this time the *source* is on the remote machine, and the *target* location/name is just **.** . Remember, **.**, a single period, means "**here**" in unix. It stands for "*the current directory*". This should prompt you for your password on the remote machine, then copy the requested file to the current directory on the local machine. | If you were successful, you should have a file on your local machine that has the added sentence: "*This file was successfully copied to physics-nix with scp*". That's how you *know* you have pulled the copy from physics-nix.serv.pacific.edu, *after* you put it there in the first place. | What's the difference between *sftp* and *scp*? ----------------------------------------------- Basically, *sftp* and *scp* do the same thing--move files from one computer to another. However, *sftp* is a bigger program, whereas *scp* is a oneliner that allows you to grab a single (or multiple) file(s), if you know where on the remote machine they are. You have to give scp "the full path/name" information to the file in order to find it. With *sftp*, you get an ``sftp>`` prompt. SFTP really gives you a "*file transfer shell*--an environment in which you have a prompt and a number of commands that you can type, such as ``help``, ``ls``, ``cd ..``, etc. This ``sftp`` shell allows you to move around within the *sftp* environment (on the remote machine). So, if you can't remember where the file is located on the remote machine, *sftp* would be the way to go. You can login to the remote machine, ``cd`` around, ``ls`` to view the directory contents, and then grab the files you need with the ``get`` command. | Homework ======== If you did the above exercises, you are ready for Homework 10, which is here: `Homework 10 `_.