Saving information generated during a job is easy.
On a FloydHub deep learning server your code has access to a directory called
/output directory is a special directory that is used to store
information you want to save for future use after a job finishes. Anything
saved in the
/output directory at the time a job finishes will be preserved
and can be accessed and reused later.
The most common thing users save is model checkpoints, but anything that ends
up in the
/output directory at the end of a job will be saved (use your
Let's work through a couple of examples to see how to save data during a job.
This job prints the string "Hello, world!", and saves it to a file called
hello.txt is located in the
/output directory, it will
be saved and available after the job finishes:
$ floyd init my_awesome_hello_world_project $ floyd run "echo 'Hello, world!' > /output/hello.txt" Syncing code ...
If you are not familiar with what
echo 'Hello, world!' > /output/hello.txt does, here's a quick
echo 'Hello, world!'part outputs the string
>part of the command redirects the printed output of
echo 'Hello, world!'(which is, of course,
Hello world!) to the file specified after the
/output/hello.txtpart of the command specifies where the
Hello,world!should be written to:
hello.txtis in the
/outputdirectory, it will be preserved for future reference and use.
In this example, we'll use Python to save some data to a file in the
directory. Put this code in a file named
with open('/output/myfile.txt', 'a') as f: f.write('Please save me!\n')
If you run this code locally on your computer, you'll probably get something like this:
Traceback (most recent call last): File "save_example.py", line 1, in <module> with open('/output/myfile.txt', 'a') as f: IOError: [Errno 2] No such file or directory: '/output/myfile.txt'
That's because there is no
/output directory on your computer. In contrast,
every job on FloydHub runs on a server that has a
/output directory, so the
command won't fail on FloydHub. Let's run it with the following commands:
$ floyd init save_example_2 $ floyd run "python save_example.py" Creating project run. Total upload size: 267.0B Syncing code ...
Now that we've completed a couple trivial examples, let's do something more useful and realistic.
Here is a sample Tensorflow example that saves a model checkpoint. Because we
write (save) the data to the
/output directory, we'll be able to use it
later. A future job can use this model checkpoint as a starting point.
Consider this partial code, and note the call to
import tensorflow as tf ... saver = tf.train.Saver() with tf.Session() as sess: sess.run(init) ... save_path = saver.save(sess, '/output/model.ckpt') print("Model saved in file: %s" % save_path) ...
Because model is stored under the special
/output directory, it will be saved
even after your job ends, and can be used again in future jobs.
Viewing Saved Output Data¶
You can view the saved output of a job using the
floyd output command:
$ floyd output mckay/projects/quick-start/1 Opening output directory in your browser...
Alternatively, you can browse or download the saved output by visiting the
Output tab of the job on your dashboard as shown in the image below:
Using output as a data source¶
You can use the output of one job as the input to your next job. To see how to mount output data, please see this guide
Help make this document better¶
This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (by clicking the edit icon at the top next to the title).
- Open an issue about this page to report a problem.