How Do I Upload a Zip File to Google Drive?
How to successfully add large information sets to Google Drive
And use them in Google Colab…
Introduction
In this post, I will explicate how to add large information sets to Google Drive so they can exist accessed from Google Colab for processing and modeling.
Whereas uploading a single file can be done with the elevate and drop interface of Google Drive, it becomes more difficult with a big number of files. Dragging the whole folder containing 1GB of files just fails and freezes Google Bulldoze. The culling is to elevate a zipped folder. This procedure is commonly successful and does not take fifty-fifty that long (a couple of minutes with 1GB file) merely the problem comes with unzipping the file in Google Drive itself which results in random files missing.
My trials for iii days to successfully upload photograph data for CNN training in society to use free GPU led me to establish this culling. I am going to describe here step by footstep how to upload successfully large information sets and then they tin be processed by Google Colab and take advantage of VMs provided by their services.
Steps
1. Zippo the folder with the files. In my case, I had a folder called 'train' with 70257 .jpg files that were taking around 1GB.
2. Upload the zipped file using Google Drive Interface. I have uploaded mine to an empty directory chosen data.
3. Open a new Google Colab file and mount it to Google Drive to be able to admission the zip file. If yous practise non take Colab installed you can follow this article that explains how to exercise it.
The control beneath volition start the mounting process.
from google.colab import bulldoze
bulldoze.mount('/content/gdrive') You will be asked to authorize access to Google Drive.
Follow instructions to give potency by copy-pasting the lawmaking and y'all should be mounted.
four. At present extract files to the local environment with the post-obit command.
!unzip gdrive/My\ Drive/information/train.zip Annotation that my train.zero file in 'data' folder is located in the Google Drive root directory. Y'all will need to modify the path accordingly to where your file is located.
You should see file unzipping.
Information technology takes less than 1 minute to unzip 1GB and so you should not wait too long. Once you are comfortable that this command is working you tin can apply the variation beneath that suppresses the output.
!unzip gdrive/My\ Drive/data/railroad train.zip > /dev/null In one case the cell has executed you tin can meet the files have appeared in the local train folder. Y'all can find it on the left-mitt side of the colab interface.
5. You tin can utilise the files for annihilation correct now. Juts access them from the new 'train' binder. In my case, I can display the first epitome using the following code.
import tensorflow as tf
img = tf.keras.preprocessing.epitome.load_img('railroad train/abs_000.jpg')
img 
You tin now utilise your data for annihilation y'all wish. In my example, it was training a CNN using free GPU.
Summary
The process that I described above worked best for me! And then please gratuitous to re-create it.
On the other mitt, information technology would be much better to unzip the files in Google Drive, so they only stay at that place. This yet was 'mission impossible' for me for thee days in a row. Every time I have unzipped the files and saved them in Google Drive there were missing photos in the terminate.
I have tried to practice it programmatically with Colab as well every bit using Zip Extractor connected to Google Bulldoze. Both methods resulted in missing files with no alert most it. I have also tried two different Google accounts and the problem persisted.
After losing 3 days of trial and error. I came with the process described above and it works smoothly for me every time I rerun the code.
I promise it helps others that try to load big data sets to Google Drive in order to process them in Colab. If you are one that has institute a better method or was able to unzip large files in Google Bulldoze, I would exist curious how this has been done.
Happy coding!
PS: I am writing articles that explain basic Data Science concepts in a elementary and comprehensible manner on Medium and aboutdatablog.com . You tin subscribe to my email list to get notified every time I write a new article. And if you are not a Medium member withal yous can bring together here .
Below there are another posts yous may savor:
Source: https://towardsdatascience.com/how-to-successfully-add-large-data-sets-to-google-drive-130beb320f1a
0 Response to "How Do I Upload a Zip File to Google Drive?"
Post a Comment