Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
tensorflow
GitHub Repository: tensorflow/docs-l10n
 Path: blob/master/site/ko/io/tutorials/azure.ipynb
25118 views
Kernel: Python 3
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

TensorFlow를 사용한 Azure Blob 저장소

주의: 이 노트북은 Python 패키지 외에도 npm install --user를 사용하여 패키지를 설치합니다. 로컬에서 실행할 때 주의하세요.

개요

이 튜토리얼에서는 TensorFlow IO의 Azure 파일 시스템 통합을 통해 TensorFlow와 함께 Azure Blob 저장소에서 파일 읽기 및 쓰기를 사용하는 방법을 보여줍니다.

Azure Blob 저장소에서 파일을 읽고 쓰려면 Azure 저장소 계정이 필요합니다. Azure Storage Key는 환경 변수를 통해 제공되어야 합니다.

os.environ['TF_AZURE_STORAGE_KEY'] = '<key>'

저장소 계정 이름과 컨테이너 이름은 파일 이름 uri의 일부입니다.

azfs://<storage-account-name>/<container-name>/<path>

이 튜토리얼에서는 데모 목적으로 Azure 저장소 에뮬레이터인 Azurite를 선택적으로 설정할 수 있습니다. Azurite 에뮬레이터에서는 TensorFlow를 사용하여 Azure Blob 저장소 인터페이스를 통해 파일을 읽고 쓸 수 있습니다.

설정 및 사용

필수 패키지를 설치하고 런타임 다시 시작하기

try:
  %tensorflow_version 2.x 
except Exception:
  pass

!pip install tensorflow-io
TensorFlow 2.x selected. Collecting tensorflow-io Downloading https://files.pythonhosted.org/packages/c0/d0/c5d7adce72c6a6d7c9a59c062150f60b5404c706578a0922f7dc2835713c/tensorflow_io-0.12.0-cp36-cp36m-manylinux2010_x86_64.whl (20.1MB) |████████████████████████████████| 20.1MB 42.7MB/s Requirement already satisfied: tensorflow<2.2.0,>=2.1.0 in /tensorflow-2.1.0/python3.6 (from tensorflow-io) (2.1.0) Requirement already satisfied: opt-einsum>=2.3.2 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (3.1.0) Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (2.1.0) Requirement already satisfied: grpcio>=1.8.6 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.27.2) Requirement already satisfied: wheel>=0.26; python_version >= "3" in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.34.2) Requirement already satisfied: google-pasta>=0.1.6 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.1.8) Requirement already satisfied: tensorboard<2.2.0,>=2.1.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (2.1.0) Requirement already satisfied: wrapt>=1.11.1 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.12.0) Requirement already satisfied: scipy==1.4.1; python_version >= "3" in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.4.1) Requirement already satisfied: protobuf>=3.8.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (3.11.3) Requirement already satisfied: termcolor>=1.1.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.1.0) Requirement already satisfied: absl-py>=0.7.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.9.0) Requirement already satisfied: keras-applications>=1.0.8 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.0.8) Requirement already satisfied: astor>=0.6.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.8.1) Requirement already satisfied: gast==0.2.2 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.2.2) Requirement already satisfied: keras-preprocessing>=1.1.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.1.0) Requirement already satisfied: numpy<2.0,>=1.16.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.18.1) Requirement already satisfied: six>=1.12.0 in /tensorflow-2.1.0/python3.6 (from tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.14.0) Requirement already satisfied: setuptools>=41.0.0 in /tensorflow-2.1.0/python3.6 (from tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (45.2.0) Requirement already satisfied: markdown>=2.6.8 in /tensorflow-2.1.0/python3.6 (from tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (3.2.1) Requirement already satisfied: google-auth<2,>=1.6.3 in /tensorflow-2.1.0/python3.6 (from tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.11.2) Requirement already satisfied: requests<3,>=2.21.0 in /tensorflow-2.1.0/python3.6 (from tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (2.22.0) Requirement already satisfied: werkzeug>=0.11.15 in /tensorflow-2.1.0/python3.6 (from tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.0.0) Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /tensorflow-2.1.0/python3.6 (from tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.4.1) Requirement already satisfied: h5py in /tensorflow-2.1.0/python3.6 (from keras-applications>=1.0.8->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (2.10.0) Requirement already satisfied: cachetools<5.0,>=2.0.0 in /tensorflow-2.1.0/python3.6 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (4.0.0) Requirement already satisfied: rsa<4.1,>=3.1.4 in /tensorflow-2.1.0/python3.6 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (4.0) Requirement already satisfied: pyasn1-modules>=0.2.1 in /tensorflow-2.1.0/python3.6 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.2.8) Requirement already satisfied: idna<2.9,>=2.5 in /tensorflow-2.1.0/python3.6 (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /tensorflow-2.1.0/python3.6 (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /tensorflow-2.1.0/python3.6 (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (2019.11.28) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /tensorflow-2.1.0/python3.6 (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.25.8) Requirement already satisfied: requests-oauthlib>=0.7.0 in /tensorflow-2.1.0/python3.6 (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (1.3.0) Requirement already satisfied: pyasn1>=0.1.3 in /tensorflow-2.1.0/python3.6 (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (0.4.8) Requirement already satisfied: oauthlib>=3.0.0 in /tensorflow-2.1.0/python3.6 (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow<2.2.0,>=2.1.0->tensorflow-io) (3.1.0) Installing collected packages: tensorflow-io Successfully installed tensorflow-io-0.12.0

Azurite 설치 및 설정하기(선택 사항)

Azure 저장소 계정을 사용할 수 없는 경우 Azure 저장소 인터페이스를 에뮬레이트하는 Azurite를 설치하고 설정하기 위해 다음이 필요합니다.

!npm install azurite@2.7.0
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN saveError ENOENT: no such file or directory, open '/content/package.json' npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN enoent ENOENT: no such file or directory, open '/content/package.json' npm WARN content No description npm WARN content No repository field. npm WARN content No README data npm WARN content No license field. + [email protected] added 116 packages from 141 contributors in 6.591s 
# The path for npm might not be exposed in PATH env,
# you can find it out through 'npm bin' command
npm_bin_path = get_ipython().getoutput('npm bin')[0]
print('npm bin path: ', npm_bin_path)

# Run `azurite-blob -s` as a background process. 
# IPython doesn't recognize `&` in inline bash cells.
get_ipython().system_raw(npm_bin_path + '/' + 'azurite-blob -s &')
npm bin path: /content/node_modules/.bin

TensorFlow를 사용하여 Azure 저장소에 파일 읽기 및 쓰기

다음은 TensorFlow의 API를 사용하여 Azure 저장소에서 파일을 읽고 쓰는 예입니다.

tensorflow-io 패키지를 가져오면 tensorflow-io가 사용할 azfs를 자동으로 등록하므로 TensorFlow의 다른 파일 시스템(예: POSIX 또는 GCS)과 동일한 방식으로 동작합니다.

Azure 저장소 키는 TF_AZURE_STORAGE_KEY 환경 변수를 통해 제공되어야 합니다. 그렇지 않으면 TF_AZURE_USE_DEV_STORAGE를 True로 설정하여 Azurite 에뮬레이터를 대신 사용할 수 있습니다.

import os
import tensorflow as tf
import tensorflow_io as tfio

# Switch to False to use Azure Storage instead:
use_emulator = True

if use_emulator:
  os.environ['TF_AZURE_USE_DEV_STORAGE'] = '1'
  account_name = 'devstoreaccount1'
else:
  # Replace <key> with Azure Storage Key, and <account> with Azure Storage Account
  os.environ['TF_AZURE_STORAGE_KEY'] = '<key>'
  account_name = '<account>'

  # Alternatively, you can use a shared access signature (SAS) to authenticate with the Azure Storage Account
  os.environ['TF_AZURE_STORAGE_SAS'] = '<your sas>'
  account_name = '<account>'

pathname = 'az://{}/aztest'.format(account_name)
tf.io.gfile.mkdir(pathname)

filename = pathname + '/hello.txt'
with tf.io.gfile.GFile(filename, mode='w') as w:
  w.write("Hello, world!")

with tf.io.gfile.GFile(filename, mode='r') as r:
  print(r.read())
Hello, world!

구성

TensorFlow에서 Azure Blob 저장소의 구성은 항상 환경 변수를 통해 수행됩니다. 다음은 사용 가능한 구성의 전체 목록입니다.

  • TF_AZURE_USE_DEV_STORAGE: 'az://devstoreaccount1/container/file.txt'와 같은 연결에 로컬 개발 저장소 에뮬레이터를 사용하려면 1로 설정합니다. 다른 모든 설정보다 우선하므로 다른 연결을 사용하도록 unset할 수 있습니다.

  • TF_AZURE_STORAGE_KEY: 사용 중인 저장소 계정의 계정 키입니다.

  • TF_AZURE_STORAGE_USE_HTTP: https 전송을 사용하지 않으려면 아무 값으로나 설정합니다. https 기본값을 사용하려면 unset을 사용합니다.

  • TF_AZURE_STORAGE_BLOB_ENDPOINT: Blob 저장소의 엔드포인트로 설정합니다. 기본값은 .core.windows.net입니다.