Share
Embed
pixiv App Night 20190611
6月11日に開催された「pixiv App Night」の発表資料です。
「わたしは機械学習プロジェクトで 技術的負債を抱えました」
ARIYAMA Keiji
June 11, 2019
Transcript
43 slides
-
C-LIS CO., LTD. ༗ࢁܓೋʢ,FJKJ"3*:"."ʣ $-*4$0 -5% Photo :
Koji MORIGUCHI (MORIGCHOWDER) "OESPJEΞϓϦ։ൃνϣοτσΩϧ ʮझຯͰػցֶशΛ͍ͬͯΔऀͩʯ ͬͯ·ͤΜ -
ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ
͚ σʔληοτཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ͚ σʔληοτ ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧஔ ը૾ૹ৴ ఆ݁Ռ -
ϥϕϧͷछྨ PSJHJOBM@BSU OTGX GBWPSJUF QIPUP JMMVTU DPNJD GBDF
GFNBMF NFHBOF TDISPPM@VOJGPSN CMB[FS@VOJGPSN TBJMPS@VOJGPSN HM LFNPOP NBMF CM DBU EPH GPPE EJTMJLF -
def download(self, url, output_dir): response = requests.get(url) response.raise_for_status() fd, file_path
= tempfile.mkstemp(dir=output_dir) os.close(fd) with open(file_path, mode='wb') as fp: fp.write(response.content) response.close() return file_path ϑΝΠϧσΟεΫϦϓλΛރׇͤ͞Δ IUUQTOJTIJNVSBIBUFOBEJBSZPSHFOUSZ -
ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ
͚ σʔληοτཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ͚ σʔληοτ ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧஔ ը૾ૹ৴ ఆ݁Ռ -
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will
be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From model_res5.py:28: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead. -
def layers(tag_name, image, keep_prob=0.5, training=False): scope = '%s/%s'
% (tag_name, NAME) with tf.variable_scope(scope): conv = tf.layers.conv2d(image, BASE_CHANNEL, [1, 1], [1, 1], padding='SAME', activation=tf.nn.relu, use_bias=True, trainable=training, name='conv_top') def layers(tag_name, image, rate=0.5, training=False): scope = '%s/%s' % (tag_name, NAME) with tf.variable_scope(scope): conv = tf.keras.layers.Conv2D(BASE_CHANNEL, [1, 1], [1, 1], padding='SAME', activation=tf.nn.relu, use_bias=True, name='conv_top')(image) -
ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ
͚ σʔληοτཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ͚ σʔληοτ ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧஔ ը૾ૹ৴ ఆ݁Ռ -
̎ͭͷϓϩηεʢֶशͱධՁʣ train process eval process ֶशࡁΈύϥϝʔλʔ ֶशσʔληοτ ධՁσʔληοτ
࠷ྑύϥϝʔλʔ ߋ৽ ධՁ มߋΛࢹ ߋ৽ -
- name: train-single-rocm container: image: rocm/tensorflow:rocm2.2-tf1.13-python3 command: [/workdir/scripts/train_single.sh] env: -
name: DATASET_DIR value: "/dataset/tfrecords_classifier_latest" - name: CATALOGS_DIR value: "{{inputs.parameters.catalogs-dir}}" - name: TRAIN_DIR value: "data/{{inputs.parameters.train-dir}}" - name: SUMMARY_DIR value: "data/{{inputs.parameters.summary-dir}}" - name: CONFIG_FILE value: "scripts/{{inputs.parameters.config-file}}" - name: BATCH_SIZE value: "{{inputs.parameters.batch-size}}" - name: LERANING_RATE value: "{{inputs.parameters.learning-rate}}" - name: STEPS value: "{{inputs.parameters.steps}}" resources: limits: amd.com/gpu: 2 requests: amd.com/gpu: 2 - name: train-single-cuda container: image: tensorflow/tensorflow:1.13.1-gpu-py3 command: [/workdir/scripts/train_single.sh] env: - name: DATASET_DIR value: "/dataset/tfrecords_classifier_latest" - name: CATALOGS_DIR value: "{{inputs.parameters.catalogs-dir}}" - name: TRAIN_DIR value: "data/{{inputs.parameters.train-dir}}" - name: SUMMARY_DIR value: "data/{{inputs.parameters.summary-dir}}" - name: CONFIG_FILE value: "scripts/{{inputs.parameters.config-file}}" - name: BATCH_SIZE value: "{{inputs.parameters.batch-size}}" - name: LERANING_RATE value: "{{inputs.parameters.learning-rate}}" - name: STEPS value: "{{inputs.parameters.steps}}" resources: limits: nvidia.com/gpu: 2 requests: nvidia.com/gpu: 2 -
#!/bin/sh HIP_VISIBLE_DEVICES=0 CUDA_VISIBLE_DEVICES=0 \ python3 sources/server/picture_single_discriminator/eval.py \ --learning_config $CONFIG_FILE \
--tfrecords_dir $DATASET_DIR \ --catalogs_dir $CATALOGS_DIR \ --train_dir $TRAIN_DIR \ --summary_dir $SUMMARY_DIR & HIP_VISIBLE_DEVICES=1 CUDA_VISIBLE_DEVICES=1 \ python3 sources/server/picture_single_discriminator/train.py \ --learning_config $CONFIG_FILE \ --tfrecords_dir $DATASET_DIR \ --catalogs_dir $CATALOGS_DIR \ --train_dir $TRAIN_DIR \ --summary_dir $SUMMARY_DIR \ --batch_size $BATCH_SIZE \ --learning_rate $LERANING_RATE \ --num_gpus 1 \ --step $STEPS -
ධՁϓϩηεΛഇࢭ train process eval process ֶशࡁΈύϥϝʔλʔ ֶशσʔληοτ ධՁσʔληοτ
࠷ྑύϥϝʔλʔ ߋ৽ ධՁ มߋΛࢹ ߋ৽ × -
ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ
͚ σʔληοτཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ͚ σʔληοτ ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧஔ ը૾ૹ৴ ఆ݁Ռ