Want to train directly on a Spark DataFrame from pyspark.Īre using a standard gradient descent optimization process as your training loop. We recommend using Horovod Spark Estimators if you:Īre using Keras ( tf.keras or keras) or PyTorch for training. Processing (from Spark DataFrames to deep learning datasets), model training loop, model checkpointing, metrics Use the same underlying mechanism to launch Horovod on Spark executors, but the Estimator API abstracts the data We provide two APIs for running Horovod on Spark: a high level Estimator API and a lower level Run API. In situations where training data originates from Spark, this enablesĪ tight model design loop in which data processing, model training, and The horovod.spark package provides a convenient wrapper around Horovod that makes running distributed training
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |