Centralized Learning vs. Distributed Learning

Centralized vs. Distributed Learning (Ref. Bo et. al., When Machine Learning Meets Privacy: A Survey and Outlook)

Centralized Learning

• Training data is centralized in a machine
• The centralized entity trains and hosts the model
• Used in outsourced models (ML-as-a-service) e.g. Amazon AWS ML
• Pros: High accuracy, Cons: Operators have access to sensitive training data

Distributed Learning

Note: There are three variants: collaborative learning, federated learning and split learning. In this, I am focusing on federated learning. This is how it works:

• Each client downloads the global model from a centralized location
• Build an updated local model using local training data
• Share model parameters with the centralized server
• The centralized server updates the global model by averaging input from all local clients

Why DL?

• Dataset could be too large to process in a centralized server
• Building customized models for each user based on their data
• Users are not willing to share their data

Important characteristics of DL:

• Training data is local to devices (Pros: more privacy preserving)
• Cons: the global model’s accuracy is usually less than that of a model built with centralized data
• Scalable (as the workload is distributed, it makes the solution scalable)
• Ability build customized local models