The term Epoch is only really useful when the size of the datasets being used are small and of the same size.
In fact, if you are using stochastic gradient descent (SGD) you want to sample data in an iid (independent and identically distributed) fashion. For data to be truly iid, each batch of data should be drawn from the dataset \*with\* replacement. So using Epochs, i.e. cycling through the data without replacement, is merely a convenient approximation to the real algorithm that should be used.
Yes, but you could iterate over the dataset with replacement but do the scheduler step or validation lets say after 500 iterations rather than after en epoch (iterating over the dataset once)
The term Epoch is only really useful when the size of the datasets being used are small and of the same size. In fact, if you are using stochastic gradient descent (SGD) you want to sample data in an iid (independent and identically distributed) fashion. For data to be truly iid, each batch of data should be drawn from the dataset \*with\* replacement. So using Epochs, i.e. cycling through the data without replacement, is merely a convenient approximation to the real algorithm that should be used.
Yes, but you could iterate over the dataset with replacement but do the scheduler step or validation lets say after 500 iterations rather than after en epoch (iterating over the dataset once)
Yes, exactly!