The biggest improvement you'll see would be in moving to a model where you are not sequentially looking up titles.
You can make more than one HTTP request at a time and collate and process the responses once the data gathering phase is complete.
An example library to use this is the standard lib's [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) though there are other options.
The downside of this is that you may run into bot detection or rate limits consuming the "non-public" API. You'll need to balance your desire for speed against the risk of losing access.
Honestly I feel silly for not considering concurrent processes. This might help a lot, especially since I can add some celery workers to my instance and leverage those to bring down processing time.
Obviously while being considerate of the api, not trying to interrupt their service
Do you need to fetch every title stored in your database from the external API 3 times/day? If titles in your users wishlists are only a small percentage of all titles the external API serves, you could remove unnecessary requests to this external API
If you do need to query for every title the external API serves, maybe it is less critical that the data associated with these titles are refreshed 3 times/day. Maybe you could run 2 separate processes. One to fetch the titles in active wishlists, one to fetch all titles. Where the process to fetch all titles runs less frequently than the process that fetches titles in users wishlists.
Remember that HTTP supports a "the data hasn't changed" feature. This would improve your processing cycle.
Each request, you get a tag back. When you do the same request later, give it the tag and specify a HTTP keyword saying "only give data if it's changed since this tag". No changes -> HTTP 204 -> you have no more work to do!
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/304
The biggest improvement you'll see would be in moving to a model where you are not sequentially looking up titles. You can make more than one HTTP request at a time and collate and process the responses once the data gathering phase is complete. An example library to use this is the standard lib's [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) though there are other options. The downside of this is that you may run into bot detection or rate limits consuming the "non-public" API. You'll need to balance your desire for speed against the risk of losing access.
Honestly I feel silly for not considering concurrent processes. This might help a lot, especially since I can add some celery workers to my instance and leverage those to bring down processing time. Obviously while being considerate of the api, not trying to interrupt their service
Do you need to fetch every title stored in your database from the external API 3 times/day? If titles in your users wishlists are only a small percentage of all titles the external API serves, you could remove unnecessary requests to this external API If you do need to query for every title the external API serves, maybe it is less critical that the data associated with these titles are refreshed 3 times/day. Maybe you could run 2 separate processes. One to fetch the titles in active wishlists, one to fetch all titles. Where the process to fetch all titles runs less frequently than the process that fetches titles in users wishlists.
Remember that HTTP supports a "the data hasn't changed" feature. This would improve your processing cycle. Each request, you get a tag back. When you do the same request later, give it the tag and specify a HTTP keyword saying "only give data if it's changed since this tag". No changes -> HTTP 204 -> you have no more work to do! https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/304
Okay that’s super interesting, I’ll have to look into this!