Problem:

We have an existing application that does a lot of web retrieving and parsing using a custom process and system we've developed. This data is core to our business, and it's critical to our scale that this data be retrieved efficiently.

Our retriever logic is spread over a number of different services, and even languages. In addition, we have a prototype that we've built that adds incremental functionality into our app. But, this again, is in a SEPARATE API-based application.

Need:

So, I'm looking to

• Move the logic for handling sessions and proxies into this same platform. right now, it's embedded into our main Rails application.
• Merge the current APIs which reside in separate endpoints into a single API-based endpoint.
• Reimplement the logic of the new prototype that we've built into the same platform.
• Centralize caching to S3 for retrieves to reduce additional retries
• Implement session management and cookie management
• Write data to a Postgres database after parsing as sessions are called
• Manage session information in the Postgres database as well
• Implement a cron type function that will regularly retrieve data as needed
• Implement an API on the front end for our Rails application to accelerate job requests.

Plan:

We've evaluated alternatives and are looking to move our logic to Crawlee. We currently believe that this can provide a lot of the capabilities we're rolling ourselves currently and have to maintain. We've wrapped this in Express, and we currently have some logic there to manage things.