2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | Started.
2026-09-05 02:02:14.496 | error | iqjkg5bmexmcur | 2026-09-05 06:01:43.983 | ERROR    | __main__:embed_handler:265 - Embed job failed at validate\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | Traceback (most recent call last):\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/runpod/handler.py", line 381, in <module>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     runpod.serverless.start({"handler": handler})\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      │          │                 └ <function handler at 0x7b4b96b45ee0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      │          └ <function start at 0x7b4e7ae06700>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      └ <module 'runpod.serverless' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <module 'runpod' from '/app/.venv/lib/python3.13/site-packages/runpod/__init__.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py", line 181, in start\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     worker.main(config)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      │    └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      └ <function main at 0x7b4e7ae060c0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <module 'runpod.serverless.worker' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 66, in main\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     run_worker(config)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │          └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <function run_worker at 0x7b4e7ae05c60>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 53, in run_worker\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     job_scaler.start()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │          └ <function JobScaler.start at 0x7b4e7ae05580>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 121, in start\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     asyncio.run(self.run())\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │       │   │    └ <function JobScaler.run at 0x7b4e7ae056c0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │       │   └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │       └ <function run at 0x7b4e7d59c9a0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <module 'asyncio' from '/usr/local/lib/python3.13/asyncio/__init__.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 196, in run\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     return runner.run(main)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │      │   └ <coroutine object JobScaler.run at 0x7b4b96ba9690>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │      └ <function Runner.run at 0x7b4e7d5f1120>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            └ <asyncio.runners.Runner object at 0x7b4b96b1c7d0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 119, in run\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     return self._loop.run_until_complete(task)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │    │     │                  └ <Task pending name='Task-12' coro=<JobScaler.run() running at /app/.venv/lib/python3.13/site-packages/runpod/serverless/modul...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │    │     └ <function BaseEventLoop.run_until_complete at 0x7b4e7d5eac00>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            └ <asyncio.runners.Runner object at 0x7b4b96b1c7d0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 713, in run_until_complete\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     self.run_forever()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    └ <function BaseEventLoop.run_forever at 0x7b4e7d5eab60>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 684, in run_forever\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     self._run_once()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    └ <function BaseEventLoop._run_once at 0x7b4e7d5f09a0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2061, in _run_once\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     handle._run()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      └ <function Handle._run at 0x7b4e7d59dee0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96c76080>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     self._context.run(self._callback, *self._args)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            │    │           └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96c76080>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            │    └ <member '_callback' of 'Handle' objects>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96c76080>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    └ <member '_context' of 'Handle' objects>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96c76080>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 348, in handle_job\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     await self.jobs_handler(session, self.config, job)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |           │    │            │        │    │       └ {'delayTime': 12473, 'id': 'sync-7d9915ad-4517-40f6-88f3-5262031f84d9-u1', 'input': {'base': 'probe', 'task': 'embed'}, 'stat...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |           │    │            │        │    └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |           │    │            │        └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |           │    │            └ <aiohttp.client.ClientSession object at 0x7b4b96bf9400>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |           │    └ <function handle_job at 0x7b4e7ae04ae0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |           └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 219, in handle_job\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     job_result = await run_job(config["handler"], job)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |                        │       │                  └ {'delayTime': 12473, 'id': 'sync-7d9915ad-4517-40f6-88f3-5262031f84d9-u1', 'input': {'base': 'probe', 'task': 'embed'}, 'stat...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |                        │       └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |                        └ <function run_job at 0x7b4e7ae04b80>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 257, in run_job\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     handler_return = handler(job)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |                      │       └ {'delayTime': 12473, 'id': 'sync-7d9915ad-4517-40f6-88f3-5262031f84d9-u1', 'input': {'base': 'probe', 'task': 'embed'}, 'stat...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |                      └ <function handler at 0x7b4b96b45ee0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/runpod/handler.py", line 284, in handler\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     return embed_handler(payload)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │             └ {'base': 'probe', 'task': 'embed'}\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            └ <function embed_handler at 0x7b4b96b45e40>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | > File "/app/runpod/handler.py", line 221, in embed_handler\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     chunks_url = payload["chunks_url"]\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |                  └ {'base': 'probe', 'task': 'embed'}\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | KeyError: 'chunks_url'\n
2026-09-05 02:02:14.496 | error | iqjkg5bmexmcur | Failed to return job results. | 400, message='Bad Request', url='https://api.runpod.ai/v2/e9d1h9f9oehnr7/job-done/iqjkg5bmexmcur/sync-7d9915ad-4517-40f6-88f3-5262031f84d9-u1?gpu=NVIDIA+GeForce+RTX+5090&isStream=false'
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | Finished.
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | Jobs in queue: 1
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | Started.
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | 2026-09-05 06:01:46.072 | INFO     | __main__:embed_handler:232 - GPU: NVIDIA GeForce RTX 5090\n
2026-09-05 02:02:14.496 | error | iqjkg5bmexmcur | 2026-09-05 06:01:46.504 | ERROR    | __main__:embed_handler:265 - Embed job failed at download\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | Traceback (most recent call last):\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/runpod/handler.py", line 381, in <module>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     runpod.serverless.start({"handler": handler})\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      │          │                 └ <function handler at 0x7b4b96b45ee0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      │          └ <function start at 0x7b4e7ae06700>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      └ <module 'runpod.serverless' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <module 'runpod' from '/app/.venv/lib/python3.13/site-packages/runpod/__init__.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py", line 181, in start\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     worker.main(config)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      │    └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      └ <function main at 0x7b4e7ae060c0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <module 'runpod.serverless.worker' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 66, in main\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     run_worker(config)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │          └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <function run_worker at 0x7b4e7ae05c60>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 53, in run_worker\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     job_scaler.start()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │          └ <function JobScaler.start at 0x7b4e7ae05580>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 121, in start\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     asyncio.run(self.run())\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │       │   │    └ <function JobScaler.run at 0x7b4e7ae056c0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │       │   └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │       └ <function run at 0x7b4e7d59c9a0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <module 'asyncio' from '/usr/local/lib/python3.13/asyncio/__init__.py'>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 196, in run\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     return runner.run(main)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │      │   └ <coroutine object JobScaler.run at 0x7b4b96ba9690>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │      └ <function Runner.run at 0x7b4e7d5f1120>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            └ <asyncio.runners.Runner object at 0x7b4b96b1c7d0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 119, in run\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     return self._loop.run_until_complete(task)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │    │     │                  └ <Task pending name='Task-12' coro=<JobScaler.run() running at /app/.venv/lib/python3.13/site-packages/runpod/serverless/modul...\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │    │     └ <function BaseEventLoop.run_until_complete at 0x7b4e7d5eac00>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |            └ <asyncio.runners.Runner object at 0x7b4b96b1c7d0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 713, in run_until_complete\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     self.run_forever()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    └ <function BaseEventLoop.run_forever at 0x7b4e7d5eab60>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 684, in run_forever\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     self._run_once()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    └ <function BaseEventLoop._run_once at 0x7b4e7d5f09a0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2061, in _run_once\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     handle._run()\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │      └ <function Handle._run at 0x7b4e7d59dee0>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96bc4b50>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     self._context.run(self._callback, *self._args)\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            │    │           └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96bc4b50>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            │    └ <member '_callback' of 'Handle' objects>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    │            └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96bc4b50>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     │    └ <member '_context' of 'Handle' objects>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7b4b96bc4b50>()>\n
2026-09-05 02:02:14.496 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 348, in handle_job\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     await self.jobs_handler(session, self.config, job)\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │    │            │        │    │       └ {'delayTime': 50, 'id': '1a841fb0-3e05-4044-941c-a877a8f1b6f9-u2', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'https...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │    │            │        │    └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │    │            │        └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │    │            └ <aiohttp.client.ClientSession object at 0x7b4b96bf9400>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │    └ <function handle_job at 0x7b4e7ae04ae0>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7b4b96bf8830>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 219, in handle_job\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     job_result = await run_job(config["handler"], job)\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |                        │       │                  └ {'delayTime': 50, 'id': '1a841fb0-3e05-4044-941c-a877a8f1b6f9-u2', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'https...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |                        │       └ {'handler': <function handler at 0x7b4b96b45ee0>, 'reference_counter_start': 3098222.023935961, 'rp_args': {'rp_log_level': N...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |                        └ <function run_job at 0x7b4e7ae04b80>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 257, in run_job\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     handler_return = handler(job)\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |                      │       └ {'delayTime': 50, 'id': '1a841fb0-3e05-4044-941c-a877a8f1b6f9-u2', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'https...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |                      └ <function handler at 0x7b4b96b45ee0>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |   File "/app/runpod/handler.py", line 284, in handler\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     return embed_handler(payload)\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |            │             └ {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |            └ <function embed_handler at 0x7b4b96b45e40>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | > File "/app/runpod/handler.py", line 235, in embed_handler\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     texts = download_json(chunks_url)["texts"]\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |             │             └ 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.chunks.json?AWSAccessKeyId=AKIAVRUVWZZ2...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |             └ <function download_json at 0x7b4b96b45da0>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |   File "/app/runpod/handler.py", line 209, in download_json\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     response.raise_for_status()\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     │        └ <function Response.raise_for_status at 0x7b4e7df36a20>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     └ <Response [400 Bad Request]>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |   File "/app/.venv/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |     raise HTTPStatusError(message, request=request, response=self)\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │               │                │                 └ <Response [400 Bad Request]>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │               │                └ <Request('GET', 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.chunks.json?AWSAccessKe...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           │               └ "Client error '400 Bad Request' for url 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17...\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur |           └ <class 'httpx.HTTPStatusError'>\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | \n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.chunks.json?AWSAccessKeyId=AKIAVRUVWZZ226UOGW7L&Signature=m9zl7RzCDqnrB%2Bzl5%2B0zTlfHulE%3D&Expires=1788602505'\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\n
2026-09-05 02:02:14.497 | error | iqjkg5bmexmcur | 2026-09-05 06:01:46.886 | ERROR    | __main__:embed_handler:270 - Marker upload failed too: Client error '400 Bad Request' for url 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.embed.complete.json?AWSAccessKeyId=AKIAVRUVWZZ226UOGW7L&Signature=0MYp098T9EhvatM%2BOdLR6lNsPKo%3D&Expires=1788602505'\n
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\n
2026-09-05 02:02:14.497 | error | iqjkg5bmexmcur | Failed to return job results. | 400, message='Bad Request', url='https://api.runpod.ai/v2/e9d1h9f9oehnr7/job-done/iqjkg5bmexmcur/1a841fb0-3e05-4044-941c-a877a8f1b6f9-u2?gpu=NVIDIA+GeForce+RTX+5090&isStream=false'
2026-09-05 02:02:14.497 | info | iqjkg5bmexmcur | Finished.
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | --- Starting Serverless Worker |  Version 1.12.0 ---\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Running 8 fitness check(s)...
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | GPU binary test passed: 1 GPU(s) healthy (CUDA 13.0)
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Memory check passed: 154.45GB available (of 186.34GB total)
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Disk space check passed: 24.98GB free (99.9% available)
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Network connectivity passed: Connected to 8.8.8.8 (23ms)
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | CUDA version check passed: 13.0 (minimum: 11.8)
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | CUDA initialization passed: 1 device(s) initialized successfully
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | GPU compute benchmark passed: Matrix multiply completed in 40ms
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | All fitness checks passed. (482.94ms)
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Jobs in queue: 1
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Jobs in progress: 1
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Started.
2026-09-05 03:15:23.415 | error | n4azcq9v5n5c8z | 2026-09-05 07:14:51.430 | ERROR    | __main__:embed_handler:265 - Embed job failed at validate\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | Traceback (most recent call last):\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 381, in <module>\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     runpod.serverless.start({"handler": handler})\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     │      │          │                 └ <function handler at 0x7baa42d7f9c0>\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     │      │          └ <function start at 0x7bad29acade0>\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     │      └ <module 'runpod.serverless' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py'>\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     └ <module 'runpod' from '/app/.venv/lib/python3.13/site-packages/runpod/__init__.py'>\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py", line 183, in start\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     worker.main(config)\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     │      │    └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.415 | info | n4azcq9v5n5c8z |     │      └ <function main at 0x7bad29aca840>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <module 'runpod.serverless.worker' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py'>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 66, in main\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     run_worker(config)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │          └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <function run_worker at 0x7bad29aca2a0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 53, in run_worker\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     job_scaler.start()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │          └ <function JobScaler.start at 0x7bad29ac9bc0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 121, in start\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     asyncio.run(self.run())\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │       │   │    └ <function JobScaler.run at 0x7bad29ac9d00>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │       │   └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │       └ <function run at 0x7bad2c3e5da0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <module 'asyncio' from '/usr/local/lib/python3.13/asyncio/__init__.py'>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 196, in run\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     return runner.run(main)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │      │   └ <coroutine object JobScaler.run at 0x7baa42dc69b0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │      └ <function Runner.run at 0x7bad2c2d8400>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            └ <asyncio.runners.Runner object at 0x7baa42ec1e50>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 119, in run\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     return self._loop.run_until_complete(task)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │    │     │                  └ <Task pending name='Task-12' coro=<JobScaler.run() running at /app/.venv/lib/python3.13/site-packages/runpod/serverless/modul...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │    │     └ <function BaseEventLoop.run_until_complete at 0x7bad2c2ddee0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            └ <asyncio.runners.Runner object at 0x7baa42ec1e50>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 713, in run_until_complete\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     self.run_forever()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    └ <function BaseEventLoop.run_forever at 0x7bad2c2dde40>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 684, in run_forever\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     self._run_once()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    └ <function BaseEventLoop._run_once at 0x7bad2c2dfc40>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2061, in _run_once\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     handle._run()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      └ <function Handle._run at 0x7bad2c3951c0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa4311c760>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     self._context.run(self._callback, *self._args)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            │    │           └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa4311c760>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            │    └ <member '_callback' of 'Handle' objects>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa4311c760>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    └ <member '_context' of 'Handle' objects>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa4311c760>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 349, in handle_job\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     await self.jobs_handler(session, self.config, job)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            │        │    │       └ {'delayTime': 2012186, 'id': 'sync-dff2e993-d11f-42f5-9f08-c9aad9c67455-u2', 'input': {'base': 'probe', 'task': 'embed'}, 'st...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            │        │    └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            │        └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            └ <aiohttp.client.ClientSession object at 0x7baa42d1eba0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    └ <function handle_job at 0x7bad29ac9120>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 219, in handle_job\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     job_result = await run_job(config["handler"], job)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |                        │       │                  └ {'delayTime': 2012186, 'id': 'sync-dff2e993-d11f-42f5-9f08-c9aad9c67455-u2', 'input': {'base': 'probe', 'task': 'embed'}, 'st...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |                        │       └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |                        └ <function run_job at 0x7bad29ac91c0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 257, in run_job\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     handler_return = handler(job)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |                      │       └ {'delayTime': 2012186, 'id': 'sync-dff2e993-d11f-42f5-9f08-c9aad9c67455-u2', 'input': {'base': 'probe', 'task': 'embed'}, 'st...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |                      └ <function handler at 0x7baa42d7f9c0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 284, in handler\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     return embed_handler(payload)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │             └ {'base': 'probe', 'task': 'embed'}\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            └ <function embed_handler at 0x7baa42d7f920>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | > File "/app/runpod/handler.py", line 221, in embed_handler\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     chunks_url = payload["chunks_url"]\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |                  └ {'base': 'probe', 'task': 'embed'}\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | KeyError: 'chunks_url'\n
2026-09-05 03:15:23.416 | error | n4azcq9v5n5c8z | Failed to return job results. | 400, message='Bad Request', url='https://api.runpod.ai/v2/e9d1h9f9oehnr7/job-done/n4azcq9v5n5c8z/sync-dff2e993-d11f-42f5-9f08-c9aad9c67455-u2?gpu=NVIDIA+GeForce+RTX+5090&isStream=false'
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | Finished.
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | Jobs in queue: 1
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | Started.
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | 2026-09-05 07:14:52.857 | INFO     | __main__:embed_handler:232 - GPU: NVIDIA GeForce RTX 5090\n
2026-09-05 03:15:23.416 | error | n4azcq9v5n5c8z | 2026-09-05 07:14:53.249 | ERROR    | __main__:embed_handler:265 - Embed job failed at download\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | Traceback (most recent call last):\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 381, in <module>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     runpod.serverless.start({"handler": handler})\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      │          │                 └ <function handler at 0x7baa42d7f9c0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      │          └ <function start at 0x7bad29acade0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      └ <module 'runpod.serverless' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py'>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <module 'runpod' from '/app/.venv/lib/python3.13/site-packages/runpod/__init__.py'>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py", line 183, in start\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     worker.main(config)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      │    └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      └ <function main at 0x7bad29aca840>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <module 'runpod.serverless.worker' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py'>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 66, in main\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     run_worker(config)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │          └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <function run_worker at 0x7bad29aca2a0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 53, in run_worker\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     job_scaler.start()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │          └ <function JobScaler.start at 0x7bad29ac9bc0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 121, in start\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     asyncio.run(self.run())\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │       │   │    └ <function JobScaler.run at 0x7bad29ac9d00>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │       │   └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │       └ <function run at 0x7bad2c3e5da0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <module 'asyncio' from '/usr/local/lib/python3.13/asyncio/__init__.py'>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 196, in run\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     return runner.run(main)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │      │   └ <coroutine object JobScaler.run at 0x7baa42dc69b0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │      └ <function Runner.run at 0x7bad2c2d8400>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            └ <asyncio.runners.Runner object at 0x7baa42ec1e50>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 119, in run\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     return self._loop.run_until_complete(task)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │    │     │                  └ <Task pending name='Task-12' coro=<JobScaler.run() running at /app/.venv/lib/python3.13/site-packages/runpod/serverless/modul...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │    │     └ <function BaseEventLoop.run_until_complete at 0x7bad2c2ddee0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |            └ <asyncio.runners.Runner object at 0x7baa42ec1e50>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 713, in run_until_complete\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     self.run_forever()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    └ <function BaseEventLoop.run_forever at 0x7bad2c2dde40>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 684, in run_forever\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     self._run_once()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    └ <function BaseEventLoop._run_once at 0x7bad2c2dfc40>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2061, in _run_once\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     handle._run()\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │      └ <function Handle._run at 0x7bad2c3951c0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42f5b490>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     self._context.run(self._callback, *self._args)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            │    │           └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42f5b490>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            │    └ <member '_callback' of 'Handle' objects>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    │            └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42f5b490>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     │    └ <member '_context' of 'Handle' objects>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42f5b490>()>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 349, in handle_job\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |     await self.jobs_handler(session, self.config, job)\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            │        │    │       └ {'delayTime': 1921895, 'id': 'a4f23ebf-39ff-44f2-afa9-a8fb804eaeeb-u2', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': '...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            │        │    └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            │        └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    │            └ <aiohttp.client.ClientSession object at 0x7baa42d1eba0>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           │    └ <function handle_job at 0x7bad29ac9120>\n
2026-09-05 03:15:23.416 | info | n4azcq9v5n5c8z |           └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 219, in handle_job\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     job_result = await run_job(config["handler"], job)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                        │       │                  └ {'delayTime': 1921895, 'id': 'a4f23ebf-39ff-44f2-afa9-a8fb804eaeeb-u2', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': '...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                        │       └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                        └ <function run_job at 0x7bad29ac91c0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 257, in run_job\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     handler_return = handler(job)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                      │       └ {'delayTime': 1921895, 'id': 'a4f23ebf-39ff-44f2-afa9-a8fb804eaeeb-u2', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': '...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                      └ <function handler at 0x7baa42d7f9c0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 284, in handler\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     return embed_handler(payload)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │             └ {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            └ <function embed_handler at 0x7baa42d7f920>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | > File "/app/runpod/handler.py", line 235, in embed_handler\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     texts = download_json(chunks_url)["texts"]\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |             │             └ 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.chunks.json?AWSAccessKeyId=AKIAVRUVWZZ2...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |             └ <function download_json at 0x7baa42d7f880>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 209, in download_json\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     response.raise_for_status()\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │        └ <function Response.raise_for_status at 0x7bad2cd0ea20>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <Response [400 Bad Request]>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     raise HTTPStatusError(message, request=request, response=self)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │               │                │                 └ <Response [400 Bad Request]>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │               │                └ <Request('GET', 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.chunks.json?AWSAccessKe...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │               └ "Client error '400 Bad Request' for url 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           └ <class 'httpx.HTTPStatusError'>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.chunks.json?AWSAccessKeyId=AKIAVRUVWZZ226UOGW7L&Signature=sa%2FsBn9ZBfwu9%2FAi5FlKai%2B%2FI2E%3D&Expires=1788604970'\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\n
2026-09-05 03:15:23.417 | error | n4azcq9v5n5c8z | 2026-09-05 07:14:53.660 | ERROR    | __main__:embed_handler:270 - Marker upload failed too: Client error '400 Bad Request' for url 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3f17.embed.complete.json?AWSAccessKeyId=AKIAVRUVWZZ226UOGW7L&Signature=qXXYE7Hq8L9T6qlUdaUVKlOBdaM%3D&Expires=1788604970'\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\n
2026-09-05 03:15:23.417 | error | n4azcq9v5n5c8z | Failed to return job results. | 400, message='Bad Request', url='https://api.runpod.ai/v2/e9d1h9f9oehnr7/job-done/n4azcq9v5n5c8z/a4f23ebf-39ff-44f2-afa9-a8fb804eaeeb-u2?gpu=NVIDIA+GeForce+RTX+5090&isStream=false'
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | Finished.
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | Jobs in queue: 1
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | Started.
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | 2026-09-05 07:14:55.463 | INFO     | __main__:embed_handler:232 - GPU: NVIDIA GeForce RTX 5090\n
2026-09-05 03:15:23.417 | error | n4azcq9v5n5c8z | 2026-09-05 07:14:55.860 | ERROR    | __main__:embed_handler:265 - Embed job failed at download\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | Traceback (most recent call last):\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 381, in <module>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     runpod.serverless.start({"handler": handler})\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │      │          │                 └ <function handler at 0x7baa42d7f9c0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │      │          └ <function start at 0x7bad29acade0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │      └ <module 'runpod.serverless' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py'>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <module 'runpod' from '/app/.venv/lib/python3.13/site-packages/runpod/__init__.py'>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/__init__.py", line 183, in start\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     worker.main(config)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │      │    └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │      └ <function main at 0x7bad29aca840>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <module 'runpod.serverless.worker' from '/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py'>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 66, in main\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     run_worker(config)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │          └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <function run_worker at 0x7bad29aca2a0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/worker.py", line 53, in run_worker\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     job_scaler.start()\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │          └ <function JobScaler.start at 0x7bad29ac9bc0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 121, in start\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     asyncio.run(self.run())\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │       │   │    └ <function JobScaler.run at 0x7bad29ac9d00>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │       │   └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │       └ <function run at 0x7bad2c3e5da0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <module 'asyncio' from '/usr/local/lib/python3.13/asyncio/__init__.py'>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 196, in run\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     return runner.run(main)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │      │   └ <coroutine object JobScaler.run at 0x7baa42dc69b0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │      └ <function Runner.run at 0x7bad2c2d8400>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            └ <asyncio.runners.Runner object at 0x7baa42ec1e50>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 119, in run\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     return self._loop.run_until_complete(task)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │    │     │                  └ <Task pending name='Task-12' coro=<JobScaler.run() running at /app/.venv/lib/python3.13/site-packages/runpod/serverless/modul...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │    │     └ <function BaseEventLoop.run_until_complete at 0x7bad2c2ddee0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            └ <asyncio.runners.Runner object at 0x7baa42ec1e50>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 713, in run_until_complete\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     self.run_forever()\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    └ <function BaseEventLoop.run_forever at 0x7bad2c2dde40>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 684, in run_forever\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     self._run_once()\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    └ <function BaseEventLoop._run_once at 0x7bad2c2dfc40>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2061, in _run_once\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     handle._run()\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │      └ <function Handle._run at 0x7bad2c3951c0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42e0f7c0>()>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     self._context.run(self._callback, *self._args)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    │            │    │           └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42e0f7c0>()>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    │            │    └ <member '_callback' of 'Handle' objects>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    │            └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42e0f7c0>()>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     │    └ <member '_context' of 'Handle' objects>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     └ <Handle <_asyncio.TaskStepMethWrapper object at 0x7baa42e0f7c0>()>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py", line 349, in handle_job\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     await self.jobs_handler(session, self.config, job)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │    │            │        │    │       └ {'delayTime': 5336, 'id': 'f36dc1dd-c00b-419c-a264-9b2ffe0cfc76-u1', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'htt...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │    │            │        │    └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │    │            │        └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │    │            └ <aiohttp.client.ClientSession object at 0x7baa42d1eba0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           │    └ <function handle_job at 0x7bad29ac9120>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |           └ <runpod.serverless.modules.rp_scale.JobScaler object at 0x7baa42d1dfd0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 219, in handle_job\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     job_result = await run_job(config["handler"], job)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                        │       │                  └ {'delayTime': 5336, 'id': 'f36dc1dd-c00b-419c-a264-9b2ffe0cfc76-u1', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'htt...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                        │       └ {'handler': <function handler at 0x7baa42d7f9c0>, 'reference_counter_start': 2836374.855874326, 'rp_args': {'rp_log_level': N...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                        └ <function run_job at 0x7bad29ac91c0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_job.py", line 257, in run_job\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     handler_return = handler(job)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                      │       └ {'delayTime': 5336, 'id': 'f36dc1dd-c00b-419c-a264-9b2ffe0cfc76-u1', 'input': {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'htt...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |                      └ <function handler at 0x7baa42d7f9c0>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |   File "/app/runpod/handler.py", line 284, in handler\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     return embed_handler(payload)\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            │             └ {'base': 'e8cf6cc4320f3f17', 'chunks_url': 'https://qubera-extracts.s3.amazonaws.com/documents/e8cf6cc4320f3f17/e8cf6cc4320f3...\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |            └ <function embed_handler at 0x7baa42d7f920>\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | \n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z | > File "/app/runpod/handler.py", line 235, in embed_handler\n
2026-09-05 03:15:23.417 | info | n4azcq9v5n5c8z |     texts = download_json(chunks_url)["texts"]\n