UniqueSoft
/
firecrawl
mirror of https://github.com/mendableai/firecrawl
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,058 Commits 223 Branches 1 Tag 45 MiB
Commit Graph

1058 Commits

Author SHA1 Message Date
Gergo Moricz 10957b748b fix(bull): requeue jobs after restart 2024-07-12 13:55:53 +02:00
Nicolas 961b27811d
Merge pull request #386 from mendableai/feat/fire-engine-fallback-for-sitemap 
[Feat] Added fire-engine fallback for getting sitemaps
 2024-07-11 20:38:01 -04:00
Nicolas 84de63dbeb
Merge pull request #375 from StefanTerdell/self-host-qol 
Self-hosting quality of life fixes
 2024-07-11 20:37:39 -04:00
Nicolas 30c1118713
Merge pull request #326 from mendableai/feat/save-docs-on-supabase 
[Feat] Added implementation for saving docs on supabase
 2024-07-11 20:27:41 -04:00
jhoseph88 68828a5b5c Pass along current, total, current_step, and current_url in js sdk 2024-07-11 19:37:09 -04:00
Gergo Moricz 7e3a368684 fix: unpause globally 2024-07-12 00:05:35 +02:00
Gergo Moricz ee1d41406e feat: unpause by http request 2024-07-11 23:56:36 +02:00
Gergo Moricz f64a2d8668 fix: rename fly tomls to original 2024-07-11 23:21:02 +02:00
Gergo Moricz bd84290b9e fix: reenable hyperdx 2024-07-11 23:20:51 +02:00
Gergo Moricz 09bca05b20 feat: fix iteration 3 (actually works) 2024-07-11 23:14:15 +02:00
Gergo Moricz 9cd7d79b64 feat: avoid double SIGINT crashing 2024-07-11 20:35:15 +02:00
Gergo Moricz eaa8db4b19 fix(fly): raise kill timeout for graceful shutdown 2024-07-11 20:09:06 +02:00
Gergo Moricz bffb9f8fd0 feat: stuck job restoration iteration 2 2024-07-11 20:08:21 +02:00
rafaelsideguide 86d0e88a91 removed hyperdx (they also have graceful shutdown) and tried to change the process for running on server. It didn't work. 2024-07-10 18:29:55 -03:00
rafaelsideguide 7c3cc89a80 Merge branch 'feat/save-docs-on-supabase' of https://github.com/mendableai/firecrawl into feat/save-docs-on-supabase 2024-07-09 18:48:29 -03:00
rafaelsideguide 9ad06fdf56 added fire-engine fallback for getting sitemaps 2024-07-09 16:07:53 -03:00
Gergo Moricz 1a07e9d23b feat: pick up and commit interrupted jobs from/to DB 2024-07-09 15:57:38 +02:00
Gergo Moricz 6a524e1bae feat: pick up and commit interrupted jobs from/to DB 2024-07-09 14:56:47 +02:00
Gergo Moricz 77aa46588f feat: graceful exit handler 2024-07-09 14:29:32 +02:00
Nicolas fcc67a3c9e
Merge pull request #370 from mendableai/bug/fixing-cicd 
dependabot for security checks, fixed crawl test
 2024-07-08 19:26:18 -03:00
Eric Ciarla 51e6579d70 Update README.md 2024-07-08 17:09:17 -04:00
Eric Ciarla f8e79dd729 Update integrations 2024-07-08 17:02:20 -04:00
Eric Ciarla afb49e21e7 Update SDKs to MIT license 2024-07-08 13:37:53 -04:00
Nicolas 002bfdf639
Merge pull request #374 from StefanTerdell/fix-372 
Only check Supabase if configured to when cancelling job
 2024-07-07 22:53:15 -03:00
Stefan Terdell 188fe56203 Optional jobId webhook URL templating 2024-07-07 15:11:45 +02:00
Stefan Terdell fcedcccf2d Include self hosted webhook env and allow connections to localhost 2024-07-07 15:10:35 +02:00
Stefan Terdell a2ae5f81d9 Only check Supabase if configured to 2024-07-07 15:06:31 +02:00
Nicolas e779dbbe26
Merge pull request #371 from mendableai/feat/issue-342 
[Feat] Added veeva to special case params
 2024-07-06 12:25:22 -03:00
rafaelsideguide c2bba54b4f Added veeva to special case params 2024-07-05 16:58:07 -03:00
rafaelsideguide a2cdc520e6 dependabot for security checks, fixed crawl test 2024-07-05 14:49:03 -03:00
Rafael Miller 2b36de605a
Merge pull request #349 from mendableai/dependabot/npm_and_yarn/apps/api/prod-deps-5b38a50718 
apps/api(deps): bump the prod-deps group in /apps/api with 28 updates
 2024-07-05 14:05:50 -03:00
rafaelsideguide 0ab6cef471 Merge remote-tracking branch 'origin/main' into dependabot/npm_and_yarn/apps/api/prod-deps-5b38a50718 2024-07-05 14:00:10 -03:00
Nicolas 914897c9d2 Merge branch 'main' into feat/save-docs-on-supabase 2024-07-05 12:27:22 -03:00
rafaelsideguide 538dc63035 Fixing rate-limiter-flexible package version 
Redis version <3.0.2 throws TS bug:
https://github.com/animir/node-rate-limiter-flexible/issues/228
 2024-07-05 12:12:00 -03:00
rafaelsideguide d4e1a9724f Merge branch 'dependabot/npm_and_yarn/apps/test-suite/dev-deps-ffe2a14739' 2024-07-05 11:16:49 -03:00
Rafael Miller c570fa92cf
Merge pull request #347 from mendableai/dependabot/npm_and_yarn/apps/test-suite/prod-deps-d16537e256 
apps/test-suite(deps): bump the prod-deps group in /apps/test-suite with 6 updates
 2024-07-05 10:18:35 -03:00
Nicolas a7aaa7e57e
Update SELF_HOST.md 2024-07-04 17:49:09 -03:00
Nicolas 5551f704ba
Merge pull request #362 from snippet/self-host-docs-ts-playwright-service 
(Docs) Self Host added new ts playwright service instructions
 2024-07-04 17:48:41 -03:00
Nicolas 8f46b8218a
Merge pull request #361 from snippet/ts-playwright-service-docker 
setting up docker to ts playwright service
 2024-07-04 17:47:41 -03:00
Nicolas 32849b017f Nick: 2024-07-03 20:18:11 -03:00
Nicolas 5ecd9cb6f5
Merge pull request #363 from mendableai/nsc/logging-scrapers 
Logging for all scraper methods
 2024-07-03 18:47:22 -03:00
Nicolas 066d92f643 Update single_url.ts 2024-07-03 18:38:17 -03:00
Nicolas f5b2fbd7e8 Nick: revision 2024-07-03 18:06:53 -03:00
Nicolas 2d30cc6117 Nick: comments 2024-07-03 18:01:54 -03:00
Nicolas 90c54c32fd Nick: refactor 2024-07-03 18:01:17 -03:00
Nicolas 90cf799a3c Update single_url.ts 2024-07-03 17:56:21 -03:00
Nicolas b36406e465 Nick: log scrpaers 2024-07-03 17:28:53 -03:00
Jeff Pereira 8d09c5f9b5 (Docs) Self Host added new ts playwright service instructions 2024-07-03 12:00:44 -07:00
Jeff Pereira b4292c1ea3 setting up docker to ts playwright service 2024-07-03 11:55:39 -07:00
Nicolas abb44bb112
Merge pull request #346 from mendableai/dependabot/pip/apps/playwright-service/prod-deps-8f04296377 
apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates
 2024-07-03 01:07:09 -03:00