UniqueSoft
/
firecrawl
mirror of https://github.com/mendableai/firecrawl
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,676 Commits 223 Branches 1 Tag 45 MiB
Commit Graph

48 Commits

Author SHA1 Message Date
rafaelsideguide 291d9e375b now using compiled go/C lib with koffi 2024-09-03 10:56:07 -03:00
Nicolas 41eb620959 Nick: prompt option, still need to convert to new structured outputs 2024-08-29 21:00:57 -03:00
Nicolas f277a0e2bb Update package.json 2024-08-26 18:17:09 -03:00
Gergő Móricz e7f267b6fe Merge branch 'main' into v1-webscraper 2024-08-23 17:21:54 +02:00
rafaelsideguide 5f60a55967 workflow and npm now running v1 tests 2024-08-22 15:28:49 -03:00
Gergő Móricz 920702cdde Update builder to handle uploading sourcemaps 2024-08-21 19:08:03 +02:00
Gergő Móricz 86942728e3 Add metadata for queue-worker and Express 2024-08-21 17:58:27 +02:00
Nicolas 35decb1af2 Nick: 2024-08-21 12:35:03 -03:00
Gergő Móricz eb84673b06 feat: crawl status websocket WIP 2024-08-17 01:04:14 +02:00
Nicolas 78ca94251c
Merge pull request #480 from mendableai/nsc/hyper-v81 
Reduce metrics ingestion w/ HyperDX v0.8.1
 2024-08-16 14:34:14 -04:00
Gergő Móricz 8b7569f8f3 add zod, create middleware, update openapi declaration, add crawl logic 2024-08-15 23:30:33 +02:00
Gergő Móricz 5fc7fcb77c
Merge branch 'main' into feat/queue-scrapes 2024-08-07 16:35:44 +02:00
Nicolas e195ddbef4 Merge branch 'main' into nsc/hyper-v81 2024-08-05 20:47:39 -04:00
Nicolas a28ecc1f61 Nick: caching 2024-07-30 18:59:35 -04:00
Nicolas 7e002a8b06 Nick: bull mq 2024-07-30 13:27:23 -04:00
Nicolas c446942306 Nick: 2024-07-29 21:28:29 -04:00
Nicolas 968a2dc753 Nick: 2024-07-29 18:37:09 -04:00
Nicolas 04942bb9de Nick: 2024-07-29 18:31:43 -04:00
Gergo Moricz 8160c311c0 fix queue stuck bug via lock setting changes 2024-07-17 21:31:25 +02:00
Nicolas c9073a747c Nick: 2024-07-16 22:41:13 -04:00
Nicolas 30c1118713
Merge pull request #326 from mendableai/feat/save-docs-on-supabase 
[Feat] Added implementation for saving docs on supabase
 2024-07-11 20:27:41 -04:00
Gergo Moricz ee1d41406e feat: unpause by http request 2024-07-11 23:56:36 +02:00
rafaelsideguide 538dc63035 Fixing rate-limiter-flexible package version 
Redis version <3.0.2 throws TS bug:
https://github.com/animir/node-rate-limiter-flexible/issues/228
 2024-07-05 12:12:00 -03:00
dependabot[bot] c2e00d1998
apps/api(deps): bump the prod-deps group in /apps/api with 28 updates 
Bumps the prod-deps group in /apps/api with 28 updates:

| Package | From | To |
| --- | --- | --- |
| [@anthropic-ai/sdk](https://github.com/anthropics/anthropic-sdk-typescript) | `0.20.9` | `0.24.3` |
| [@bull-board/api](https://github.com/felixmosh/bull-board/tree/HEAD/packages/api) | `5.19.2` | `5.20.5` |
| [@bull-board/express](https://github.com/felixmosh/bull-board/tree/HEAD/packages/express) | `5.19.2` | `5.20.5` |
| [@hyperdx/node-opentelemetry](https://github.com/hyperdxio/hyperdx-js) | `0.7.0` | `0.8.0` |
| [@nangohq/node](https://github.com/NangoHQ/nango/tree/HEAD/packages/node-client) | `0.36.101` | `0.40.8` |
| [@sentry/node](https://github.com/getsentry/sentry-javascript) | `7.116.0` | `8.13.0` |
| [@supabase/supabase-js](https://github.com/supabase/supabase-js) | `2.43.4` | `2.44.2` |
| [ajv](https://github.com/ajv-validator/ajv) | `8.15.0` | `8.16.0` |
| [async-mutex](https://github.com/DirtyHairy/async-mutex) | `0.4.1` | `0.5.0` |
| [bull](https://github.com/OptimalBits/bull) | `4.12.9` | `4.15.0` |
| [date-fns](https://github.com/date-fns/date-fns) | `2.30.0` | `3.6.0` |
| [express-rate-limit](https://github.com/express-rate-limit/express-rate-limit) | `6.11.2` | `7.3.1` |
| [glob](https://github.com/isaacs/node-glob) | `10.4.1` | `10.4.2` |
| [json-schema-to-zod](https://github.com/StefanTerdell/json-schema-to-zod) | `2.1.0` | `2.3.0` |
| [keyword-extractor](https://github.com/michaeldelorenzo/keyword-extractor) | `0.0.25` | `0.0.28` |
| [langchain](https://github.com/langchain-ai/langchainjs) | `0.1.37` | `0.2.8` |
| [logsnag](https://github.com/LogSnag/logsnag.js) | `0.1.8` | `1.0.0` |
| [mongoose](https://github.com/Automattic/mongoose) | `8.4.1` | `8.4.4` |
| [natural](https://github.com/NaturalNode/natural) | `6.12.0` | `7.0.7` |
| [openai](https://github.com/openai/openai-node) | `4.47.3` | `4.52.2` |
| [promptable](https://github.com/promptable/Promptable.js) | `0.0.9` | `0.0.10` |
| [puppeteer](https://github.com/puppeteer/puppeteer) | `22.10.0` | `22.12.1` |
| [rate-limiter-flexible](https://github.com/animir/node-rate-limiter-flexible) | `2.4.2` | `5.0.3` |
| [resend](https://github.com/resendlabs/resend-node) | `3.2.0` | `3.4.0` |
| [stripe](https://github.com/stripe/stripe-node) | `12.18.0` | `16.1.0` |
| [unstructured-client](https://github.com/Unstructured-IO/unstructured-js-client) | `0.9.4` | `0.11.3` |
| [uuid](https://github.com/uuidjs/uuid) | `9.0.1` | `10.0.0` |
| [zod-to-json-schema](https://github.com/StefanTerdell/zod-to-json-schema) | `3.23.0` | `3.23.1` |


Updates `@anthropic-ai/sdk` from 0.20.9 to 0.24.3
- [Release notes](https://github.com/anthropics/anthropic-sdk-typescript/releases)
- [Changelog](https://github.com/anthropics/anthropic-sdk-typescript/blob/main/CHANGELOG.md)
- [Commits](https://github.com/anthropics/anthropic-sdk-typescript/compare/sdk-v0.20.9...sdk-v0.24.3)

Updates `@bull-board/api` from 5.19.2 to 5.20.5
- [Release notes](https://github.com/felixmosh/bull-board/releases)
- [Changelog](https://github.com/felixmosh/bull-board/blob/master/CHANGELOG.md)
- [Commits](https://github.com/felixmosh/bull-board/commits/v5.20.5/packages/api)

Updates `@bull-board/express` from 5.19.2 to 5.20.5
- [Release notes](https://github.com/felixmosh/bull-board/releases)
- [Changelog](https://github.com/felixmosh/bull-board/blob/master/CHANGELOG.md)
- [Commits](https://github.com/felixmosh/bull-board/commits/v5.20.5/packages/express)

Updates `@hyperdx/node-opentelemetry` from 0.7.0 to 0.8.0
- [Release notes](https://github.com/hyperdxio/hyperdx-js/releases)
- [Commits](https://github.com/hyperdxio/hyperdx-js/compare/@hyperdx/node-opentelemetry@0.7.0...@hyperdx/node-opentelemetry@0.8.0)

Updates `@nangohq/node` from 0.36.101 to 0.40.8
- [Release notes](https://github.com/NangoHQ/nango/releases)
- [Changelog](https://github.com/NangoHQ/nango/blob/master/CHANGELOG.md)
- [Commits](https://github.com/NangoHQ/nango/commits/v0.40.8/packages/node-client)

Updates `@sentry/node` from 7.116.0 to 8.13.0
- [Release notes](https://github.com/getsentry/sentry-javascript/releases)
- [Changelog](https://github.com/getsentry/sentry-javascript/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-javascript/compare/7.116.0...8.13.0)

Updates `@supabase/supabase-js` from 2.43.4 to 2.44.2
- [Release notes](https://github.com/supabase/supabase-js/releases)
- [Changelog](https://github.com/supabase/supabase-js/blob/master/RELEASE.md)
- [Commits](https://github.com/supabase/supabase-js/compare/v2.43.4...v2.44.2)

Updates `ajv` from 8.15.0 to 8.16.0
- [Release notes](https://github.com/ajv-validator/ajv/releases)
- [Commits](https://github.com/ajv-validator/ajv/compare/v8.15.0...v8.16.0)

Updates `async-mutex` from 0.4.1 to 0.5.0
- [Changelog](https://github.com/DirtyHairy/async-mutex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/DirtyHairy/async-mutex/compare/v0.4.1...v0.5.0)

Updates `bull` from 4.12.9 to 4.15.0
- [Release notes](https://github.com/OptimalBits/bull/releases)
- [Changelog](https://github.com/OptimalBits/bull/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/OptimalBits/bull/compare/v4.12.9...v4.15.0)

Updates `date-fns` from 2.30.0 to 3.6.0
- [Release notes](https://github.com/date-fns/date-fns/releases)
- [Changelog](https://github.com/date-fns/date-fns/blob/main/CHANGELOG.md)
- [Commits](https://github.com/date-fns/date-fns/compare/v2.30.0...v3.6.0)

Updates `express-rate-limit` from 6.11.2 to 7.3.1
- [Release notes](https://github.com/express-rate-limit/express-rate-limit/releases)
- [Commits](https://github.com/express-rate-limit/express-rate-limit/compare/v6.11.2...v7.3.1)

Updates `glob` from 10.4.1 to 10.4.2
- [Changelog](https://github.com/isaacs/node-glob/blob/main/changelog.md)
- [Commits](https://github.com/isaacs/node-glob/compare/v10.4.1...v10.4.2)

Updates `json-schema-to-zod` from 2.1.0 to 2.3.0
- [Commits](https://github.com/StefanTerdell/json-schema-to-zod/commits)

Updates `keyword-extractor` from 0.0.25 to 0.0.28
- [Release notes](https://github.com/michaeldelorenzo/keyword-extractor/releases)
- [Commits](https://github.com/michaeldelorenzo/keyword-extractor/compare/0.0.25...0.0.28)

Updates `langchain` from 0.1.37 to 0.2.8
- [Release notes](https://github.com/langchain-ai/langchainjs/releases)
- [Changelog](https://github.com/langchain-ai/langchainjs/blob/main/release_workspace.js)
- [Commits](https://github.com/langchain-ai/langchainjs/compare/0.1.37...0.2.8)

Updates `logsnag` from 0.1.8 to 1.0.0
- [Commits](https://github.com/LogSnag/logsnag.js/compare/v0.1.8...v1.0.0)

Updates `mongoose` from 8.4.1 to 8.4.4
- [Release notes](https://github.com/Automattic/mongoose/releases)
- [Changelog](https://github.com/Automattic/mongoose/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Automattic/mongoose/compare/8.4.1...8.4.4)

Updates `natural` from 6.12.0 to 7.0.7
- [Release notes](https://github.com/NaturalNode/natural/releases)
- [Commits](https://github.com/NaturalNode/natural/compare/v6.12.0...v7.0.7)

Updates `openai` from 4.47.3 to 4.52.2
- [Release notes](https://github.com/openai/openai-node/releases)
- [Changelog](https://github.com/openai/openai-node/blob/master/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-node/compare/v4.47.3...v4.52.2)

Updates `promptable` from 0.0.9 to 0.0.10
- [Commits](https://github.com/promptable/Promptable.js/commits)

Updates `puppeteer` from 22.10.0 to 22.12.1
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/release-please-config.json)
- [Commits](https://github.com/puppeteer/puppeteer/compare/puppeteer-v22.10.0...puppeteer-v22.12.1)

Updates `rate-limiter-flexible` from 2.4.2 to 5.0.3
- [Release notes](https://github.com/animir/node-rate-limiter-flexible/releases)
- [Commits](https://github.com/animir/node-rate-limiter-flexible/commits/v5.0.3)

Updates `resend` from 3.2.0 to 3.4.0
- [Release notes](https://github.com/resendlabs/resend-node/releases)
- [Commits](https://github.com/resendlabs/resend-node/compare/v3.2.0...v3.4.0)

Updates `stripe` from 12.18.0 to 16.1.0
- [Release notes](https://github.com/stripe/stripe-node/releases)
- [Changelog](https://github.com/stripe/stripe-node/blob/master/CHANGELOG.md)
- [Commits](https://github.com/stripe/stripe-node/compare/v12.18.0...v16.1.0)

Updates `unstructured-client` from 0.9.4 to 0.11.3
- [Release notes](https://github.com/Unstructured-IO/unstructured-js-client/releases)
- [Changelog](https://github.com/Unstructured-IO/unstructured-js-client/blob/main/RELEASES.md)
- [Commits](https://github.com/Unstructured-IO/unstructured-js-client/compare/v0.9.4...v0.11.3)

Updates `uuid` from 9.0.1 to 10.0.0
- [Changelog](https://github.com/uuidjs/uuid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/uuidjs/uuid/compare/v9.0.1...v10.0.0)

Updates `zod-to-json-schema` from 3.23.0 to 3.23.1
- [Release notes](https://github.com/StefanTerdell/zod-to-json-schema/releases)
- [Changelog](https://github.com/StefanTerdell/zod-to-json-schema/blob/master/changelog.md)
- [Commits](https://github.com/StefanTerdell/zod-to-json-schema/commits)

---
updated-dependencies:
- dependency-name: "@anthropic-ai/sdk"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: "@bull-board/api"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: "@bull-board/express"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: "@hyperdx/node-opentelemetry"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: "@nangohq/node"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: "@sentry/node"
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: "@supabase/supabase-js"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: ajv
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: async-mutex
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: bull
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: date-fns
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: express-rate-limit
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: glob
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
- dependency-name: json-schema-to-zod
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: keyword-extractor
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
- dependency-name: langchain
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: logsnag
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: mongoose
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
- dependency-name: natural
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: openai
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: promptable
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
- dependency-name: puppeteer
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: rate-limiter-flexible
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: resend
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: stripe
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: unstructured-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod-deps
- dependency-name: zod-to-json-schema
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-07-02 12:52:43 +00:00
rafaelsideguide 9c539e9113 Fixed includeHTML to use cleanedHtml as response 2024-06-18 16:26:54 -03:00
rafaelsideguide c54e797eb1 (╯°□°)╯︵ ┻━┻ 2024-06-18 11:51:28 -03:00
rafaelsideguide b2bd562bb2 transcribed from e2e to unit tests for many cases 2024-06-17 17:09:44 -03:00
rafaelsideguide ad7795f973 Merge remote-tracking branch 'origin/main' into test/load-testing 2024-06-14 15:14:01 -03:00
Nicolas 5683bb2cc8 Nick: 2024-06-05 13:20:26 -07:00
rafaelsideguide 6920ec8a61 bugfixing. already on main 2024-06-04 11:05:50 -03:00
rafaelsideguide 04a0bef0fb Merge branch 'main' into test/load-testing 2024-05-22 11:26:19 -03:00
rafaelsideguide 75f4e34d8e Merge branch 'main' into test/load-testing 2024-05-21 10:28:02 -03:00
Nicolas 9e61d431f0 Nick: hyper dx integration init 2024-05-20 13:36:34 -07:00
rafaelsideguide d667e1417b added fly staging load test 
- being rate limited. Need to add the token to the rate-limit functions
 2024-05-17 19:09:19 -03:00
Nicolas 9d635cb2a3 Nick: docx support 2024-05-16 11:48:02 -07:00
Eric Ciarla caf3f9eede Add Posthog Logging 2024-05-02 15:30:22 -04:00
Caleb Peffer 3ca9e5153f Caleb: trying to get loggin workng 2024-04-30 09:20:15 -07:00
Caleb Peffer 4f7737c922 Caleb: added ajv json schema validation. 2024-04-29 12:12:55 -07:00
Caleb Peffer 6ee1f2d3bc Caleb: initially pulled inspiration code from https://github.com/mishushakov/llm-scraper 2024-04-28 13:59:35 -07:00
rafaelsideguide 942ac3b41c Resolved merge conflicts between feat/added-anthropic-vision-api and main 2024-04-24 09:57:45 -03:00
Nicolas 306cfe4ce1 Nick: 2024-04-23 11:15:11 -07:00
Nicolas 001bf0c504 Update package.json 2024-04-21 12:05:12 -07:00
Nicolas 52620bab16 Nick: prod and local-no-auth tests 2024-04-21 11:39:36 -07:00
rafaelsideguide ddb3b25171 adding ci-cd workflow 2024-04-18 16:28:01 -03:00
rafaelsideguide 57e5b36014 [Feat] Adding pdf parser 2024-04-18 11:43:57 -03:00
Nicolas 08ed68ff55 Nick: fixes 2024-04-17 12:44:23 -07:00
rafaelsideguide 00941d94a4 Added anthropic vision to getImageDescription function 2024-04-16 18:03:48 -03:00
Nicolas a6c2a87811 Initial commit 2024-04-15 17:01:47 -04:00