firecrawl

Commit Graph

Select branches

Hide Pull Requests

194-fixing-2

194-sdk-ci-pipeline-for-publishing-pythonnode-sdk

ETL-unicorn-example-app

added-issue-templates

bug/crawl-limit

bug/data-check-in-python-sdk

bug/dealing-with-dns-error

bug/fix-clean-jobs

bug/fix-custom-scrape-pdf-google-drive

bug/fix-issue-310

bug/fixing-cicd

bug/issue-258

bugfix/crawl-docker

bugfix/empty-excludes

bugfix/empty-tags

bugfix/fire-engine-content

bugfix/issue-291

bugfix/issue-435

bugfix/issue-466

bugfix/issue-477

bugfix/lowercase-all-urls

bugfix/malformed-url-sitemap

bugfix/normalized-api-on-crawl-status

bugfix/partial-data-js-sdk

bugfix/scrape-test-failing

bugfix/switch-case-ratelimiter

bugfix/update-env-example

calebpeffer-github-seo

cjp/contributors-guide-and

cjp/email-to-posthog-logging

cjp/image-added-to-extract

cjp/linksOnPage

cjp/switch-license-to-AGPL

dependabot/npm_and_yarn/apps/api/dev-deps-a07c4dd686

dependabot/npm_and_yarn/apps/api/prod-deps-cb57818d7f

detect-pdfs

doc/crawl-limit-default

docs/update-docs

feat/actions

feat/added-anthropic-vision-api

feat/allowbackwardcrawling-option

feat/allowed-urls

feat/beta-customers

feat/better-gdrive-pdf-fetch

feat/better-types-for-jssdk

feat/blocklist-social-media

feat/cancel-fire-engine

feat/check-redirect

feat/ci-cd

feat/ci-cd-env-secrets

feat/ci-cd-workflow

feat/concurrency-limit

feat/coupons

feat/custom-scraping-readme

feat/drizzle

feat/fire-engine-chrome-cdp

feat/fire-engine-fallback-for-sitemap

feat/fullpage-screenshot

feat/gdrive-pdfs

feat/go-html-to-md-parser

feat/go-sdk-submodule

feat/gpt-4o

feat/idempotency-key

feat/improv-self-host

feat/improving-reative-paths

feat/issue-205

feat/issue-264

feat/issue-266

feat/issue-273

feat/issue-342

feat/issue-413

feat/js-sdk-v0011

feat/key-auth

feat/logger

feat/max-depth

feat/maxDepthRelative

feat/parse-to-markdown-tables

feat/pdf-parser

feat/posthog-logging

feat/python-sdk-502

feat/queue-scrapes

feat/rate-limit-singleton

feat/rate-limits

feat/rawHtmlExtraction

feat/replace-all-paths-to-absolute-paths

feat/rpc-rework

feat/save-docs-on-supabase

feat/scrape-monitoring

feat/screenshot-support

feat/scroll-xpaths

feat/server-health-check-slack-message

feat/test-suite

feat/to-markdown

feat/type-improvements

feat/yoastseo-generated-sitemaps

fix--rawHtml-return-on-crawl

fix-for-partial-docs-timeout

fix/e2e-tests

fix/fly-deploy-go-sdk

fix/issue-644

fix/js-sdk-full-page-screenshot

fix/rename-variables

fix/screenshot-fullpage

fix/turndown-cpu-load-big-tables

fix/v1-maxDepth

go-parser-singleton

greenpay-fixes

improv/sdk-attempts

js-sdk-types-stuff

llm-extraction

llm-extraction-search

main

mog/fix-infinite-regex

mog/fix-job-stuck-2

mog/js-sdk-cjs

mog/js-sdk-tests-fix

mog/websocket-sdk

more-logging

new-read-me

new_examples

nit/change-firecrawl-email

nsc/Nick-set-the-crawl-limit-to-the-remaining-credits

nsc/abs-path-fix

nsc/added-support-for-ficrawl-npm

nsc/added-support-for-firecrawl-pip

nsc/admin-router

nsc/allowed-keywords-in-blocklist

nsc/amazon

nsc/bill-team-async

nsc/cacheable-lookup

nsc/cancel-job

nsc/check-credits-optimization

nsc/check-team-credits-limit

nsc/chrome-cdp-for-params

nsc/cjs-support

nsc/clean-content

nsc/clusters

nsc/crawler-fixes

nsc/custom-vanta-refactor

nsc/data-del

nsc/digikey-tls

nsc/docx-support

nsc/exploration-play

nsc/fire-engine-beta

nsc/fire-engine-first-layer

nsc/fix-deadlocks

nsc/fix-headers

nsc/fix-json-error-handler

nsc/fix-scaling-fix-email

nsc/fix-tables

nsc/free-credits-increase

nsc/fwd-header-auth

nsc/google-drive-pdf

nsc/hyper-dx-integration

nsc/hyper-v81

nsc/improved-billing-notifications

nsc/improved-blocklist

nsc/improvemnts-fixes-misc

nsc/includeOnlyTags

nsc/initial-web-refac

nsc/job-logs

nsc/job-priority

nsc/llm-extraction-zod-integration

nsc/load-testing-nick

nsc/logging-scrapers

nsc/manual-rate-limit

nsc/map-pagination

nsc/max-retries-fire-engine

nsc/mvp-search

nsc/new-pricing

nsc/notion-fix

nsc/pageoptions-crawler

nsc/parallelize-fetching-sitemaps

nsc/partial-data-changes

nsc/profiling-full-scrape

nsc/python-sdk-waiting-fixes

nsc/rate-limit-fixes

nsc/rate-limiter-tests

nsc/redlock-cache-auth

nsc/refactor-scraping-order

nsc/removal-of-tags-in-include-main-content

nsc/returnOnlyUrls

nsc/rpc-cloudflare-issues

nsc/search-fix-version-v0

nsc/search-js-sdk

nsc/search-playground

nsc/sentry

nsc/seperate-rate-limit

nsc/serper-params

nsc/sitemap-fix-fire-engine

nsc/slack-alerts-1

nsc/small-fe-print

nsc/small-handle-for-client-side-errors

nsc/smart-crawl-init-testing

nsc/speed-improvements

nsc/speed-up-crawl-4x

nsc/timeout-scrape

nsc/transactional-emails

nsc/usage-based-overuse

nsc/wait-for-param

nsc/webhook-self-host-fix

nsc/website-param-fixes

o1-crawler

o1_crawler_actions

rafa/investigating-crawl-bugs

refactor-index

revert-246-194-sdk-ci-pipeline-for-publishing-pythonnode-sdk

test-sdks

test/crawl-options

test/load-testing

tests/crawl-limit-unit-tests

tests/e2e-to-unit

tests/metadata

tests/suite-v1

uiTemplate

v1-webscraper

v1/llm-extract

v1/mockup-controllers

v1/node-sdk

v1/python-sdk

v1/webhooks

website-specific-params

#10

#109

#11

#110

#118

#119

#12

#120

#126

#129

#13

#130

#131

#132

#134

#135

#137

#139

#14

#140

#142

#143

#144

#145

#148

#149

#151

#153

#158

#16

#160

#166

#167

#168

#17

#170

#174

#175

#176

#178

#180

#181

#183

#185

#186

#190

#191

#196

#20

#200

#204

#207

#21

#211

#212

#213

#216

#218

#22

#221

#223

#224

#228

#229

#232

#234

#235

#236

#237

#238

#239

#24

#24

#242

#244

#246

#248

#25

#250

#251

#252

#253

#255

#256

#260

#262

#263

#265

#268

#269

#27

#271

#272

#274

#275

#276

#277

#278

#278

#28

#280

#280

#282

#283

#285

#29

#290

#294

#296

#297

#298

#30

#301

#302

#303

#307

#308

#31

#312

#313

#316

#317

#318

#32

#321

#322

#323

#325

#326

#328

#329

#331

#332

#336

#337

#338

#34

#343

#343

#344

#344

#346

#347

#348

#349

#35

#350

#355

#355

#357

#36

#361

#362

#363

#37

#370

#371

#373

#374

#375

#378

#379

#38

#380

#381

#382

#386

#389

#389

#391

#393

#394

#395

#396

#397

#398

#399

#4

#40

#403

#404

#405

#406

#410

#414

#417

#42

#420

#423

#424

#427

#429

#432

#433

#436

#438

#443

#444

#445

#447

#449

#450

#453

#454

#455

#457

#458

#459

#46

#460

#461

#469

#470

#471

#472

#473

#475

#478

#479

#480

#482

#483

#484

#485

#491

#492

#493

#493

#494

#496

#497

#498

#499

#5

#500

#501

#502

#503

#504

#505

#506

#508

#512

#515

#516

#52

#520

#521

#522

#523

#524

#527

#529

#53

#531

#532

#534

#535

#535

#536

#537

#541

#541

#542

#542

#549

#55

#550

#551

#552

#553

#554

#555

#559

#56

#560

#561

#562

#564

#565

#566

#571

#572

#573

#574

#575

#576

#576

#579

#580

#582

#584

#584

#586

#590

#591

#593

#594

#596

#601

#601

#603

#604

#605

#607

#608

#61

#611

#612

#613

#613

#614

#615

#615

#619

#62

#620

#623

#624

#626

#627

#628

#628

#629

#63

#631

#631

#634

#634

#635

#637

#638

#639

#64

#640

#643

#645

#646

#648

#649

#65

#653

#654

#655

#66

#664

#669

#67

#670

#671

#672

#673

#674

#676

#677

#678

#678

#679

#68

#682

#685

#685

#689

#690

#691

#692

#693

#693

#694

#694

#695

#704

#706

#707

#72

#75

#77

#8

#83

#84

#9

#90

v1.0.0

3621e191bd feat(concurrency-limit): set limit based on plan feat/concurrency-limit Gergő Móricz 2024-09-28 00:19:46 +0200
7018b469e9

Merge 26825fc8fb into c6a83ab92c skeptrune 2024-09-27 16:56:51 -0400
c6a83ab92c fix(api): entrypoint main Gergő Móricz 2024-09-27 22:16:27 +0200
e44bdf7a54 bad dockerfile Gergő Móricz 2024-09-27 21:07:11 +0200
f0a1a2e45b fix: increase ulimit -n in docker Gergő Móricz 2024-09-27 20:44:52 +0200
d5e2a80e4a fix(crawl-status): keep 10 megabyte pages if they're the only thing in the output Gergő Móricz 2024-09-27 20:41:41 +0200
975f0575b4 Nick: max retries with axios-retry Nicolas 2024-09-27 12:58:57 -0400
92961cf74f Merge branch 'main' of https://github.com/mendableai/firecrawl Nicolas 2024-09-27 12:23:45 -0400
1fdff87b3e Update single_url.ts Nicolas 2024-09-27 12:23:44 -0400
6283e8fc47 fix(logger): set default level to trace Gergő Móricz 2024-09-27 17:46:43 +0200
5e8ef4954e feat(auth): log cache key in acuc update error Gergő Móricz 2024-09-27 17:13:10 +0200
0bf709485a

Merge d0c2133946 into e98f858eb6 Rafael Miller 2024-09-27 01:42:57 +0200
e98f858eb6 fix(api): playground scrape errors Gergő Móricz 2024-09-26 22:28:14 +0200
8d44cb33bb Nick: fixed error message Nicolas 2024-09-26 16:07:15 -0400
2cb493321a fix(ACUC): do not refresh cache every set Gergő Móricz 2024-09-26 22:06:50 +0200
9bdd344b36 fix(redlock): use redlock.using for stability Gergő Móricz 2024-09-25 22:47:42 +0200
250c3bb5c6 fix(auth): move redlock settings Gergő Móricz 2024-09-25 22:27:51 +0200
81245e68fa fix(auth/redlock): retry cached ACUC lock for 20 seconds Gergő Móricz 2024-09-25 22:25:13 +0200
0f89f5e7cb fix(billTeam): cache update race condition Gergő Móricz 2024-09-25 22:15:02 +0200
d13a97f979 fix(credit_billing): allow spending of exact credits Gergő Móricz 2024-09-25 21:44:05 +0200
84bff8add8 fix(billTeam): update cached ACUC after billing Gergő Móricz 2024-09-25 21:37:01 +0200
f22ab5ffaf feat(db): implement bill_team RPC Gergő Móricz 2024-09-25 20:57:45 +0200
c1f68c3e0a fix(credit_billing): return chunk.remaining_credits Gergő Móricz 2024-09-25 20:37:35 +0200
2073063fb7 fix(db): fix caching and rpc error Gergő Móricz 2024-09-25 19:42:45 +0200
f8c70fe5dd feat(db): implement auth_credit_usage_chunk RPC Gergő Móricz 2024-09-25 19:25:18 +0200
20a4fa53fa Merge branch 'feat/rpc-rework' of https://github.com/mendableai/firecrawl into feat/rpc-rework feat/rpc-rework Nicolas 2024-09-26 16:07:19 -0400
f4204cf68d Nick: fixed error message Nicolas 2024-09-26 16:07:15 -0400
854e203d28 fix(ACUC): do not refresh cache every set Gergő Móricz 2024-09-26 22:06:50 +0200
29815e084b feat(v1/Document): add warning field Gergő Móricz 2024-09-26 21:19:05 +0200
095babe70b fix(queue-jobs): jobs with concurrency fails may vanish Gergő Móricz 2024-09-26 21:18:56 +0200
b696bfc854 fix(crawl-status): avoid race conditions where crawl may be deemed failed Gergő Móricz 2024-09-26 21:00:27 +0200
75e32b0e5c

Merge pull request #707 from mendableai/new_examples Eric Ciarla 2024-09-26 14:51:45 -0400
20b998e66a Delete o1_job_recommender.ipynb new_examples Eric Ciarla 2024-09-26 14:51:07 -0400
5c4d436f1e Create o1_job_recommender.py Eric Ciarla 2024-09-26 14:46:48 -0400
dec4171937 fix(queue-worker, queue-jobs): logic fixes Gergő Móricz 2024-09-26 20:39:19 +0200
d2881927c1 fix(queue-worker): remove concurrency entries when done in sentry-less branch Gergő Móricz 2024-09-26 20:29:17 +0200
53fce67ca1 feat(queue-worker): PoC of concurrency limits Gergő Móricz 2024-09-26 20:23:13 +0200
30058b1da0 Nick: increased timeout for chrome-cdp due to smart wait Nicolas 2024-09-25 19:27:02 -0400
51bc2f25fe remove actions crawler Eric Ciarla 2024-09-26 11:44:55 -0400
d7e6ec3c98 commit_actions o1_crawler_actions Eric Ciarla 2024-09-26 11:42:03 -0400
10faa2588d

Merge 503e83e83e into a9773a24a3 SebastjanPrachovskij 2024-09-26 15:52:13 +0300
289af6f89e example Eric Ciarla 2024-09-25 21:10:09 -0400
4298cb6cc4 Nick: added max retries for fire-engine nsc/max-retries-fire-engine Nicolas 2024-09-25 20:58:26 -0400
a9773a24a3 Nick: increased timeout for chrome-cdp due to smart wait Nicolas 2024-09-25 19:27:02 -0400
953d4fb197 fix(redlock): use redlock.using for stability Gergő Móricz 2024-09-25 22:47:42 +0200
eef116bef8 fix(auth): move redlock settings Gergő Móricz 2024-09-25 22:27:51 +0200
2c96d2eef6 fix(auth/redlock): retry cached ACUC lock for 20 seconds Gergő Móricz 2024-09-25 22:25:13 +0200
1cca9b8ae6 fix(billTeam): cache update race condition Gergő Móricz 2024-09-25 22:15:02 +0200
eb7317c08a fix(credit_billing): allow spending of exact credits Gergő Móricz 2024-09-25 21:44:05 +0200
e67cbc2ca1 fix(billTeam): update cached ACUC after billing Gergő Móricz 2024-09-25 21:37:01 +0200
5a8eb17a82 feat(db): implement bill_team RPC Gergő Móricz 2024-09-25 20:57:45 +0200
415fd9f333 fix(credit_billing): return chunk.remaining_credits Gergő Móricz 2024-09-25 20:37:35 +0200
417adf8e96 fix(db): fix caching and rpc error Gergő Móricz 2024-09-25 19:42:45 +0200
331e826bca feat(db): implement auth_credit_usage_chunk RPC Gergő Móricz 2024-09-25 19:25:18 +0200
abdc08edea

Merge pull request #679 from h4r5h4/fix/folder-name Eric Ciarla 2024-09-25 10:40:09 -0400
1da026b26e Update single_url.ts Nicolas 2024-09-24 23:29:48 -0400
b8266cc329 Update website_params.ts Nicolas 2024-09-24 23:28:58 -0400
f30aeaf362

Merge 9c475d63b7 into f00c0b82f9 Rafael Miller 2024-09-25 02:11:22 +0200
f00c0b82f9 fix(v1/scrape): add total wait specified in request to timeout Gergő Móricz 2024-09-24 21:56:22 +0200
3f138e559e Update website_params.ts Nicolas 2024-09-24 15:14:26 -0400
43730b5db6 feat(WebScraper): always report error of last scraper in order Gergő Móricz 2024-09-24 20:03:49 +0200
3e661a2087 fix(v1/crawl-cancel): avoid double authing Gergő Móricz 2024-09-24 18:28:46 +0200
86744f6deb

Update README.md Nicolas 2024-09-24 13:22:09 -0400
4194525640 fix(blocklist): unblock TikTok Business page Gergő Móricz 2024-09-24 16:55:19 +0200
3f61377db1

Merge cc6461383a into 4a623c084a dependabot[bot] 2024-09-24 08:50:40 +0000
db0f60b75d

Merge 742aac4ee1 into 4a623c084a dependabot[bot] 2024-09-24 08:50:40 +0000
4a623c084a fix(fly): don't use Depot builders (doesn't work) Gergő Móricz 2024-09-24 10:50:30 +0200
a59b5836d5 Revert error tallying Gergő Móricz 2024-09-24 10:27:49 +0200
a4b128e8b7 fix(rust): blocklisted error test Gergő Móricz 2024-09-23 23:03:00 +0200
483f97d21b fix(v0/search): don't sent scrape fail errors to Sentry Gergő Móricz 2024-09-23 18:49:27 +0200
d927cafeea fix(queue-worker): don't send scraping errors to sentry Gergő Móricz 2024-09-23 18:48:01 +0200
677faa27f3 fix(WebScraper): explicitly ignore 404s Gergő Móricz 2024-09-23 18:47:07 +0200
83d8287c14 fix(v0, sentry): don't send all scraping methods failed errors to Sentry Gergő Móricz 2024-09-23 18:40:21 +0200
d2f7031069 fix(WebScraper): fatal error handler triggering for 404s Gergő Móricz 2024-09-23 18:33:10 +0200
1ad97f199e

apps/playwright-service(deps): bump the prod-deps group dependabot[bot] 2024-09-23 06:57:11 +0000
742aac4ee1

apps/api(deps): bump the prod-deps group across 1 directory with 35 updates dependabot/npm_and_yarn/apps/api/prod-deps-cb57818d7f dependabot[bot] 2024-09-23 06:54:27 +0000
cc6461383a

apps/api(deps-dev): bump the dev-deps group across 1 directory with 12 updates dependabot/npm_and_yarn/apps/api/dev-deps-a07c4dd686 dependabot[bot] 2024-09-23 06:49:57 +0000
c701b30a88

apps/test-suite(deps-dev): bump the dev-deps group dependabot[bot] 2024-09-23 06:19:44 +0000
56c7085973

apps/test-suite(deps): bump the prod-deps group dependabot[bot] 2024-09-23 06:18:52 +0000
4721aa1687

Merge pull request #690 from mendableai/nsc/search-fix-version-v0 Nicolas 2024-09-21 21:12:01 -0400
848a2b364a Update package.json nsc/search-fix-version-v0 Nicolas 2024-09-21 21:11:23 -0400
dfdbae74c6 Update fireEngine.ts Nicolas 2024-09-21 21:10:05 -0400
fbb5f23016 Update index.ts Nicolas 2024-09-21 20:53:33 -0400
ae55a0b51c

Merge c171baf963 into 607e46267c Nicolas 2024-09-22 02:48:47 +0530
607e46267c Update package.json Nicolas 2024-09-20 19:46:17 -0400
db161ac55a Nick: press + write Nicolas 2024-09-20 19:45:23 -0400
380dcc2fd6

Merge pull request #682 from mendableai/feat/actions Nicolas 2024-09-20 18:43:19 -0400
3fc5ce17d2 Nick: fixed error handling for v0 scrape feat/actions Nicolas 2024-09-20 18:35:30 -0400
0690cfeaad Merge branch 'main' into feat/actions Nicolas 2024-09-20 18:24:13 -0400
95e4c8920b fix(sdk/rust): license Gergő Móricz 2024-09-20 21:55:05 +0200
e1a34b0a99 Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot" Gergő Móricz 2024-09-20 21:43:22 +0200
815bfc8f07 feat(scrape): scroll down/up with actions if fullpagescreenshot Gergő Móricz 2024-09-20 21:42:09 +0200
d663bbf0ca feat(actions): add scroll Gergő Móricz 2024-09-20 21:41:53 +0200
3dd912ec91 feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor Gergő Móricz 2024-09-20 21:02:53 +0200
719dfbccbb Update docs Gergő Móricz 2024-09-20 20:30:46 +0200
939040bf44 Update docs and example Gergő Móricz 2024-09-20 20:08:33 +0200
3ec0bbe28d feat(sdk/rust/crawl): paginate through results Gergő Móricz 2024-09-20 19:40:32 +0200
a078cdbd9d Rust SDK 1.0.0 Gergő Móricz 2024-09-20 19:36:07 +0200
93a20442e3 feat(sdk/rust): first batch of changes for 1.0.0 Gergő Móricz 2024-09-19 22:22:57 +0200
f915ce3177 Update docs and example Gergő Móricz 2024-09-20 20:08:33 +0200