chore(release): 0.3.3

chore: resolves test issues
feat: Shows total keywords count in domains page
2025-06-26 18:15:54 +00:00 · 2023-11-12 20:59:08 +06:00 · 2023-11-12 20:58:03 +06:00 · 2023-11-11 12:48:31 +06:00 · 2023-11-11 11:42:14 +06:00 · 2023-11-11 11:29:59 +06:00
68 changed files with 5274 additions and 12196 deletions
--- a/.eslintrc.json
+++ b/.eslintrc.json
@@ -12,6 +12,7 @@
   "no-await-in-loop": "off",
   "arrow-body-style":"off",
   "max-len": ["error", {"code": 150, "ignoreComments": true, "ignoreUrls": true}],
+   "import/no-extraneous-dependencies": "off",
   "import/extensions": [
      "error",
      "ignorePackages",
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,114 @@

 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.

+### [0.3.3](https://github.com/towfiqi/serpbear/compare/v0.3.2...v0.3.3) (2023-11-12)
+
+
+### Features
+
+* Adds ability to visit pages from Insight tab ([60c68bd](https://github.com/towfiqi/serpbear/commit/60c68bd339db7aeed35aea035dd21691702ffee3))
+* Domains now show their favicon. ([2339e31](https://github.com/towfiqi/serpbear/commit/2339e31af9e90bf918f5bcd4f23114f38cef0313)), closes [#130](https://github.com/towfiqi/serpbear/issues/130)
+* Shows total keywords count in domains page ([fbd23ed](https://github.com/towfiqi/serpbear/commit/fbd23ede256062c72ec2f7e3983a0a02f0240725))
+
+
+### Bug Fixes
+
+* Resolves Website Thumbnail missing issue ([4a60271](https://github.com/towfiqi/serpbear/commit/4a60271cac1209dc02748c4d31943bb21c9ecaf2)), closes [#131](https://github.com/towfiqi/serpbear/issues/131)
+
+### [0.3.2](https://github.com/towfiqi/serpbear/compare/v0.3.1...v0.3.2) (2023-11-09)
+
+
+### Bug Fixes
+
+* Resolves issue with adding long tld emails ([9b9b74a](https://github.com/towfiqi/serpbear/commit/9b9b74af4c249e27458d29ba052e96ab2db8b640)), closes [#127](https://github.com/towfiqi/serpbear/issues/127)
+
+### [0.3.1](https://github.com/towfiqi/serpbear/compare/v0.3.0...v0.3.1) (2023-11-04)
+
+
+### Bug Fixes
+
+* Removes dev files from docker volumes ([454454a](https://github.com/towfiqi/serpbear/commit/454454a422bab4d37a2d43ad95868e293a97b88e))
+* Updates vulnerable dependencies ([97dd0b1](https://github.com/towfiqi/serpbear/commit/97dd0b131be4cec73d07f35062334dd1881f0013))
+
+## [0.3.0](https://github.com/towfiqi/serpbear/compare/v0.2.6...v0.3.0) (2023-11-03)
+
+
+### Features
+
+* Adds ability to disable/clear retry queue for failed keywords ([dc3c7a7](https://github.com/towfiqi/serpbear/commit/dc3c7a722b18248115969c51f2495ccf1c43926d))
+* Adds ability to search w/o case sensitivity ([4748ffc](https://github.com/towfiqi/serpbear/commit/4748ffc382161c5d861b8d43e8eba466a031e2bc)), closes [#115](https://github.com/towfiqi/serpbear/issues/115)
+* Displays the Best position of the keywords ([fc183d2](https://github.com/towfiqi/serpbear/commit/fc183d246d55e0eecf43c91f6da8a59192e8e771)), closes [#89](https://github.com/towfiqi/serpbear/issues/89)
+* Refresh All feature now shows update real-time ([1d6b2be](https://github.com/towfiqi/serpbear/commit/1d6b2be95aa133b7998f5cf098f15aa32f5badd2))
+* Remembers last selected coutry ([d3d336f](https://github.com/towfiqi/serpbear/commit/d3d336fa71cc789624b10f3cdd1a2b5983053e6f)), closes [#101](https://github.com/towfiqi/serpbear/issues/101)
+
+
+### Bug Fixes
+
+*  Resolves missing keyword scrape spinner issue ([f57bca2](https://github.com/towfiqi/serpbear/commit/f57bca23daa3fe888af4c19a681dcec6b6100d83))
+* Cron stopped on failing to parse failed queue ([8a949ce](https://github.com/towfiqi/serpbear/commit/8a949ce4c078ff377e91a95c4b86ef2b15dae88b)), closes [#116](https://github.com/towfiqi/serpbear/issues/116)
+* Fixes import order error in some instances. ([be80ed7](https://github.com/towfiqi/serpbear/commit/be80ed7ef3dd0a315c5ad67d17e61a4797dc274c)), closes [#114](https://github.com/towfiqi/serpbear/issues/114)
+* Fixes issue with adding hyphenated subdomains. ([c0470cf](https://github.com/towfiqi/serpbear/commit/c0470cfa9d0dac86317c886065b461cfe82ffb16))
+* Fixes the weekly cron day issue. ([392122a](https://github.com/towfiqi/serpbear/commit/392122a7101683342830e900c6f0c39f9272bb34)), closes [#118](https://github.com/towfiqi/serpbear/issues/118)
+* Fxies special character keyword scrape issue. ([9feff13](https://github.com/towfiqi/serpbear/commit/9feff13f18a4d72203dde694a147831f990b37fb)), closes [#113](https://github.com/towfiqi/serpbear/issues/113) [#122](https://github.com/towfiqi/serpbear/issues/122)
+
+### [0.2.6](https://github.com/towfiqi/serpbear/compare/v0.2.5...v0.2.6) (2023-03-29)
+
+
+### Features
+
+* Add option to Delay Between scrapes. ([0a83924](https://github.com/towfiqi/serpbear/commit/0a83924ffe2243c52849c167c6c15d9688ff1dc7)), closes [#87](https://github.com/towfiqi/serpbear/issues/87)
+* Integrates Space Serp. ([0538a8c](https://github.com/towfiqi/serpbear/commit/0538a8c01601d2f6365848580591a248528e67c7))
+
+
+### Bug Fixes
+
+* **components:** fix typo "Goolge" -> "Google" ([dce7c41](https://github.com/towfiqi/serpbear/commit/dce7c412e813fc845973f36ad1c9fa91df4a6611))
+* Fixes first Keryword Error cut off issue. ([d950515](https://github.com/towfiqi/serpbear/commit/d9505158c439a924a1c86eb8243faf2a15bed43e))
+* Fixes lags when tracking thousands of keywords ([9757fde](https://github.com/towfiqi/serpbear/commit/9757fde02ec83405546733381104c54ed6510681)), closes [#88](https://github.com/towfiqi/serpbear/issues/88)
+
+### [0.2.5](https://github.com/towfiqi/serpbear/compare/v0.2.4...v0.2.5) (2023-03-07)
+
+
+### Features
+
+* Adds current App version Number in Footer. ([b83df5f](https://github.com/towfiqi/serpbear/commit/b83df5f3dbd64db657d31f0526438e7165e1b475))
+* Adds Keyword Scraping Interval Settings. ([3b6d034](https://github.com/towfiqi/serpbear/commit/3b6d034d6f7da0b4259070220fffff44184dd680)), closes [#81](https://github.com/towfiqi/serpbear/issues/81) [#76](https://github.com/towfiqi/serpbear/issues/76)
+
+
+### Bug Fixes
+
+* Fixes Broken Image thumbnail loading issue. ([5dd366b](https://github.com/towfiqi/serpbear/commit/5dd366b91e2a94e658bf5250a8a0fa64c09e1c11))
+* Settings Update Toast was not showing up. ([b9d58a7](https://github.com/towfiqi/serpbear/commit/b9d58a721df12f3f34220a3ae5da6897e23c83ec))
+
+### [0.2.4](https://github.com/towfiqi/serpbear/compare/v0.2.3...v0.2.4) (2023-02-15)
+
+
+### Features
+
+* Keyword ranking pages can now be clicked. ([c5af94a](https://github.com/towfiqi/serpbear/commit/c5af94a1469713ed4092253d26953ee0ed28c25d))
+
+
+### Bug Fixes
+
+* Fixes broken Login on windows ([c406588](https://github.com/towfiqi/serpbear/commit/c406588953035e4177a64011c13eb0e3aedffe89))
+* Fixes Node Cron memory leak issue. ([3c48d13](https://github.com/towfiqi/serpbear/commit/3c48d130b6f229a4ac27ec43ef1ea3a6640cecf6))
+
+### [0.2.3](https://github.com/towfiqi/serpbear/compare/v0.2.2...v0.2.3) (2023-01-12)
+
+
+### Features
+
+* Ability to tag multiple keywords at once ([9e9dad7](https://github.com/towfiqi/serpbear/commit/9e9dad7631691b2a836fdd4c522b1f933b17e285)), closes [#54](https://github.com/towfiqi/serpbear/issues/54)
+* Set USERNAME as well as USER variable ([b50733d](https://github.com/towfiqi/serpbear/commit/b50733defc2c06e0f92ca3e88fd1f74684eee9c0))
+
+
+### Bug Fixes
+
+* Fixes Position and View Sort. ([8139e39](https://github.com/towfiqi/serpbear/commit/8139e399c13ab8be767facef9a19c67dec06ed64)), closes [#46](https://github.com/towfiqi/serpbear/issues/46)
+* Fixes wrong CTR value for Search Console Data ([cb24696](https://github.com/towfiqi/serpbear/commit/cb24696a1f47b02a11c68cd1c673ea8b1bacd144)), closes [#48](https://github.com/towfiqi/serpbear/issues/48)
+* Mobile Keyword Scraping not working. ([a1108d2](https://github.com/towfiqi/serpbear/commit/a1108d240ea38ab0886ef3722b0c937ec5a45591)), closes [#58](https://github.com/towfiqi/serpbear/issues/58)
+* ScrapingAnt Mobile Keyword Scrape not working ([acc0b39](https://github.com/towfiqi/serpbear/commit/acc0b39d80d4f9371967a0d425ed205c5d866eea))
+
 ### [0.2.2](https://github.com/towfiqi/serpbear/compare/v0.2.1...v0.2.2) (2022-12-25)


--- a/4
+++ b/4
@@ -10,6 +10,8 @@ COPY . .
 FROM node:lts-alpine AS builder
 WORKDIR /app
 COPY --from=deps /app ./
+RUN rm -rf /app/data
+RUN rm -rf /app/__tests__
 RUN npm run build


@@ -29,7 +31,7 @@ COPY --from=builder --chown=nextjs:nodejs /app/cron.js ./
 COPY --from=builder --chown=nextjs:nodejs /app/email ./email
 RUN rm package.json
 RUN npm init -y 
-RUN npm i cryptr dotenv node-cron @googleapis/searchconsole
+RUN npm i cryptr dotenv croner @googleapis/searchconsole
 RUN npm i -g concurrently

 USER nextjs
--- a/README.md
+++ b/README.md
@@ -18,7 +18,7 @@ SerpBear is an Open Source Search Engine Position Tracking App. It allows you to
 -   **Zero Cost to RUN:** Run the App on mogenius.com or Fly.io for free.

 #### How it Works
-The App uses third party website scrapers like ScrapingAnt, ScrapingRobot, SerpApi or Your given Proxy ips to scrape google search results to see if your domain appears in the search result for the given keyword. Also, When you connect your Googel Search Console account, the app shows actual search visits for each tracked keywords. You can also discover new keywords, and find the most performing keywords, countries, pages.
+The App uses third party website scrapers like ScrapingAnt, ScrapingRobot, SearchApi, SerpApi or Your given Proxy ips to scrape google search results to see if your domain appears in the search result for the given keyword. Also, When you connect your Googel Search Console account, the app shows actual search visits for each tracked keywords. You can also discover new keywords, and find the most performing keywords, countries, pages.

 #### Getting Started
 -   **Step 1:** Deploy & Run the App.
@@ -40,6 +40,8 @@ The App uses third party website scrapers like ScrapingAnt, ScrapingRobot, SerpA
 | whatsmyserp.com | $49/mo| 30,000/mo| No |
 | serply.io | $49/mo | 5000/mo | Yes |
 | serpapi.com | From $50/mo** | From 5,000/mo** | Yes |
+| spaceserp.com | $59/lifetime | 15,000/mo | Yes |
+| SearchApi.io | From $40/mo | From 10,000/mo | Yes |

 (*) Free upto a limit. If you are using ScrapingAnt you can lookup 10,000 times per month for free.
 (**) Free up to 100 per month. Paid from 5,000 to 10,000,000+ per month.
--- a/mocks/data.ts
+++ b/mocks/data.ts
@@ -2,10 +2,11 @@ export const dummyDomain = {
   ID: 1,
   domain: 'compressimage.io',
   slug: 'compressimage-io',
-   keywordCount: 0,
+   keywordCount: 10,
+   avgPosition: 24,
   lastUpdated: '2022-11-11T10:00:32.243',
   added: '2022-11-11T10:00:32.244',
-   tags: [],
+   tags: '',
   notification: true,
   notification_interval: 'daily',
   notification_emails: '',
@@ -33,7 +34,7 @@ export const dummyKeywords = [
       lastResult: [],
       sticky: false,
       updating: false,
-       lastUpdateError: 'false',
+       lastUpdateError: false as false,
   },
   {
       ID: 2,
@@ -56,6 +57,23 @@ export const dummyKeywords = [
       lastResult: [],
       sticky: false,
       updating: false,
-       lastUpdateError: 'false',
+       lastUpdateError: false as false,
   },
 ];
+
+export const dummySettings = {
+   scaping_api: '',
+   scraper_type: 'none',
+   notification_interval: 'never',
+   notification_email: '',
+   notification_email_from: '',
+   smtp_server: '',
+   smtp_port: '',
+   smtp_username: '',
+   smtp_password: '',
+   scrape_retry: false,
+   search_console_integrated: false,
+   screenshot_key: '',
+   available_scapers: [],
+   failed_queue: [],
+};
--- a/mocks/utils.tsx
+++ b/mocks/utils.tsx
--- a/test/components/Modal.test.tsx
+++ b/test/components/Modal.test.tsx
@@ -1,35 +0,0 @@
-import { render, screen } from '@testing-library/react';
-import React from 'react';
-import Modal from '../../components/common/Modal';
-
-// jest.mock('React', () => ({
-//    ...jest.requireActual('React'),
-//    useEffect: jest.fn(),
-// }));
-// const mockedUseEffect = useEffect as jest.Mock<any>;
-
-// jest.mock('../../components/common/Icon', () => () => <div data-testid="icon"/>);
-
-describe('Modal Component', () => {
-   it('Renders without crashing', async () => {
-       render(<Modal closeModal={() => console.log() }><div></div></Modal>);
-      //  mockedUseEffect.mock.calls[0]();
-       expect(document.querySelector('.modal')).toBeInTheDocument();
-   });
-//    it('Sets up the escapae key shortcut', async () => {
-//       render(<Modal closeModal={() => console.log() }><div></div></Modal>);
-//       expect(mockedUseEffect).toBeCalled();
-//   });
-   it('Displays the Given Content', async () => {
-      render(<Modal closeModal={() => console.log() }>
-        <div>
-           <h1>Hello Modal!!</h1>
-        </div>
-      </Modal>);
-      expect(await screen.findByText('Hello Modal!!')).toBeInTheDocument();
-   });
-   it('Renders Modal Title', async () => {
-      render(<Modal closeModal={() => console.log() } title="Sample Modal Title"><p>Some Modal Content</p></Modal>);
-      expect(await screen.findByText('Sample Modal Title')).toBeInTheDocument();
-   });
-});
--- a/test/components/Sidebar.test.tsx
+++ b/test/components/Sidebar.test.tsx
@@ -1,11 +0,0 @@
-import { render, screen } from '@testing-library/react';
-import Sidebar from '../../components/common/Sidebar';
-
-describe('Sidebar Component', () => {
-   it('renders without crashing', async () => {
-       render(<Sidebar domains={[]} showAddModal={() => console.log() } />);
-       expect(
-           await screen.findByText('SerpBear'),
-       ).toBeInTheDocument();
-   });
-});
--- a/test/hooks/domains.tests.tsx
+++ b/test/hooks/domains.tests.tsx
@@ -1,21 +0,0 @@
-import { waitFor } from '@testing-library/react';
-// import { useFetchDomains } from '../../services/domains';
-// import { createWrapper } from '../utils';
-
-jest.mock('next/router', () => ({
-   useRouter: () => ({
-     query: { slug: 'compressimage-io' },
-     push: (link:string) => { console.log('Pushed', link); },
-   }),
-}));
-
-describe('DomainHooks', () => {
-   it('useFetchDomains should fetch the Domains', async () => {
-      // const { result } = renderHook(() => useFetchDomains(), { wrapper: createWrapper() });
-      const result = { current: { isSuccess: false, data: '' } };
-      await waitFor(() => {
-         console.log('result.current: ', result.current.data);
-         return expect(result.current.isSuccess).toBe(true);
-      });
-   });
-});
--- a/tests/components/DomainItem.test.tsx
+++ b/tests/components/DomainItem.test.tsx
@@ -0,0 +1,35 @@
+import { fireEvent, render } from '@testing-library/react';
+import DomainItem from '../../components/domains/DomainItem';
+import { dummyDomain } from '../../__mocks__/data';
+
+const updateThumbMock = jest.fn();
+const domainItemProps = {
+   domain: dummyDomain,
+   selected: false,
+   isConsoleIntegrated: false,
+   thumb: '',
+   updateThumb: updateThumbMock,
+};
+
+describe('DomainItem Component', () => {
+   it('renders without crashing', async () => {
+      const { container } = render(<DomainItem {...domainItemProps} />);
+      expect(container.querySelector('.domItem')).toBeInTheDocument();
+   });
+   it('renders keywords count', async () => {
+      const { container } = render(<DomainItem {...domainItemProps} />);
+      const domStatskeywords = container.querySelector('.dom_stats div:nth-child(1)');
+      expect(domStatskeywords?.textContent).toBe('Keywords10');
+   });
+   it('renders avg position', async () => {
+      const { container } = render(<DomainItem {...domainItemProps} />);
+      const domStatsAvg = container.querySelector('.dom_stats div:nth-child(2)');
+      expect(domStatsAvg?.textContent).toBe('Avg position24');
+   });
+   it('updates domain thumbnail on relevant button click', async () => {
+      const { container } = render(<DomainItem {...domainItemProps} />);
+      const reloadThumbbBtn = container.querySelector('.domain_thumb button');
+      if (reloadThumbbBtn) fireEvent.click(reloadThumbbBtn);
+      expect(updateThumbMock).toHaveBeenCalledWith(dummyDomain.domain);
+   });
+});
--- a/tests/components/Icon.test.tsx
+++ b/tests/components/Icon.test.tsx
--- a/tests/components/Keyword.test.tsx
+++ b/tests/components/Keyword.test.tsx
@@ -1,8 +1,14 @@
 import { fireEvent, render, screen } from '@testing-library/react';
 import Keyword from '../../components/keywords/Keyword';
-import { dummyKeywords } from '../data';
+import { dummyKeywords } from '../../__mocks__/data';

-const keywordFunctions = {
+const keywordProps = {
+   keywordData: dummyKeywords[0],
+   selected: false,
+   index: 0,
+   showSCData: false,
+   scDataType: '',
+   style: {},
   refreshkeyword: jest.fn(),
   favoriteKeyword: jest.fn(),
   removeKeyword: jest.fn(),
@@ -10,35 +16,37 @@ const keywordFunctions = {
   manageTags: jest.fn(),
   showKeywordDetails: jest.fn(),
 };
-
+jest.mock('react-chartjs-2', () => ({
+   Line: () => null,
+ }));
 describe('Keyword Component', () => {
   it('renders without crashing', async () => {
-       render(<Keyword keywordData={dummyKeywords[0]} {...keywordFunctions} selected={false} />);
+       render(<Keyword {...keywordProps} />);
       expect(await screen.findByText('compress image')).toBeInTheDocument();
   });
   it('Should Render Position Correctly', async () => {
-      render(<Keyword keywordData={dummyKeywords[0]} {...keywordFunctions} selected={false} />);
+      render(<Keyword {...keywordProps} />);
      const positionElement = document.querySelector('.keyword_position');
      expect(positionElement?.childNodes[0].nodeValue).toBe('19');
   });
   it('Should Display Position Change arrow', async () => {
-      render(<Keyword keywordData={dummyKeywords[0]} {...keywordFunctions} selected={false} />);
+      render(<Keyword {...keywordProps} />);
      const positionElement = document.querySelector('.keyword_position i');
      expect(positionElement?.textContent).toBe('▲ 1');
   });
   it('Should Display the SERP Page URL', async () => {
-      render(<Keyword keywordData={dummyKeywords[0]} {...keywordFunctions} selected={false} />);
+      render(<Keyword {...keywordProps} />);
      const positionElement = document.querySelector('.keyword_url');
      expect(positionElement?.textContent).toBe('/');
   });
   it('Should Display the Keyword Options on dots Click', async () => {
-      render(<Keyword keywordData={dummyKeywords[0]} {...keywordFunctions} selected={false} />);
-      const button = document.querySelector('.keyword .keyword_dots');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      const { container } = render(<Keyword {...keywordProps} />);
+      const button = container.querySelector('.keyword_dots');
+      if (button) fireEvent.click(button);
      expect(document.querySelector('.keyword_options')).toBeVisible();
   });
   // it('Should favorite Keywords', async () => {
-   //    render(<Keyword keywordData={dummyKeywords[0]} {...keywordFunctions} selected={false} />);
+   //    render(<Keyword {...keywordProps} />);
   //    const button = document.querySelector('.keyword .keyword_dots');
   //    if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
   //    const option = document.querySelector('.keyword .keyword_options li:nth-child(1) a');
--- a/tests/components/Modal.test.tsx
+++ b/tests/components/Modal.test.tsx
@@ -0,0 +1,28 @@
+import { fireEvent, render, screen } from '@testing-library/react';
+import Modal from '../../components/common/Modal';
+
+const closeModalMock = jest.fn();
+describe('Modal Component', () => {
+   it('Renders without crashing', async () => {
+       render(<Modal closeModal={closeModalMock }><div></div></Modal>);
+       expect(document.querySelector('.modal')).toBeInTheDocument();
+   });
+   it('Displays the Given Content', async () => {
+      render(<Modal closeModal={closeModalMock}>
+        <div>
+           <h1>Hello Modal!!</h1>
+        </div>
+      </Modal>);
+      expect(await screen.findByText('Hello Modal!!')).toBeInTheDocument();
+   });
+   it('Renders Modal Title', async () => {
+      render(<Modal closeModal={closeModalMock} title="Sample Modal Title"><p>Some Modal Content</p></Modal>);
+      expect(await screen.findByText('Sample Modal Title')).toBeInTheDocument();
+   });
+   it('Closes the modal on close button click', async () => {
+      const { container } = render(<Modal closeModal={closeModalMock} title="Sample Modal Title"><p>Some Modal Content</p></Modal>);
+      const closeBtn = container.querySelector('.modal-close');
+      if (closeBtn) fireEvent.click(closeBtn);
+      expect(closeModalMock).toHaveBeenCalled();
+   });
+});
--- a/tests/components/Sidebar.test.tsx
+++ b/tests/components/Sidebar.test.tsx
@@ -0,0 +1,23 @@
+import { fireEvent, render, screen } from '@testing-library/react';
+import Sidebar from '../../components/common/Sidebar';
+import { dummyDomain } from '../../__mocks__/data';
+
+const addDomainMock = jest.fn();
+jest.mock('next/router', () => jest.requireActual('next-router-mock'));
+
+describe('Sidebar Component', () => {
+   it('renders without crashing', async () => {
+       render(<Sidebar domains={[dummyDomain]} showAddModal={addDomainMock} />);
+       expect(screen.getByText('SerpBear')).toBeInTheDocument();
+   });
+   it('renders domain list', async () => {
+      render(<Sidebar domains={[dummyDomain]} showAddModal={addDomainMock} />);
+      expect(screen.getByText('compressimage.io')).toBeInTheDocument();
+   });
+   it('calls showAddModal on Add Domain button click', async () => {
+      render(<Sidebar domains={[dummyDomain]} showAddModal={addDomainMock} />);
+      const addDomainBtn = screen.getByTestId('add_domain');
+      fireEvent.click(addDomainBtn);
+      expect(addDomainMock).toHaveBeenCalledWith(true);
+   });
+});
--- a/tests/components/Topbar.test.tsx
+++ b/tests/components/Topbar.test.tsx
@@ -1,9 +1,15 @@
 import { render, screen } from '@testing-library/react';
 import TopBar from '../../components/common/TopBar';

+jest.mock('next/router', () => ({
+   useRouter: () => ({
+      pathname: '/',
+   }),
+}));
+
 describe('TopBar Component', () => {
   it('renders without crashing', async () => {
-       render(<TopBar showSettings={() => console.log() } />);
+       render(<TopBar showSettings={jest.fn} showAddModal={jest.fn} />);
       expect(
           await screen.findByText('SerpBear'),
       ).toBeInTheDocument();
--- a/tests/hooks/domains.test.tsx
+++ b/tests/hooks/domains.test.tsx
@@ -0,0 +1,27 @@
+import { renderHook, waitFor } from '@testing-library/react';
+import mockRouter from 'next-router-mock';
+
+import { useFetchDomains } from '../../services/domains';
+import { createWrapper } from '../../__mocks__/utils';
+import { dummyDomain } from '../../__mocks__/data';
+
+jest.mock('next/router', () => jest.requireActual('next-router-mock'));
+
+fetchMock.mockIf(`${window.location.origin}/api/domains`, async () => {
+   return new Promise((resolve) => {
+      resolve({
+         body: JSON.stringify({ domains: [dummyDomain] }),
+         status: 200,
+      });
+   });
+});
+
+describe('DomainHooks', () => {
+   it('useFetchDomains should fetch the Domains', async () => {
+      const { result } = renderHook(() => useFetchDomains(mockRouter), { wrapper: createWrapper() });
+      // const result = { current: { isSuccess: false, data: '' } };
+      await waitFor(() => {
+         return expect(result.current.isLoading).toBe(false);
+      });
+   });
+});
--- a/tests/pages/domain.test.tsx
+++ b/tests/pages/domain.test.tsx
@@ -1,17 +1,26 @@
 import { fireEvent, render, screen } from '@testing-library/react';
+import { QueryClient, QueryClientProvider } from 'react-query';
+import { MockResponseInitFunction } from 'jest-fetch-mock';
 import SingleDomain from '../../pages/domain/[slug]';
 import { useAddDomain, useDeleteDomain, useFetchDomains, useUpdateDomain } from '../../services/domains';
 import { useAddKeywords, useDeleteKeywords, useFavKeywords, useFetchKeywords, useRefreshKeywords } from '../../services/keywords';
-import { dummyDomain, dummyKeywords } from '../data';
+import { dummyDomain, dummyKeywords, dummySettings } from '../../__mocks__/data';
+import { useFetchSettings } from '../../services/settings';

 jest.mock('../../services/domains');
 jest.mock('../../services/keywords');
+jest.mock('../../services/settings');
+
 jest.mock('next/router', () => ({
   useRouter: () => ({
     query: { slug: dummyDomain.slug },
   }),
 }));

+jest.mock('react-chartjs-2', () => ({
+   Line: () => null,
+}));
+
 const useFetchDomainsFunc = useFetchDomains as jest.Mock<any>;
 const useFetchKeywordsFunc = useFetchKeywords as jest.Mock<any>;
 const useDeleteKeywordsFunc = useDeleteKeywords as jest.Mock<any>;
@@ -21,9 +30,12 @@ const useAddDomainFunc = useAddDomain as jest.Mock<any>;
 const useAddKeywordsFunc = useAddKeywords as jest.Mock<any>;
 const useUpdateDomainFunc = useUpdateDomain as jest.Mock<any>;
 const useDeleteDomainFunc = useDeleteDomain as jest.Mock<any>;
+const useFetchSettingsFunc = useFetchSettings as jest.Mock<any>;

 describe('SingleDomain Page', () => {
+   const queryClient = new QueryClient();
   beforeEach(() => {
+      useFetchSettingsFunc.mockImplementation(() => ({ data: { settings: dummySettings }, isLoading: false }));
      useFetchDomainsFunc.mockImplementation(() => ({ data: { domains: [dummyDomain] }, isLoading: false }));
      useFetchKeywordsFunc.mockImplementation(() => ({ keywordsData: { keywords: dummyKeywords }, keywordsLoading: false }));
      useDeleteKeywordsFunc.mockImplementation(() => ({ mutate: () => { } }));
@@ -38,158 +50,163 @@ describe('SingleDomain Page', () => {
      jest.clearAllMocks();
   });
   it('Render without crashing.', async () => {
-      const { getByTestId } = render(<SingleDomain />);
-      // screen.debug(undefined, Infinity);
-      expect(getByTestId('domain-header')).toBeInTheDocument();
-      // expect(await result.findByText(/compressimage/i)).toBeInTheDocument();
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      expect(screen.getByTestId('domain-header')).toBeInTheDocument();
   });
   it('Should Call the useFetchDomains hook on render.', async () => {
-      render(<SingleDomain />);
-      // screen.debug(undefined, Infinity);
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
      expect(useFetchDomains).toHaveBeenCalled();
-      // expect(await result.findByText(/compressimage/i)).toBeInTheDocument();
   });
   it('Should Render the Keywords', async () => {
-      render(<SingleDomain />);
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
      const keywordsCount = document.querySelectorAll('.keyword').length;
      expect(keywordsCount).toBe(2);
   });
   it('Should Display the Keywords Details Sidebar on Keyword Click.', async () => {
-      render(<SingleDomain />);
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
      const keywords = document.querySelectorAll('.keyword');
      const firstKeyword = keywords && keywords[0].querySelector('a');
-      if (firstKeyword) fireEvent(firstKeyword, new MouseEvent('click', { bubbles: true }));
+      if (firstKeyword) fireEvent.click(firstKeyword);
+      const fn: MockResponseInitFunction = async () => {
+         return new Promise((resolve) => {
+            resolve({
+               body: JSON.stringify({ keyword: dummyKeywords[0] }),
+               status: 200,
+            });
+         });
+      };
+      fetchMock.mockIf(`${window.location.origin}/api/keyword?id=${dummyKeywords[0].ID}`, fn);
+
      expect(screen.getByTestId('keywordDetails')).toBeVisible();
   });
   it('Should Display the AddDomain Modal on Add Domain Button Click.', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=add_domain]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('add_domain');
+      if (button) fireEvent.click(button);
      expect(screen.getByTestId('adddomain_modal')).toBeVisible();
   });
   it('Should Display the AddKeywords Modal on Add Keyword Button Click.', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=add_keyword]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('add_keyword');
+      if (button) fireEvent.click(button);
      expect(screen.getByTestId('addkeywords_modal')).toBeVisible();
   });

   it('Should display the Domain Settings on Settings Button click.', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=show_domain_settings]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('show_domain_settings');
+      if (button) fireEvent.click(button);
      expect(screen.getByTestId('domain_settings')).toBeVisible();
   });

   it('Device Tab change should be functioning.', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=mobile_tab]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('mobile_tab');
+      if (button) fireEvent.click(button);
      const keywordsCount = document.querySelectorAll('.keyword').length;
      expect(keywordsCount).toBe(0);
   });

   it('Search Filter should function properly', async () => {
-      render(<SingleDomain />);
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
      const inputNode = screen.getByTestId('filter_input');
-      fireEvent.change(inputNode, { target: { value: 'compressor' } }); // triggers onChange event
+      if (inputNode) fireEvent.change(inputNode, { target: { value: 'compressor' } }); // triggers onChange event
      expect(inputNode.getAttribute('value')).toBe('compressor');
      const keywordsCount = document.querySelectorAll('.keyword').length;
      expect(keywordsCount).toBe(1);
   });

   it('Country Filter should function properly', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=filter_button]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('filter_button');
+      if (button) fireEvent.click(button);
      expect(document.querySelector('.country_filter')).toBeVisible();

      const countrySelect = document.querySelector('.country_filter .selected');
-      if (countrySelect) fireEvent(countrySelect, new MouseEvent('click', { bubbles: true }));
+      if (countrySelect) fireEvent.click(countrySelect);
      expect(document.querySelector('.country_filter .select_list')).toBeVisible();
      const firstCountry = document.querySelector('.country_filter .select_list ul li:nth-child(1)');
-      if (firstCountry) fireEvent(firstCountry, new MouseEvent('click', { bubbles: true }));
+      if (firstCountry) fireEvent.click(firstCountry);
      const keywordsCount = document.querySelectorAll('.keyword').length;
      expect(keywordsCount).toBe(0);
   });

   // Tags Filter should function properly
   it('Tags Filter should Render & Function properly', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=filter_button]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('filter_button');
+      if (button) fireEvent.click(button);
      expect(document.querySelector('.tags_filter')).toBeVisible();

      const countrySelect = document.querySelector('.tags_filter .selected');
-      if (countrySelect) fireEvent(countrySelect, new MouseEvent('click', { bubbles: true }));
+      if (countrySelect) fireEvent.click(countrySelect);
      expect(document.querySelector('.tags_filter .select_list')).toBeVisible();
      expect(document.querySelectorAll('.tags_filter .select_list ul li').length).toBe(1);

      const firstTag = document.querySelector('.tags_filter .select_list ul li:nth-child(1)');
-      if (firstTag) fireEvent(firstTag, new MouseEvent('click', { bubbles: true }));
+      if (firstTag) fireEvent.click(firstTag);
      expect(document.querySelectorAll('.keyword').length).toBe(1);
   });

   it('Sort Options Should be visible Sort Button on Click.', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=sort_button]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('sort_button');
+      if (button) fireEvent.click(button);
      expect(document.querySelector('.sort_options')).toBeVisible();
   });

   it('Sort: Position should sort keywords accordingly', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=sort_button]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
-
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('sort_button');
+      if (button) fireEvent.click(button);
      // Test Top Position Sort
      const topPosSortOption = document.querySelector('ul.sort_options li:nth-child(1)');
-      if (topPosSortOption) fireEvent(topPosSortOption, new MouseEvent('click', { bubbles: true }));
+      if (topPosSortOption) fireEvent.click(topPosSortOption);
      const firstKeywordTitle = document.querySelector('.domKeywords_keywords .keyword:nth-child(1) a')?.textContent;
      expect(firstKeywordTitle).toBe('compress image');

      // Test Lowest Position Sort
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      if (button) fireEvent.click(button);
      const lowestPosSortOption = document.querySelector('ul.sort_options li:nth-child(2)');
-      if (lowestPosSortOption) fireEvent(lowestPosSortOption, new MouseEvent('click', { bubbles: true }));
+      if (lowestPosSortOption) fireEvent.click(lowestPosSortOption);
      const secondKeywordTitle = document.querySelector('.domKeywords_keywords .keyword:nth-child(1) a')?.textContent;
      expect(secondKeywordTitle).toBe('image compressor');
   });

   it('Sort: Date Added should sort keywords accordingly', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=sort_button]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('sort_button');
+      if (button) fireEvent.click(button);

      // Test Top Position Sort
      const topPosSortOption = document.querySelector('ul.sort_options li:nth-child(3)');
-      if (topPosSortOption) fireEvent(topPosSortOption, new MouseEvent('click', { bubbles: true }));
+      if (topPosSortOption) fireEvent.click(topPosSortOption);
      const firstKeywordTitle = document.querySelector('.domKeywords_keywords .keyword:nth-child(1) a')?.textContent;
      expect(firstKeywordTitle).toBe('compress image');

      // Test Lowest Position Sort
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      if (button) fireEvent.click(button);
      const lowestPosSortOption = document.querySelector('ul.sort_options li:nth-child(4)');
-      if (lowestPosSortOption) fireEvent(lowestPosSortOption, new MouseEvent('click', { bubbles: true }));
+      if (lowestPosSortOption) fireEvent.click(lowestPosSortOption);
      const secondKeywordTitle = document.querySelector('.domKeywords_keywords .keyword:nth-child(1) a')?.textContent;
      expect(secondKeywordTitle).toBe('image compressor');
   });

   it('Sort: Alphabetical should sort keywords accordingly', async () => {
-      render(<SingleDomain />);
-      const button = document.querySelector('[data-testid=sort_button]');
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      render(<QueryClientProvider client={queryClient}><SingleDomain /></QueryClientProvider>);
+      const button = screen.getByTestId('sort_button');
+      if (button) fireEvent.click(button);

      // Test Top Position Sort
      const topPosSortOption = document.querySelector('ul.sort_options li:nth-child(5)');
-      if (topPosSortOption) fireEvent(topPosSortOption, new MouseEvent('click', { bubbles: true }));
+      if (topPosSortOption) fireEvent.click(topPosSortOption);
      const firstKeywordTitle = document.querySelector('.domKeywords_keywords .keyword:nth-child(1) a')?.textContent;
      expect(firstKeywordTitle).toBe('compress image');

      // Test Lowest Position Sort
-      if (button) fireEvent(button, new MouseEvent('click', { bubbles: true }));
+      if (button) fireEvent.click(button);
      const lowestPosSortOption = document.querySelector('ul.sort_options li:nth-child(6)');
-      if (lowestPosSortOption) fireEvent(lowestPosSortOption, new MouseEvent('click', { bubbles: true }));
+      if (lowestPosSortOption) fireEvent.click(lowestPosSortOption);
      const secondKeywordTitle = document.querySelector('.domKeywords_keywords .keyword:nth-child(1) a')?.textContent;
      expect(secondKeywordTitle).toBe('image compressor');
   });
--- a/tests/pages/domains.test.tsx
+++ b/tests/pages/domains.test.tsx
@@ -0,0 +1,49 @@
+import { fireEvent, render, screen } from '@testing-library/react';
+import { QueryClient, QueryClientProvider } from 'react-query';
+import * as ReactQuery from 'react-query';
+import { dummyDomain } from '../../__mocks__/data';
+import Domains from '../../pages/domains';
+
+jest.mock('next/router', () => jest.requireActual('next-router-mock'));
+jest.spyOn(ReactQuery, 'useQuery').mockImplementation(jest.fn().mockReturnValue(
+   { data: { domains: [dummyDomain] }, isLoading: false, isSuccess: true },
+));
+
+fetchMock.mockIf(`${window.location.origin}/api/domains`, async () => {
+   return new Promise((resolve) => {
+      resolve({
+         body: JSON.stringify({ domains: [dummyDomain] }),
+         status: 200,
+      });
+   });
+});
+
+describe('Domains Page', () => {
+   const queryClient = new QueryClient();
+   it('Renders without crashing', async () => {
+      render(
+          <QueryClientProvider client={queryClient}>
+              <Domains />
+          </QueryClientProvider>,
+      );
+      expect(screen.getByTestId('domains')).toBeInTheDocument();
+   });
+   it('Renders the Domain Component', async () => {
+      const { container } = render(
+          <QueryClientProvider client={queryClient}>
+              <Domains />
+          </QueryClientProvider>,
+      );
+      expect(container.querySelector('.domItem')).toBeInTheDocument();
+   });
+   it('Should Display Add Domain Modal on relveant Button Click.', async () => {
+      render(<QueryClientProvider client={queryClient}><Domains /></QueryClientProvider>);
+      const button = screen.getByTestId('addDomainButton');
+      if (button) fireEvent.click(button);
+      expect(screen.getByTestId('adddomain_modal')).toBeVisible();
+   });
+   it('Should Display the version number in Footer.', async () => {
+      render(<QueryClientProvider client={queryClient}><Domains /></QueryClientProvider>);
+      expect(screen.getByText('SerpBear v0.0.0')).toBeVisible();
+   });
+});
--- a/tests/pages/index.test.tsx
+++ b/tests/pages/index.test.tsx
@@ -2,21 +2,16 @@ import { render, screen } from '@testing-library/react';
 import { QueryClient, QueryClientProvider } from 'react-query';
 import Home from '../../pages/index';

+const routerPush = jest.fn();
+jest.mock('next/router', () => ({
+   useRouter: () => ({
+      push: routerPush,
+   }),
+}));
+
 describe('Home Page', () => {
   const queryClient = new QueryClient();
   it('Renders without crashing', async () => {
-      // const dummyDomain = {
-      //    ID: 1,
-      //    domain: 'compressimage.io',
-      //    slug: 'compressimage-io',
-      //    keywordCount: 0,
-      //    lastUpdated: '2022-11-11T10:00:32.243',
-      //    added: '2022-11-11T10:00:32.244',
-      //    tags: [],
-      //    notification: true,
-      //    notification_interval: 'daily',
-      //    notification_emails: '',
-      // };
      render(
          <QueryClientProvider client={queryClient}>
              <Home />
@@ -26,12 +21,12 @@ describe('Home Page', () => {
      expect(await screen.findByRole('main')).toBeInTheDocument();
      expect(screen.queryByText('Add Domain')).not.toBeInTheDocument();
   });
-   it('Should Display the Add Domain Modal when there are no Domains.', async () => {
+   it('Should redirect to /domains route.', async () => {
       render(
           <QueryClientProvider client={queryClient}>
               <Home />
           </QueryClientProvider>,
       );
-       expect(await screen.findByText('Add Domain')).toBeInTheDocument();
+       expect(routerPush).toHaveBeenCalledWith('/domains');
   });
 });
--- a/components/common/ChartSlim.tsx
+++ b/components/common/ChartSlim.tsx
@@ -35,7 +35,7 @@ const ChartSlim = ({ labels, sreies }:ChartProps) => {
     },
   };

-   return <div className='w-[120px] h-[30px] rounded border border-gray-200'>
+   return <div className='w-[100px] h-[30px] rounded border border-gray-200'>
         <Line
            datasetIdKey='XXX'
            options={options}
--- a/components/common/Sidebar.tsx
+++ b/components/common/Sidebar.tsx
@@ -1,3 +1,4 @@
+/* eslint-disable @next/next/no-img-element */
 import React from 'react';
 import Link from 'next/link';
 import { useRouter } from 'next/router';
@@ -26,11 +27,11 @@ const Sidebar = ({ domains, showAddModal } : SidebarProps) => {
                                        rounded-r-none ${((`/domain/${d.slug}` === router.asPath || `/domain/console/${d.slug}` === router.asPath
                                        || `/domain/insight/${d.slug}` === router.asPath)
                                        ? 'bg-white text-zinc-800 border border-r-0' : 'text-zinc-500')}`}>
-                                          <i className={'text-center leading-4 mr-2 inline-block rounded-full w-5 h-5 bg-orange-200 not-italic'}>
-                                             {d.domain.charAt(0)}
-                                          </i>
+                                          <img
+                                          className={' inline-block mr-1'}
+                                          src={`https://www.google.com/s2/favicons?domain=${d.domain}&sz=16`} alt={d.domain}
+                                          />
                                          {d.domain}
-                                          {/* <span>0</span> */}
                                       </a>
                                    </Link>
                                 </li>)
--- a/components/domains/AddDomain.tsx
+++ b/components/domains/AddDomain.tsx
@@ -1,6 +1,7 @@
 import React, { useState } from 'react';
 import Modal from '../common/Modal';
 import { useAddDomain } from '../../services/domains';
+import { isValidDomain } from '../../utils/validators';

 type AddDomainProps = {
   closeModal: Function
@@ -13,7 +14,7 @@ const AddDomain = ({ closeModal }: AddDomainProps) => {

   const addDomain = () => {
      // console.log('ADD NEW DOMAIN', newDomain);
-      if (/^[a-zA-Z0-9][a-zA-Z0-9-]{0,61}[a-zA-Z0-9](?:\.[a-zA-Z]{2,})+$/.test(newDomain.trim())) {
+      if (isValidDomain(newDomain.trim())) {
         setNewDomainError(false);
         // TODO: Domain Action
         addMutate(newDomain.trim());
--- a/components/domains/DomainItem.tsx
+++ b/components/domains/DomainItem.tsx
@@ -9,10 +9,12 @@ import Icon from '../common/Icon';
 type DomainItemProps = {
   domain: DomainType,
   selected: boolean,
-   isConsoleIntegrated: boolean
+   isConsoleIntegrated: boolean,
+   thumb: string,
+   updateThumb: Function,
 }

-const DomainItem = ({ domain, selected, isConsoleIntegrated = false }: DomainItemProps) => {
+const DomainItem = ({ domain, selected, isConsoleIntegrated = false, thumb, updateThumb }: DomainItemProps) => {
   const { keywordsUpdated, slug, keywordCount = 0, avgPosition = 0, scVisits = 0, scImpressions = 0, scPosition = 0 } = domain;
   // const router = useRouter();
   return (
@@ -20,11 +22,23 @@ const DomainItem = ({ domain, selected, isConsoleIntegrated = false }: DomainIte
         <Link href={`/domain/${slug}`} passHref={true}>
         <a className='flex flex-col lg:flex-row'>
            <div className={`flex-1 p-6 flex ${!isConsoleIntegrated ? 'basis-1/3' : ''}`}>
-               <div className="domain_thumb w-20 h-20 mr-6 bg-slate-100 rounded border border-gray-200 overflow-hidden">
-                  <img src={`https://image.thum.io/get/maxAge/96/width/200/https://${domain.domain}`} alt={domain.domain} />
+               <div className="group domain_thumb w-20 h-20 mr-6 bg-slate-100 rounded
+                  border border-gray-200 overflow-hidden flex justify-center relative">
+                  <button
+                     className=' absolute right-1 top-0 text-gray-400 p-1 transition-all
+                     invisible opacity-0 group-hover:visible group-hover:opacity-100 hover:text-gray-600 z-10'
+                     title='Reload Website Screenshot'
+                     onClick={(e) => { e.preventDefault(); e.stopPropagation(); updateThumb(domain.domain); }}
+                  >
+                     <Icon type="reload" size={12} />
+                  </button>
+                  <img
+                  className={`self-center ${!thumb ? 'max-w-[50px]' : ''}`}
+                  src={thumb || `https://www.google.com/s2/favicons?domain=${domain.domain}&sz=128`} alt={domain.domain}
+                  />
               </div>
               <div className="domain_details flex-1">
-                  <h3 className='font-semibold text-base mb-2 capitalize'>{domain.domain}</h3>
+                  <h3 className='font-semibold text-base mb-2'>{domain.domain}</h3>
                 {keywordsUpdated && (
                  <span className=' text-gray-600 text-xs'>
                     Updated <TimeAgo title={dayjs(keywordsUpdated).format('DD-MMM-YYYY, hh:mm:ss A')} date={keywordsUpdated} />
--- a/components/domains/DomainSettings.tsx
+++ b/components/domains/DomainSettings.tsx
@@ -41,7 +41,7 @@ const DomainSettings = ({ domain, closeModal }: DomainSettingsProps) => {
      let error: DomainSettingsError | null = null;
      if (domainSettings.notification_emails) {
         const notification_emails = domainSettings.notification_emails.split(',');
-         const invalidEmails = notification_emails.find((x) => /^\w+([.-]?\w+)*@\w+([.-]?\w+)*(\.\w{2,3})+$/.test(x) === false);
+         const invalidEmails = notification_emails.find((x) => /^\w+([.-]?\w+)*@\w+([.-]?\w+)*(\.\w{2,15})+$/.test(x) === false);
         console.log('invalidEmails: ', invalidEmails);
         if (invalidEmails) {
            error = { type: 'email', msg: 'Invalid Email' };
--- a/components/insight/Insight.tsx
+++ b/components/insight/Insight.tsx
@@ -12,7 +12,7 @@ type SCInsightProps = {
   isConsoleIntegrated: boolean,
 }

-const SCInsight = ({ insight, isLoading = true, isConsoleIntegrated = true }: SCInsightProps) => {
+const SCInsight = ({ insight, isLoading = true, isConsoleIntegrated = true, domain }: SCInsightProps) => {
   const [activeTab, setActiveTab] = useState<string>('stats');

   const insightItems = insight[activeTab as keyof InsightDataType];
@@ -108,7 +108,7 @@ const SCInsight = ({ insight, isLoading = true, isConsoleIntegrated = true }: SC
                              (item:SCInsightItem, index: number) => {
                              const insightItemCount = insight ? insightItems : [];
                              const lastItem = !!(insightItemCount && (index === insightItemCount.length));
-                              return <InsightItem key={index} item={item} type={activeTab} lastItem={lastItem} />;
+                              return <InsightItem key={index} item={item} type={activeTab} lastItem={lastItem} domain={domain?.domain || ''} />;
                           },
                        )
                     }
@@ -117,7 +117,7 @@ const SCInsight = ({ insight, isLoading = true, isConsoleIntegrated = true }: SC
                     )}
                     {!isConsoleIntegrated && (
                        <p className=' p-9 pt-[10%] text-center text-gray-500'>
-                        Goolge Search has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
+                        Google Search has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
                        </p>
                     )}
                  </div>
--- a/components/insight/InsightItem.tsx
+++ b/components/insight/InsightItem.tsx
@@ -5,10 +5,11 @@ import Icon from '../common/Icon';
 type InsightItemProps = {
   item: SCInsightItem,
   lastItem: boolean,
-   type: string
+   type: string,
+   domain: string
 }

-const InsightItem = ({ item, lastItem, type }:InsightItemProps) => {
+const InsightItem = ({ item, lastItem, type, domain }:InsightItemProps) => {
   const { clicks, impressions, ctr, position, country = 'zzz', keyword, page, keywords = 0, countries: cntrs = 0, date } = item;
   let firstItem = keyword;
   if (type === 'pages') { firstItem = page; } if (type === 'stats') {
@@ -24,7 +25,7 @@ const InsightItem = ({ item, lastItem, type }:InsightItemProps) => {

         <div className=' w-3/4 lg:flex-1 lg:basis-20 lg:w-auto font-semibold'>
            {type === 'countries' && <span className={`fflag fflag-${country} w-[18px] h-[12px] mr-2`} />}
-            {firstItem}
+            {type === 'pages' && domain ? <a href={`https://${domain}${page}`} target='_blank' rel="noreferrer">{firstItem}</a> : firstItem}
         </div>

         <div className='keyword_pos text-center inline-block mr-3 lg:mr-0 lg:flex-1'>
--- a/components/keywords/AddKeywords.tsx
+++ b/components/keywords/AddKeywords.tsx
@@ -21,7 +21,8 @@ type KeywordsInput = {

 const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {
   const [error, setError] = useState<string>('');
-   const [newKeywordsData, setNewKeywordsData] = useState<KeywordsInput>({ keywords: '', device: 'desktop', country: 'US', domain, tags: '' });
+   const defCountry = localStorage.getItem('default_country') || 'US';
+   const [newKeywordsData, setNewKeywordsData] = useState<KeywordsInput>({ keywords: '', device: 'desktop', country: defCountry, domain, tags: '' });
   const { mutate: addMutate, isLoading: isAdding } = useAddKeywords(() => closeModal(false));
   const deviceTabStyle = 'cursor-pointer px-3 py-2 rounded mr-2';

@@ -64,7 +65,10 @@ const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {
                     selected={[newKeywordsData.country]}
                     options={Object.keys(countries).map((countryISO:string) => { return { label: countries[countryISO][0], value: countryISO }; })}
                     defaultLabel='All Countries'
-                     updateField={(updated:string[]) => setNewKeywordsData({ ...newKeywordsData, country: updated[0] })}
+                     updateField={(updated:string[]) => {
+                        setNewKeywordsData({ ...newKeywordsData, country: updated[0] });
+                        localStorage.setItem('default_country', updated[0]);
+                     }}
                     rounded='rounded'
                     maxHeight={48}
                     flags={true}
--- a/components/keywords/AddTags.tsx
+++ b/components/keywords/AddTags.tsx
@@ -0,0 +1,60 @@
+import { useState } from 'react';
+import { useUpdateKeywordTags } from '../../services/keywords';
+import Icon from '../common/Icon';
+import Modal from '../common/Modal';
+
+type AddTagsProps = {
+   keywords: KeywordType[],
+   closeModal: Function
+}
+
+const AddTags = ({ keywords = [], closeModal }: AddTagsProps) => {
+   const [tagInput, setTagInput] = useState('');
+   const [inputError, setInputError] = useState('');
+   const { mutate: updateMutate } = useUpdateKeywordTags(() => { setTagInput(''); });
+
+   const addTag = () => {
+      if (keywords.length === 0) { return; }
+      if (!tagInput) {
+         setInputError('Please Insert a Tag!');
+         setTimeout(() => { setInputError(''); }, 3000);
+         return;
+      }
+
+      const tagsArray = tagInput.split(',').map((t) => t.trim());
+      const tagsPayload:any = {};
+      keywords.forEach((keyword:KeywordType) => {
+         tagsPayload[keyword.ID] = [...keyword.tags, ...tagsArray];
+      });
+      updateMutate({ tags: tagsPayload });
+   };
+
+   return (
+      <Modal closeModal={() => { closeModal(false); }} title={`Add New Tags to ${keywords.length} Selected Keyword`}>
+         <div className="relative">
+            {inputError && <span className="absolute top-[-24px] text-red-400 text-sm font-semibold">{inputError}</span>}
+            <span className='absolute text-gray-400 top-3 left-2'><Icon type="tags" size={16} /></span>
+            <input
+               className='w-full border rounded border-gray-200 py-3 px-4 pl-8 outline-none focus:border-indigo-300'
+               placeholder='Insert Tags. eg: tag1, tag2'
+               value={tagInput}
+               onChange={(e) => setTagInput(e.target.value)}
+               onKeyDown={(e) => {
+                  if (e.code === 'Enter') {
+                    e.preventDefault();
+                    addTag();
+                  }
+               }}
+            />
+            <button
+            className=" absolute right-2 top-2 cursor-pointer rounded p-2 px-4 bg-indigo-600 text-white font-semibold text-sm"
+            onClick={addTag}>
+               Apply
+            </button>
+         </div>
+      </Modal>
+
+   );
+};
+
+export default AddTags;
--- a/components/keywords/Keyword.tsx
+++ b/components/keywords/Keyword.tsx
@@ -9,6 +9,7 @@ import { generateTheChartData } from '../common/generateChartData';
 type KeywordProps = {
   keywordData: KeywordType,
   selected: boolean,
+   index: number,
   refreshkeyword: Function,
   favoriteKeyword: Function,
   removeKeyword: Function,
@@ -18,6 +19,7 @@ type KeywordProps = {
   lastItem?:boolean,
   showSCData: boolean,
   scDataType: string,
+   style: Object
 }

 const Keyword = (props: KeywordProps) => {
@@ -32,6 +34,8 @@ const Keyword = (props: KeywordProps) => {
      manageTags,
      lastItem,
      showSCData = true,
+      style,
+      index,
      scDataType = 'threeDays',
   } = props;
   const {
@@ -63,10 +67,23 @@ const Keyword = (props: KeywordProps) => {
      return status;
   }, [history, position]);

+   const bestPosition: false | {position: number, date: string} = useMemo(() => {
+      let bestPos;
+      if (Object.keys(history).length > 0) {
+         const historyArray = Object.keys(history).map((itemID) => ({ date: itemID, position: history[itemID] }))
+         .sort((a, b) => a.position - b.position);
+         if (historyArray[0]) {
+            bestPos = { ...historyArray[0] };
+         }
+      }
+
+      return bestPos || false;
+   }, [history]);
+
   const optionsButtonStyle = 'block px-2 py-2 cursor-pointer hover:bg-indigo-50 hover:text-blue-700';

   const renderPosition = (pos:number, type?:string) => {
-      if (pos === 0) {
+      if (!updating && pos === 0) {
         return <span className='text-gray-400' title='Not in Top 100'>{'>100'}</span>;
      }
      if (updating && type !== 'sc') {
@@ -78,8 +95,10 @@ const Keyword = (props: KeywordProps) => {
   return (
      <div
      key={keyword}
+      style={style}
      className={`keyword relative py-5 px-4 text-gray-600 border-b-[1px] border-gray-200 lg:py-4 lg:px-6 lg:border-0 
      lg:flex lg:justify-between lg:items-center ${selected ? ' bg-indigo-50 keyword--selected' : ''} ${lastItem ? 'border-b-0' : ''}`}>
+
         <div className=' w-3/4 lg:flex-1 lg:basis-20 lg:w-auto font-semibold cursor-pointer'>
            <button
               className={`p-0 mr-2 leading-[0px] inline-block rounded-sm pt-0 px-[1px] pb-[3px] border 
@@ -100,22 +119,40 @@ const Keyword = (props: KeywordProps) => {
               </button>
            }
         </div>
+
         <div
         className={`keyword_position absolute bg-[#f8f9ff] w-fit min-w-[50px] h-12 p-2 text-base mt-[-20px] rounded right-5 lg:relative
-          lg:bg-transparent lg:w-auto lg:h-auto lg:mt-0 lg:p-0 lg:text-sm lg:flex-1 lg:basis-40 lg:grow-0 lg:right-0 text-center font-semibold`}>
+          lg:bg-transparent lg:w-auto lg:h-auto lg:mt-0 lg:p-0 lg:text-sm lg:flex-1 lg:basis-24 lg:grow-0 lg:right-0 text-center font-semibold`}>
            {renderPosition(position)}
            {!updating && positionChange > 0 && <i className=' not-italic ml-1 text-xs text-[#5ed7c3]'>▲ {positionChange}</i>}
            {!updating && positionChange < 0 && <i className=' not-italic ml-1 text-xs text-red-300'>▼ {positionChange}</i>}
         </div>
+
+         <div
+         title={bestPosition && bestPosition.date
+            ? new Date(bestPosition.date).toLocaleDateString('en-US', { weekday: 'long', year: 'numeric', month: 'short', day: 'numeric' }) : ''
+         }
+         className={`keyword_best hidden bg-[#f8f9ff] w-fit min-w-[50px] h-12 p-2 text-base mt-[-20px] rounded right-5 lg:relative lg:block
+          lg:bg-transparent lg:w-auto lg:h-auto lg:mt-0 lg:p-0 lg:text-sm lg:flex-1 lg:basis-16 lg:grow-0 lg:right-0 text-center font-semibold`}>
+            {bestPosition ? bestPosition.position || '-' : (position || '-')}
+         </div>
+
         {chartData.labels.length > 0 && (
-            <div className='lg:flex-1 hidden lg:block'>
+            <div
+               className='hidden basis-32 grow-0 cursor-pointer lg:block'
+               onClick={() => showKeywordDetails()}>
               <ChartSlim labels={chartData.labels} sreies={chartData.sreies} />
            </div>
         )}
+
         <div
         className={`keyword_url inline-block mt-4 mr-5 ml-5 lg:flex-1 text-gray-400 lg:m-0 max-w-[70px] 
         overflow-hidden text-ellipsis whitespace-nowrap lg:max-w-none lg:pr-5`}>
-            <span className='mr-3 lg:hidden'><Icon type="link-alt" size={14} color="#999" /></span>{turncatedURL || '-'}</div>
+            <a href={url} target="_blank" rel="noreferrer"><span className='mr-3 lg:hidden'>
+               <Icon type="link-alt" size={14} color="#999" /></span>{turncatedURL || '-'}
+            </a>
+         </div>
+
         <div
         className='inline-block mt-[4] top-[-5px] relative lg:flex-1 lg:m-0'>
            <span className='mr-2 lg:hidden'><Icon type="clock" size={14} color="#999" /></span>
@@ -168,8 +205,10 @@ const Keyword = (props: KeywordProps) => {
               </ul>
            )}
         </div>
+
         {lastUpdateError && lastUpdateError.date && showPositionError && (
-            <div className=' absolute mt-[-70px] p-2 bg-white z-30 border border-red-200 rounded w-[220px] left-4 shadow-sm text-xs lg:bottom-12'>
+            <div className={`absolute p-2 bg-white z-30 border border-red-200 rounded w-[220px] left-4 shadow-sm text-xs 
+            ${index > 2 ? 'lg:bottom-12 mt-[-70px]' : ' top-12'}`}>
               Error Updating Keyword position (Tried <TimeAgo
                                                         title={dayjs(lastUpdateError.date).format('DD-MMM-YYYY, hh:mm:ss A')}
                                                         date={lastUpdateError.date} />)
@@ -180,7 +219,8 @@ const Keyword = (props: KeywordProps) => {
                  {lastUpdateError.scraper && <strong className='capitalize'>{lastUpdateError.scraper}: </strong>}{lastUpdateError.error}
               </div>
            </div>
-            )}
+         )}
+
      </div>
   );
 };
--- a/components/keywords/KeywordDetails.tsx
+++ b/components/keywords/KeywordDetails.tsx
@@ -85,7 +85,7 @@ const KeywordDetails = ({ keyword, closeDetails }:KeywordDetailsProps) => {
                  <h3 className=' text-lg font-bold'>
                     <span title={countries[keyword.country][0]}
                     className={`fflag fflag-${keyword.country} w-[18px] h-[12px] mr-2`} /> {keyword.keyword}
-                     <span className='py-1 px-2 rounded bg-blue-50 text-blue-700 text-xs font-bold'>{keyword.position}</span>
+                     <span className='py-1 px-2 ml-2 rounded bg-blue-50 text-blue-700 text-xs font-bold'>{keyword.position}</span>
                  </h3>
                  <button
                  className='absolute top-2 right-2 p-2 px-3 text-gray-400 hover:text-gray-700 transition-all hover:rotate-90'
--- a/components/keywords/KeywordFilter.tsx
+++ b/components/keywords/KeywordFilter.tsx
@@ -82,10 +82,10 @@ const KeywordFilters = (props: KeywordFilterProps) => {
      { value: 'alpha_desc', label: 'Alphabetically(Z-A)' },
   ];
   if (integratedConsole) {
-      sortOptionChoices.push({ value: 'imp_asc', label: `Most Viewed${isConsole ? ' (Default)' : ''}` });
-      sortOptionChoices.push({ value: 'imp_desc', label: 'Least Viewed' });
-      sortOptionChoices.push({ value: 'visits_asc', label: 'Most Visited' });
-      sortOptionChoices.push({ value: 'visits_desc', label: 'Least Visited' });
+      sortOptionChoices.push({ value: 'imp_desc', label: `Most Viewed${isConsole ? ' (Default)' : ''}` });
+      sortOptionChoices.push({ value: 'imp_asc', label: 'Least Viewed' });
+      sortOptionChoices.push({ value: 'visits_desc', label: 'Most Visited' });
+      sortOptionChoices.push({ value: 'visits_asc', label: 'Least Visited' });
   }
   if (isConsole) {
      sortOptionChoices.splice(2, 2);
--- a/components/keywords/KeywordTagManager.tsx
+++ b/components/keywords/KeywordTagManager.tsx
@@ -2,6 +2,7 @@ import { useState } from 'react';
 import { useUpdateKeywordTags } from '../../services/keywords';
 import Icon from '../common/Icon';
 import Modal from '../common/Modal';
+import AddTags from './AddTags';

 type keywordTagManagerProps = {
   keyword: KeywordType|undefined,
@@ -10,9 +11,8 @@ type keywordTagManagerProps = {
 }

 const KeywordTagManager = ({ keyword, closeModal }: keywordTagManagerProps) => {
-   const [tagInput, setTagInput] = useState('');
-   const [inputError, setInputError] = useState('');
-   const { mutate: updateMutate } = useUpdateKeywordTags(() => { setTagInput(''); });
+   const [showAddTag, setShowAddTag] = useState<boolean>(false);
+   const { mutate: updateMutate } = useUpdateKeywordTags(() => { });

   const removeTag = (tag:String) => {
      if (!keyword) { return; }
@@ -20,24 +20,6 @@ const KeywordTagManager = ({ keyword, closeModal }: keywordTagManagerProps) => {
      updateMutate({ tags: { [keyword.ID]: newTags } });
   };

-   const addTag = () => {
-      if (!keyword) { return; }
-      if (!tagInput) {
-         setInputError('Please Insert a Tag!');
-         setTimeout(() => { setInputError(''); }, 3000);
-         return;
-      }
-      if (keyword.tags.includes(tagInput)) {
-         setInputError('Tag Exist!');
-         setTimeout(() => { setInputError(''); }, 3000);
-         return;
-      }
-
-      console.log('New Tag: ', tagInput);
-      const newTags = [...keyword.tags, tagInput.trim()];
-      updateMutate({ tags: { [keyword.ID]: newTags } });
-   };
-
   return (
      <Modal closeModal={() => { closeModal(false); }} title={`Tags for Keyword "${keyword && keyword.keyword}"`}>
         <div className="text-sm my-8 ">
@@ -53,31 +35,27 @@ const KeywordTagManager = ({ keyword, closeModal }: keywordTagManagerProps) => {
                                 </button>
                           </li>;
                  })}
+                  <li className='inline-block py-1 px-1'>
+                     <button
+                     title='Add New Tag'
+                     className="cursor-pointer rounded p-1 px-3 bg-indigo-600 text-white font-semibold text-sm"
+                     onClick={() => setShowAddTag(true)}>+</button>
+                  </li>
               </ul>
            )}
            {keyword && keyword.tags.length === 0 && (
-               <div className="text-center w-full text-gray-500">No Tags Added to this Keyword.</div>
+               <div className="text-center w-full text-gray-500">
+                  No Tags Added to this Keyword. <button className=' text-indigo-600' onClick={() => setShowAddTag(true)}>+ Add Tag</button>
+               </div>
            )}
         </div>
-         <div className="relative">
-            {inputError && <span className="absolute top-[-24px] text-red-400 text-sm font-semibold">{inputError}</span>}
-            <span className='absolute text-gray-400 top-3 left-2'><Icon type="tags" size={16} /></span>
-            <input
-               className='w-full border rounded border-gray-200 py-3 px-4 pl-8 outline-none focus:border-indigo-300'
-               placeholder='Insert Tags'
-               value={tagInput}
-               onChange={(e) => setTagInput(e.target.value)}
-               onKeyDown={(e) => {
-                  if (e.code === 'Enter') {
-                    e.preventDefault();
-                    addTag();
-                  }
-               }}
-            />
-            <button className=" absolute right-2 top-2 cursor-pointer rounded p-1 px-4 bg-blue-600 text-white font-bold" onClick={addTag}>+</button>
-         </div>
+         {showAddTag && keyword && (
+            <AddTags
+               keywords={[keyword]}
+               closeModal={() => setShowAddTag(false)}
+               />
+         )}
      </Modal>
-
   );
 };

--- a/components/keywords/KeywordsTable.tsx
+++ b/components/keywords/KeywordsTable.tsx
@@ -1,6 +1,7 @@
-import React, { useState, useMemo } from 'react';
+import React, { useState, useMemo, useEffect } from 'react';
 import { Toaster } from 'react-hot-toast';
 import { CSSTransition } from 'react-transition-group';
+import { FixedSizeList as List, ListChildComponentProps } from 'react-window';
 import AddKeywords from './AddKeywords';
 import { filterKeywords, keywordsByDevice, sortKeywords } from '../../utils/sortFilter';
 import Icon from '../common/Icon';
@@ -10,6 +11,7 @@ import KeywordFilters from './KeywordFilter';
 import Modal from '../common/Modal';
 import { useDeleteKeywords, useFavKeywords, useRefreshKeywords } from '../../services/keywords';
 import KeywordTagManager from './KeywordTagManager';
+import AddTags from './AddTags';

 type KeywordsTableProps = {
   domain: DomainType | null,
@@ -28,6 +30,9 @@ const KeywordsTable = (props: KeywordsTableProps) => {
   const [showKeyDetails, setShowKeyDetails] = useState<KeywordType|null>(null);
   const [showRemoveModal, setShowRemoveModal] = useState<boolean>(false);
   const [showTagManager, setShowTagManager] = useState<null|number>(null);
+   const [showAddTags, setShowAddTags] = useState<boolean>(false);
+   const [isMobile, setIsMobile] = useState<boolean>(false);
+   const [SCListHeight, setSCListHeight] = useState(500);
   const [filterParams, setFilterParams] = useState<KeywordFilters>({ countries: [], tags: [], search: '' });
   const [sortBy, setSortBy] = useState<string>('date_asc');
   const [scDataType, setScDataType] = useState<string>('threeDays');
@@ -45,6 +50,16 @@ const KeywordsTable = (props: KeywordsTableProps) => {
      avgThirtyDays: 'Last Thirty Days Avg',
   };

+   useEffect(() => {
+      setIsMobile(!!(window.matchMedia('only screen and (max-width: 760px)').matches));
+      const resizeList = () => setSCListHeight(window.innerHeight - (isMobile ? 200 : 400));
+      resizeList();
+      window.addEventListener('resize', resizeList);
+      return () => {
+          window.removeEventListener('resize', resizeList);
+      };
+   }, [isMobile]);
+
   const processedKeywords: {[key:string] : KeywordType[]} = useMemo(() => {
      const procKeywords = keywords.filter((x) => x.device === device);
      const filteredKeywords = filterKeywords(procKeywords, filterParams);
@@ -65,6 +80,27 @@ const KeywordsTable = (props: KeywordsTableProps) => {
      }
      setSelectedKeywords(updatedSelectd);
   };
+   const Row = ({ data, index, style }:ListChildComponentProps) => {
+      const keyword = data[index];
+      return (
+         <Keyword
+         key={keyword.ID}
+         style={style}
+         index={index}
+         selected={selectedKeywords.includes(keyword.ID)}
+         selectKeyword={selectKeyword}
+         keywordData={keyword}
+         refreshkeyword={() => refreshMutate({ ids: [keyword.ID] })}
+         favoriteKeyword={favoriteMutate}
+         manageTags={() => setShowTagManager(keyword.ID)}
+         removeKeyword={() => { setSelectedKeywords([keyword.ID]); setShowRemoveModal(true); }}
+         showKeywordDetails={() => setShowKeyDetails(keyword)}
+         lastItem={index === (processedKeywords[device].length - 1)}
+         showSCData={showSCData}
+         scDataType={scDataType}
+         />
+      );
+   };

   const selectedAllItems = selectedKeywords.length === processedKeywords[device].length;

@@ -79,7 +115,7 @@ const KeywordsTable = (props: KeywordsTableProps) => {
                        className='block px-2 py-2 cursor-pointer hover:text-indigo-600'
                        onClick={() => { refreshMutate({ ids: selectedKeywords }); setSelectedKeywords([]); }}
                        >
-                           <span className=' bg-indigo-100 text-blue-700 px-1 rounded'><Icon type="reload" size={11} /></span> Refresh Keyword
+                           <span className=' bg-indigo-100 text-blue-700 px-1 rounded'><Icon type="reload" size={11} /></span> Refresh Keywords
                        </a>
                     </li>
                     <li className='inline-block mr-4'>
@@ -87,7 +123,14 @@ const KeywordsTable = (props: KeywordsTableProps) => {
                        className='block px-2 py-2 cursor-pointer hover:text-indigo-600'
                        onClick={() => setShowRemoveModal(true)}
                        >
-                           <span className=' bg-red-100 text-red-600 px-1 rounded'><Icon type="trash" size={14} /></span> Remove Keyword</a>
+                           <span className=' bg-red-100 text-red-600 px-1 rounded'><Icon type="trash" size={14} /></span> Remove Keywords</a>
+                     </li>
+                     <li className='inline-block mr-4'>
+                        <a
+                        className='block px-2 py-2 cursor-pointer hover:text-indigo-600'
+                        onClick={() => setShowAddTags(true)}
+                        >
+                           <span className=' bg-green-100 text-green-500  px-1 rounded'><Icon type="tags" size={14} /></span> Tag Keywords</a>
                     </li>
                  </ul>
               </div>
@@ -110,7 +153,7 @@ const KeywordsTable = (props: KeywordsTableProps) => {
               <div className=' lg:min-w-[800px]'>
                  <div className={`domKeywords_head domKeywords_head--${sortBy} hidden lg:flex p-3 px-6 bg-[#FCFCFF]
                   text-gray-600 justify-between items-center font-semibold border-y`}>
-                     <span className='domKeywords_head_keyword flex-1 basis-20 w-auto '>
+                     <span className='domKeywords_head_keyword flex-1 basis-[4rem] w-auto '>
                     {processedKeywords[device].length > 0 && (
                        <button
                           className={`p-0 mr-2 leading-[0px] inline-block rounded-sm pt-0 px-[1px] pb-[3px]  border border-slate-300 
@@ -122,8 +165,9 @@ const KeywordsTable = (props: KeywordsTableProps) => {
                     )}
                        Keyword
                     </span>
-                     <span className='domKeywords_head_position flex-1 basis-40 grow-0 text-center'>Position</span>
-                     <span className='domKeywords_head_history flex-1'>History (7d)</span>
+                     <span className='domKeywords_head_position flex-1 basis-24 grow-0 text-center'>Position</span>
+                     <span className='domKeywords_head_best flex-1 basis-16 grow-0 text-center'>Best</span>
+                     <span className='domKeywords_head_history flex-1 basis-32 grow-0 '>History (7d)</span>
                     <span className='domKeywords_head_url flex-1'>URL</span>
                     <span className='domKeywords_head_updated flex-1'>Updated</span>
                     {showSCData && (
@@ -132,7 +176,7 @@ const KeywordsTable = (props: KeywordsTableProps) => {
                           <div>
                              <div
                              className=' w-48 select-none cursor-pointer absolute bg-white rounded-full
-                              px-2 py-[2px] mt-[-22px] ml-3 border border-gray-200 z-50'
+                              px-2 py-[2px] mt-[-22px] ml-3 border border-gray-200 z-40'
                              onClick={() => setShowScDataTypes(!showScDataTypes)}>
                                 <Icon type="google" size={13} /> {scDataObject[scDataType]}
                                 <Icon classes="ml-2" type={showScDataTypes ? 'caret-up' : 'caret-down'} size={10} />
@@ -161,21 +205,19 @@ const KeywordsTable = (props: KeywordsTableProps) => {
                     )}
                  </div>
                  <div className='domKeywords_keywords border-gray-200 min-h-[55vh] relative'>
-                     {processedKeywords[device] && processedKeywords[device].length > 0
-                        && processedKeywords[device].map((keyword, index) => <Keyword
-                                                      key={keyword.ID}
-                                                      selected={selectedKeywords.includes(keyword.ID)}
-                                                      selectKeyword={selectKeyword}
-                                                      keywordData={keyword}
-                                                      refreshkeyword={() => refreshMutate({ ids: [keyword.ID] })}
-                                                      favoriteKeyword={favoriteMutate}
-                                                      manageTags={() => setShowTagManager(keyword.ID)}
-                                                      removeKeyword={() => { setSelectedKeywords([keyword.ID]); setShowRemoveModal(true); }}
-                                                      showKeywordDetails={() => setShowKeyDetails(keyword)}
-                                                      lastItem={index === (processedKeywords[device].length - 1)}
-                                                      showSCData={showSCData}
-                                                      scDataType={scDataType}
-                                                      />)}
+                     {processedKeywords[device] && processedKeywords[device].length > 0 && (
+                        <List
+                        innerElementType="div"
+                        itemData={processedKeywords[device]}
+                        itemCount={processedKeywords[device].length}
+                        itemSize={isMobile ? 146 : 57}
+                        height={SCListHeight}
+                        width={'100%'}
+                        className={'styled-scrollbar'}
+                        >
+                           {Row}
+                        </List>
+                     )}
                     {!isLoading && processedKeywords[device].length === 0 && (
                        <p className=' p-9 pt-[10%] text-center text-gray-500'>No Keywords Added for this Device Type.</p>
                     )}
@@ -222,6 +264,12 @@ const KeywordsTable = (props: KeywordsTableProps) => {
               closeModal={() => setShowTagManager(null)}
               />
         )}
+         {showAddTags && (
+            <AddTags
+               keywords={keywords.filter((k) => selectedKeywords.includes(k.ID))}
+               closeModal={() => setShowAddTags(false)}
+               />
+         )}
         <Toaster position='bottom-center' containerClassName="react_toaster" />
      </div>
   );
--- a/components/keywords/SCKeywordsTable.tsx
+++ b/components/keywords/SCKeywordsTable.tsx
@@ -26,7 +26,7 @@ const SCKeywordsTable = ({ domain, keywords = [], isLoading = true, isConsoleInt
   const [device, setDevice] = useState<string>('desktop');
   const [selectedKeywords, setSelectedKeywords] = useState<string[]>([]);
   const [filterParams, setFilterParams] = useState<KeywordFilters>({ countries: [], tags: [], search: '' });
-   const [sortBy, setSortBy] = useState<string>('imp_asc');
+   const [sortBy, setSortBy] = useState<string>('imp_desc');
   const [isMobile, setIsMobile] = useState<boolean>(false);
   const [SCListHeight, setSCListHeight] = useState(500);
   const { keywordsData } = useFetchKeywords(router);
@@ -214,7 +214,7 @@ const SCKeywordsTable = ({ domain, keywords = [], isLoading = true, isConsoleInt
                     )}
                     {!isConsoleIntegrated && (
                        <p className=' p-9 pt-[10%] text-center text-gray-500'>
-                        Goolge Search has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
+                        Google Search has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
                        </p>
                     )}
                  </div>
--- a/components/settings/NotificationSettings.tsx
+++ b/components/settings/NotificationSettings.tsx
@@ -0,0 +1,112 @@
+import React from 'react';
+import SelectField from '../common/SelectField';
+
+type NotificationSettingsProps = {
+   settings: SettingsType,
+   settingsError: null | {
+      type: string,
+      msg: string
+   },
+   updateSettings: Function,
+}
+
+const NotificationSettings = ({ settings, settingsError, updateSettings }:NotificationSettingsProps) => {
+   const labelStyle = 'mb-2 font-semibold inline-block text-sm text-gray-700 capitalize';
+
+   return (
+      <div>
+         <div className='settings__content styled-scrollbar p-6 text-sm'>
+            <div className="settings__section__input mb-5">
+               <label className={labelStyle}>Notification Frequency</label>
+               <SelectField
+                  multiple={false}
+                  selected={[settings.notification_interval]}
+                  options={[
+                     { label: 'Daily', value: 'daily' },
+                     { label: 'Weekly', value: 'weekly' },
+                     { label: 'Monthly', value: 'monthly' },
+                     { label: 'Never', value: 'never' },
+                  ]}
+                  defaultLabel={'Notification Settings'}
+                  updateField={(updated:string[]) => updated[0] && updateSettings('notification_interval', updated[0])}
+                  rounded='rounded'
+                  maxHeight={48}
+                  minWidth={270}
+               />
+            </div>
+            {settings.notification_interval !== 'never' && (
+               <>
+                  <div className="settings__section__input mb-5">
+                     <label className={labelStyle}>Notification Emails</label>
+                     <input
+                        className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
+                        ${settingsError?.type === 'no_email' ? ' border-red-400 focus:border-red-400' : ''} `}
+                        type="text"
+                        value={settings?.notification_email}
+                        placeholder={'test@gmail.com'}
+                        onChange={(event) => updateSettings('notification_email', event.target.value)}
+                     />
+                  </div>
+                  <div className="settings__section__input mb-5">
+                     <label className={labelStyle}>SMTP Server</label>
+                     <input
+                        className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
+                        ${settingsError?.type === 'no_smtp_server' ? ' border-red-400 focus:border-red-400' : ''} `}
+                        type="text"
+                        value={settings?.smtp_server || ''}
+                        onChange={(event) => updateSettings('smtp_server', event.target.value)}
+                     />
+                  </div>
+                  <div className="settings__section__input mb-5">
+                     <label className={labelStyle}>SMTP Port</label>
+                     <input
+                        className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
+                        ${settingsError && settingsError.type === 'no_smtp_port' ? ' border-red-400 focus:border-red-400' : ''} `}
+                        type="text"
+                        value={settings?.smtp_port || ''}
+                        onChange={(event) => updateSettings('smtp_port', event.target.value)}
+                     />
+                  </div>
+                  <div className="settings__section__input mb-5">
+                     <label className={labelStyle}>SMTP Username</label>
+                     <input
+                        className={'w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200'}
+                        type="text"
+                        value={settings?.smtp_username || ''}
+                        onChange={(event) => updateSettings('smtp_username', event.target.value)}
+                     />
+                  </div>
+                  <div className="settings__section__input mb-5">
+                     <label className={labelStyle}>SMTP Password</label>
+                     <input
+                        className={'w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200'}
+                        type="text"
+                        value={settings?.smtp_password || ''}
+                        onChange={(event) => updateSettings('smtp_password', event.target.value)}
+                     />
+                  </div>
+                  <div className="settings__section__input mb-5">
+                     <label className={labelStyle}>From Email Address</label>
+                     <input
+                        className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
+                        ${settingsError?.type === 'no_smtp_from' ? ' border-red-400 focus:border-red-400' : ''} `}
+                        type="text"
+                        value={settings?.notification_email_from || ''}
+                        placeholder="no-reply@mydomain.com"
+                        onChange={(event) => updateSettings('notification_email_from', event.target.value)}
+                     />
+                  </div>
+               </>
+            )}
+
+            </div>
+            {settingsError?.msg && (
+               <div className='absolute w-full bottom-16  text-center p-3 bg-red-100 text-red-600 text-sm font-semibold'>
+                  {settingsError.msg}
+               </div>
+            )}
+      </div>
+   );
+};
+
+export default NotificationSettings;
--- a/components/settings/ScraperSettings.tsx
+++ b/components/settings/ScraperSettings.tsx
@@ -0,0 +1,149 @@
+import React from 'react';
+import { useClearFailedQueue } from '../../services/settings';
+import Icon from '../common/Icon';
+import SelectField, { SelectionOption } from '../common/SelectField';
+
+type ScraperSettingsProps = {
+   settings: SettingsType,
+   settingsError: null | {
+      type: string,
+      msg: string
+   },
+   updateSettings: Function,
+}
+
+const ScraperSettings = ({ settings, settingsError, updateSettings }:ScraperSettingsProps) => {
+   const { mutate: clearFailedMutate, isLoading: clearingQueue } = useClearFailedQueue(() => {});
+
+   const scrapingOptions: SelectionOption[] = [
+      { label: 'Daily', value: 'daily' },
+      { label: 'Every Other Day', value: 'other_day' },
+      { label: 'Weekly', value: 'weekly' },
+      { label: 'Monthly', value: 'monthly' },
+      { label: 'Never', value: 'never' },
+   ];
+   const delayOptions: SelectionOption[] = [
+      { label: 'No Delay', value: '0' },
+      { label: '5 Seconds', value: '5000' },
+      { label: '10 Seconds', value: '10000' },
+      { label: '30 Seconds', value: '30000' },
+      { label: '1 Minutes', value: '60000' },
+      { label: '2 Minutes', value: '120000' },
+      { label: '5 Minutes', value: '300000' },
+      { label: '10 Minutes', value: '600000' },
+      { label: '15 Minutes', value: '900000' },
+      { label: '30 Minutes', value: '1800000' },
+   ];
+   const allScrapers: SelectionOption[] = settings.available_scapers ? settings.available_scapers : [];
+   const scraperOptions: SelectionOption[] = [{ label: 'None', value: 'none' }, ...allScrapers];
+   const labelStyle = 'mb-2 font-semibold inline-block text-sm text-gray-700 capitalize';
+
+   return (
+      <div>
+      <div className='settings__content styled-scrollbar p-6 text-sm'>
+
+         <div className="settings__section__select mb-5">
+            <label className={labelStyle}>Scraping Method</label>
+            <SelectField
+            options={scraperOptions}
+            selected={[settings.scraper_type || 'none']}
+            defaultLabel="Select Scraper"
+            updateField={(updatedTime:[string]) => updateSettings('scraper_type', updatedTime[0])}
+            multiple={false}
+            rounded={'rounded'}
+            minWidth={270}
+            />
+         </div>
+         {['scrapingant', 'scrapingrobot', 'serply', 'serpapi', 'spaceSerp', 'searchapi'].includes(settings.scraper_type) && (
+            <div className="settings__section__input mr-3">
+               <label className={labelStyle}>Scraper API Key or Token</label>
+               <input
+                  className={`w-full p-2 border border-gray-200 rounded mt-2 mb-3 focus:outline-none  focus:border-blue-200 
+                  ${settingsError?.type === 'no_api_key' ? ' border-red-400 focus:border-red-400' : ''} `}
+                  type="text"
+                  value={settings?.scaping_api || ''}
+                  placeholder={'API Key/Token'}
+                  onChange={(event) => updateSettings('scaping_api', event.target.value)}
+               />
+            </div>
+         )}
+         {settings.scraper_type === 'proxy' && (
+            <div className="settings__section__input mb-5">
+               <label className={labelStyle}>Proxy List</label>
+               <textarea
+                  className={`w-full p-2 border border-gray-200 rounded mb-3 text-xs 
+                  focus:outline-none min-h-[160px] focus:border-blue-200 
+                  ${settingsError?.type === 'no_email' ? ' border-red-400 focus:border-red-400' : ''} `}
+                  value={settings?.proxy}
+                  placeholder={'http://122.123.22.45:5049\nhttps://user:password@122.123.22.45:5049'}
+                  onChange={(event) => updateSettings('proxy', event.target.value)}
+               />
+            </div>
+         )}
+         {settings.scraper_type !== 'none' && (
+            <div className="settings__section__input mb-5">
+               <label className={labelStyle}>Scraping Frequency</label>
+               <SelectField
+                  multiple={false}
+                  selected={[settings?.scrape_interval || 'daily']}
+                  options={scrapingOptions}
+                  defaultLabel={'Notification Settings'}
+                  updateField={(updated:string[]) => updated[0] && updateSettings('scrape_interval', updated[0])}
+                  rounded='rounded'
+                  maxHeight={48}
+                  minWidth={270}
+               />
+               <small className=' text-gray-500 pt-2 block'>This option requires Server/Docker Instance Restart to take Effect.</small>
+            </div>
+         )}
+            <div className="settings__section__input mb-5">
+               <label className={labelStyle}>Delay Between Each keyword Scrape</label>
+               <SelectField
+                  multiple={false}
+                  selected={[settings?.scrape_delay || '0']}
+                  options={delayOptions}
+                  defaultLabel={'Delay Settings'}
+                  updateField={(updated:string[]) => updated[0] && updateSettings('scrape_delay', updated[0])}
+                  rounded='rounded'
+                  maxHeight={48}
+                  minWidth={270}
+               />
+               <small className=' text-gray-500 pt-2 block'>This option requires Server/Docker Instance Restart to take Effect.</small>
+            </div>
+            <div className="settings__section__input mb-5">
+               <label className="relative inline-flex items-center cursor-pointer w-full justify-between">
+                  <span className="text-sm font-medium text-gray-900 dark:text-gray-300 w-56">Auto Retry Failed Keyword Scrape</span>
+                  <input
+                  type="checkbox"
+                  value={settings?.scrape_retry ? 'true' : '' }
+                  checked={settings.scrape_retry || false}
+                  className="sr-only peer"
+                  onChange={() => updateSettings('scrape_retry', !settings.scrape_retry)}
+                  />
+                  <div className="relative rounded-3xl w-9 h-5 bg-gray-200 peer-focus:outline-none peer-focus:ring-4
+                  peer-focus:ring-blue-300 dark:peer-focus:ring-blue-800rounded-full peer dark:bg-gray-700
+                  peer-checked:after:translate-x-full peer-checked:after:border-white after:content-['']
+                  after:absolute after:top-[2px] after:left-[2px] after:bg-white  after:border-gray-300
+                  after:border after:rounded-full after:h-4 after:w-4
+                  after:transition-all dark:border-gray-600 peer-checked:bg-blue-600"></div>
+
+               </label>
+            </div>
+            {settings?.scrape_retry && (settings.failed_queue?.length || 0) > 0 && (
+               <div className="settings__section__input mb-5">
+                  <label className={labelStyle}>Clear Failed Retry Queue</label>
+                  <button
+                  onClick={() => clearFailedMutate()}
+                  className=' py-3 px-5 w-full rounded cursor-pointer bg-gray-100 text-gray-800
+                  font-semibold text-sm hover:bg-gray-200'>
+                     {clearingQueue && <Icon type="loading" size={14} />} Clear Failed Queue
+                       ({settings.failed_queue?.length || 0} Keywords)
+                  </button>
+               </div>
+            )}
+      </div>
+   </div>
+   );
+};
+
+export default ScraperSettings;
--- a/components/settings/Settings.tsx
+++ b/components/settings/Settings.tsx
@@ -1,8 +1,9 @@
 import React, { useEffect, useState } from 'react';
-// import { useQuery } from 'react-query';
-import useUpdateSettings, { useFetchSettings } from '../../services/settings';
+import { Toaster } from 'react-hot-toast';
+import { useFetchSettings, useUpdateSettings } from '../../services/settings';
 import Icon from '../common/Icon';
-import SelectField, { SelectionOption } from '../common/SelectField';
+import NotificationSettings from './NotificationSettings';
+import ScraperSettings from './ScraperSettings';

 type SettingsProps = {
   closeSettings: Function,
@@ -16,6 +17,8 @@ type SettingsError = {

 const defaultSettings = {
   scraper_type: 'none',
+   scrape_delay: 'none',
+   scrape_retry: false,
   notification_interval: 'daily',
   notification_email: '',
   smtp_server: '',
@@ -57,7 +60,7 @@ const Settings = ({ closeSettings }:SettingsProps) => {
      if (e.target === e.currentTarget) { closeSettings(); }
   };

-   const updateSettings = (key: string, value:string|number) => {
+   const updateSettings = (key: string, value:string|number|boolean) => {
      setSettings({ ...settings, [key]: value });
   };

@@ -89,17 +92,6 @@ const Settings = ({ closeSettings }:SettingsProps) => {
      }
   };

-   const labelStyle = 'mb-2 font-semibold inline-block text-sm text-gray-700 capitalize';
-
-   const notificationOptions: SelectionOption[] = [
-      { label: 'Daily', value: 'daily' },
-      { label: 'Weekly', value: 'weekly' },
-      { label: 'Monthly', value: 'monthly' },
-      { label: 'Never', value: 'never' },
-   ];
-   const allScrapers: SelectionOption[] = settings.available_scapers ? settings.available_scapers : [];
-   const scraperOptions: SelectionOption[] = [{ label: 'None', value: 'none' }, ...allScrapers];
-
   const tabStyle = 'inline-block px-4 py-1 rounded-full mr-3 cursor-pointer text-sm';
   return (
       <div className="settings fixed w-full h-screen top-0 left-0 z-50" onClick={closeOnBGClick}>
@@ -127,141 +119,13 @@ const Settings = ({ closeSettings }:SettingsProps) => {
                     </li>
                  </ul>
               </div>
-               {currentTab === 'scraper' && (
-                  <div>
-                     <div className='settings__content styled-scrollbar p-6 text-sm'>
-
-                        <div className="settings__section__select mb-5">
-                           <label className={labelStyle}>Scraping Method</label>
-                           <SelectField
-                           options={scraperOptions}
-                           selected={[settings.scraper_type || 'none']}
-                           defaultLabel="Select Scraper"
-                           updateField={(updatedTime:[string]) => updateSettings('scraper_type', updatedTime[0])}
-                           multiple={false}
-                           rounded={'rounded'}
-                           minWidth={270}
-                           />
-                        </div>
-                        {['scrapingant', 'scrapingrobot', 'serply', 'serpapi'].includes(settings.scraper_type) && (
-                           <div className="settings__section__input mr-3">
-                              <label className={labelStyle}>Scraper API Key or Token</label>
-                              <input
-                                 className={`w-full p-2 border border-gray-200 rounded mt-2 mb-3 focus:outline-none  focus:border-blue-200 
-                                 ${settingsError && settingsError.type === 'no_api_key' ? ' border-red-400 focus:border-red-400' : ''} `}
-                                 type="text"
-                                 value={settings?.scaping_api || ''}
-                                 placeholder={'API Key/Token'}
-                                 onChange={(event) => updateSettings('scaping_api', event.target.value)}
-                              />
-                           </div>
-                        )}
-                        {settings.scraper_type === 'proxy' && (
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>Proxy List</label>
-                              <textarea
-                                 className={`w-full p-2 border border-gray-200 rounded mb-3 text-xs 
-                                 focus:outline-none min-h-[160px] focus:border-blue-200 
-                                 ${settingsError && settingsError.type === 'no_email' ? ' border-red-400 focus:border-red-400' : ''} `}
-                                 value={settings?.proxy}
-                                 placeholder={'http://122.123.22.45:5049\nhttps://user:password@122.123.22.45:5049'}
-                                 onChange={(event) => updateSettings('proxy', event.target.value)}
-                              />
-                           </div>
-                        )}
-                     </div>
-                  </div>
+               {currentTab === 'scraper' && settings && (
+                  <ScraperSettings settings={settings} updateSettings={updateSettings} settingsError={settingsError} />
               )}

-            {currentTab === 'notification' && (
-               <div>
-                  <div className='settings__content styled-scrollbar p-6 text-sm'>
-                     <div className="settings__section__input mb-5">
-                        <label className={labelStyle}>Notification Frequency</label>
-                        <SelectField
-                           multiple={false}
-                           selected={[settings.notification_interval]}
-                           options={notificationOptions}
-                           defaultLabel={'Notification Settings'}
-                           updateField={(updated:string[]) => updated[0] && updateSettings('notification_interval', updated[0])}
-                           rounded='rounded'
-                           maxHeight={48}
-                           minWidth={270}
-                        />
-                     </div>
-                     {settings.notification_interval !== 'never' && (
-                        <>
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>Notification Emails</label>
-                              <input
-                                 className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
-                                 ${settingsError && settingsError.type === 'no_email' ? ' border-red-400 focus:border-red-400' : ''} `}
-                                 type="text"
-                                 value={settings?.notification_email}
-                                 placeholder={'test@gmail.com'}
-                                 onChange={(event) => updateSettings('notification_email', event.target.value)}
-                              />
-                           </div>
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>SMTP Server</label>
-                              <input
-                                 className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
-                                 ${settingsError && settingsError.type === 'no_smtp_server' ? ' border-red-400 focus:border-red-400' : ''} `}
-                                 type="text"
-                                 value={settings?.smtp_server || ''}
-                                 onChange={(event) => updateSettings('smtp_server', event.target.value)}
-                              />
-                           </div>
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>SMTP Port</label>
-                              <input
-                                 className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
-                                 ${settingsError && settingsError.type === 'no_smtp_port' ? ' border-red-400 focus:border-red-400' : ''} `}
-                                 type="text"
-                                 value={settings?.smtp_port || ''}
-                                 onChange={(event) => updateSettings('smtp_port', event.target.value)}
-                              />
-                           </div>
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>SMTP Username</label>
-                              <input
-                                 className={'w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200'}
-                                 type="text"
-                                 value={settings?.smtp_username || ''}
-                                 onChange={(event) => updateSettings('smtp_username', event.target.value)}
-                              />
-                           </div>
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>SMTP Password</label>
-                              <input
-                                 className={'w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200'}
-                                 type="text"
-                                 value={settings?.smtp_password || ''}
-                                 onChange={(event) => updateSettings('smtp_password', event.target.value)}
-                              />
-                           </div>
-                           <div className="settings__section__input mb-5">
-                              <label className={labelStyle}>From Email Address</label>
-                              <input
-                                 className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none  focus:border-blue-200 
-                                 ${settingsError && settingsError.type === 'no_smtp_from' ? ' border-red-400 focus:border-red-400' : ''} `}
-                                 type="text"
-                                 value={settings?.notification_email_from || ''}
-                                 placeholder="no-reply@mydomain.com"
-                                 onChange={(event) => updateSettings('notification_email_from', event.target.value)}
-                              />
-                           </div>
-                        </>
-                     )}
-
-                     </div>
-                     {settingsError && (
-                        <div className='absolute w-full bottom-16  text-center p-3 bg-red-100 text-red-600 text-sm font-semibold'>
-                           {settingsError.msg}
-                        </div>
-                     )}
-               </div>
-            )}
+               {currentTab === 'notification' && settings && (
+                  <NotificationSettings settings={settings} updateSettings={updateSettings} settingsError={settingsError} />
+               )}
               <div className=' border-t-[1px] border-gray-200 p-2 px-3'>
                  <button
                  onClick={() => performUpdate()}
@@ -270,6 +134,7 @@ const Settings = ({ closeSettings }:SettingsProps) => {
                  </button>
               </div>
            </div>
+            <Toaster position='bottom-center' containerClassName="react_toaster" />
       </div>
   );
 };
--- a/cron.js
+++ b/cron.js
@@ -1,7 +1,7 @@
 const Cryptr = require('cryptr');
 const { promises } = require('fs');
 const { readFile } = require('fs');
-const cron = require('node-cron');
+const Cron = require('croner');
 require('dotenv').config({ path: './.env.local' });

 const getAppSettings = async () => {
@@ -49,11 +49,14 @@ const generateCronTime = (interval) => {
   if (interval === 'daily') {
      cronTime = '0 0 0 * * *';
   }
+   if (interval === 'other_day') {
+      cronTime = '0 0 2-30/2 * *';
+   }
   if (interval === 'daily_morning') {
      cronTime = '0 0 3 * * *';
   }
   if (interval === 'weekly') {
-      cronTime = '0 0 0 */7 * *';
+      cronTime = '0 0 * * 1';
   }
   if (interval === 'monthly') {
      cronTime = '0 0 1 * *'; // Run every first day of the month at 00:00(midnight)
@@ -63,66 +66,30 @@ const generateCronTime = (interval) => {
 };

 const runAppCronJobs = () => {
-   // RUN SERP Scraping CRON (EveryDay at Midnight) 0 0 0 * *
-   const scrapeCronTime = generateCronTime('daily');
-   cron.schedule(scrapeCronTime, () => {
-      // console.log('### Running Keyword Position Cron Job!');
-      const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
-      fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/cron`, fetchOpts)
-      .then((res) => res.json())
-      .then((data) => console.log(data))
-      .catch((err) => {
-         console.log('ERROR Making Daily Scraper Cron Request..');
-         console.log(err);
-      });
-   }, { scheduled: true });
-
-   // Run Failed scraping CRON (Every Hour)
-   const failedCronTime = generateCronTime('hourly');
-   cron.schedule(failedCronTime, () => {
-      // console.log('### Retrying Failed Scrapes...');
-
-      readFile(`${process.cwd()}/data/failed_queue.json`, { encoding: 'utf-8' }, (err, data) => {
-         if (data) {
-            const keywordsToRetry = data ? JSON.parse(data) : [];
-            if (keywordsToRetry.length > 0) {
-               const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
-               fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/refresh?id=${keywordsToRetry.join(',')}`, fetchOpts)
-               .then((res) => res.json())
-               .then((refreshedData) => console.log(refreshedData))
-               .catch((fetchErr) => {
-                  console.log('ERROR Making failed_queue Cron Request..');
-                  console.log(fetchErr);
-               });
-            }
-         } else {
-            console.log('ERROR Reading Failed Scrapes Queue File..', err);
-         }
-      });
-   }, { scheduled: true });
-
-   // Run Google Search Console Scraper Daily
-   if (process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL) {
-      const searchConsoleCRONTime = generateCronTime('daily');
-      cron.schedule(searchConsoleCRONTime, () => {
-         const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
-         fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/searchconsole`, fetchOpts)
-         .then((res) => res.json())
-         .then((data) => console.log(data))
-         .catch((err) => {
-            console.log('ERROR Making Google Search Console Scraper Cron Request..');
-            console.log(err);
-         });
-      }, { scheduled: true });
-   }
-
-   // RUN Email Notification CRON
   getAppSettings().then((settings) => {
+      // RUN SERP Scraping CRON (EveryDay at Midnight) 0 0 0 * *
+      const scrape_interval = settings.scrape_interval || 'daily';
+      if (scrape_interval !== 'never') {
+         const scrapeCronTime = generateCronTime(scrape_interval);
+         Cron(scrapeCronTime, () => {
+            // console.log('### Running Keyword Position Cron Job!');
+            const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
+            fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/cron`, fetchOpts)
+            .then((res) => res.json())
+            // .then((data) =>{ console.log(data)})
+            .catch((err) => {
+               console.log('ERROR Making SERP Scraper Cron Request..');
+               console.log(err);
+            });
+         }, { scheduled: true });
+      }
+
+      // RUN Email Notification CRON
      const notif_interval = (!settings.notification_interval || settings.notification_interval === 'never') ? false : settings.notification_interval;
      if (notif_interval) {
         const cronTime = generateCronTime(notif_interval === 'daily' ? 'daily_morning' : notif_interval);
         if (cronTime) {
-            cron.schedule(cronTime, () => {
+            Cron(cronTime, () => {
               // console.log('### Sending Notification Email...');
               const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
               fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/notify`, fetchOpts)
@@ -136,6 +103,49 @@ const runAppCronJobs = () => {
         }
      }
   });
+
+   // Run Failed scraping CRON (Every Hour)
+   const failedCronTime = generateCronTime('hourly');
+   Cron(failedCronTime, () => {
+      // console.log('### Retrying Failed Scrapes...');
+
+      readFile(`${process.cwd()}/data/failed_queue.json`, { encoding: 'utf-8' }, (err, data) => {
+         if (data) {
+            try {
+               const keywordsToRetry = data ? JSON.parse(data) : [];
+               if (keywordsToRetry.length > 0) {
+                  const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
+                  fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/refresh?id=${keywordsToRetry.join(',')}`, fetchOpts)
+                  .then((res) => res.json())
+                  .then((refreshedData) => console.log(refreshedData))
+                  .catch((fetchErr) => {
+                     console.log('ERROR Making failed_queue Cron Request..');
+                     console.log(fetchErr);
+                  });
+               }
+            } catch (error) {
+               console.log('ERROR Reading Failed Scrapes Queue File..', error);
+            }
+         } else {
+            console.log('ERROR Reading Failed Scrapes Queue File..', err);
+         }
+      });
+   }, { scheduled: true });
+
+   // Run Google Search Console Scraper Daily
+   if (process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL) {
+      const searchConsoleCRONTime = generateCronTime('daily');
+      Cron(searchConsoleCRONTime, () => {
+         const fetchOpts = { method: 'POST', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
+         fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/searchconsole`, fetchOpts)
+         .then((res) => res.json())
+         .then((data) => console.log(data))
+         .catch((err) => {
+            console.log('ERROR Making Google Search Console Scraper Cron Request..');
+            console.log(err);
+         });
+      }, { scheduled: true });
+   }
 };

 runAppCronJobs();
--- a/database/database.ts
+++ b/database/database.ts
@@ -6,7 +6,7 @@ import Keyword from './models/keyword';
 const connection = new Sequelize({
   dialect: 'sqlite',
   host: '0.0.0.0',
-   username: process.env.USER,
+   username: process.env.USER_NAME ? process.env.USER_NAME : process.env.USER,
   password: process.env.PASSWORD,
   database: 'sequelize',
   dialectModule: sqlite3,
--- a/jest.setup.js
+++ b/jest.setup.js
@@ -1,11 +1,26 @@
 // eslint-disable-next-line no-unused-vars
 import 'isomorphic-fetch';
 import './styles/globals.css';
+import '@testing-library/jest-dom';
+import { enableFetchMocks } from 'jest-fetch-mock';
 // Optional: configure or set up a testing framework before each test.
 // If you delete this file, remove `setupFilesAfterEnv` from `jest.config.js`

 // Used for __tests__/testing-library.js
 // Learn more: https://github.com/testing-library/jest-dom
-import '@testing-library/jest-dom/extend-expect';
+
+window.matchMedia = (query) => ({
+   matches: false,
+   media: query,
+   onchange: null,
+   addListener: jest.fn(), // deprecated
+   removeListener: jest.fn(), // deprecated
+   addEventListener: jest.fn(),
+   removeEventListener: jest.fn(),
+   dispatchEvent: jest.fn(),
+});

 global.ResizeObserver = require('resize-observer-polyfill');
+
+// Enable Fetch Mocking
+enableFetchMocks();
--- a/next.config.js
+++ b/next.config.js
@@ -1,8 +1,13 @@
 /** @type {import('next').NextConfig} */
+const { version } = require('./package.json');
+
 const nextConfig = {
  reactStrictMode: true,
  swcMinify: false,
  output: 'standalone',
+  publicRuntimeConfig: {
+   version,
+ },
 };

 module.exports = nextConfig;
--- a/package-lock.json
+++ b/package-lock.json
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
 {
  "name": "serpbear",
-  "version": "0.2.2",
+  "version": "0.3.3",
  "private": true,
  "scripts": {
    "dev": "next dev",
@@ -17,7 +17,6 @@
  },
  "dependencies": {
    "@googleapis/searchconsole": "^1.0.0",
-    "@testing-library/react": "^13.4.0",
    "@types/react-transition-group": "^4.4.5",
    "axios": "^1.1.3",
    "axios-retry": "^3.3.1",
@@ -25,15 +24,15 @@
    "cheerio": "^1.0.0-rc.12",
    "concurrently": "^7.6.0",
    "cookies": "^0.8.0",
+    "croner": "^5.3.5",
    "cryptr": "^6.0.3",
    "dayjs": "^1.11.5",
    "dotenv": "^16.0.3",
    "https-proxy-agent": "^5.0.1",
    "isomorphic-fetch": "^3.0.0",
-    "jsonwebtoken": "^8.5.1",
+    "jsonwebtoken": "^9.0.2",
    "msw": "^0.49.0",
-    "next": "12.3.1",
-    "node-cron": "^3.0.2",
+    "next": "^12.3.4",
    "nodemailer": "^6.8.0",
    "react": "18.2.0",
    "react-chartjs-2": "^4.3.1",
@@ -44,15 +43,17 @@
    "react-transition-group": "^4.4.5",
    "react-window": "^1.8.8",
    "reflect-metadata": "^0.1.13",
-    "sequelize": "^6.25.2",
+    "sequelize": "^6.34.0",
    "sequelize-typescript": "^2.1.5",
-    "sqlite3": "^5.1.2"
+    "sqlite3": "^5.1.6"
  },
  "devDependencies": {
-    "@testing-library/jest-dom": "^5.16.5",
+    "@testing-library/jest-dom": "^6.1.4",
+    "@testing-library/react": "^14.0.0",
    "@types/cookies": "^0.7.7",
    "@types/cryptr": "^4.0.1",
    "@types/isomorphic-fetch": "^0.0.36",
+    "@types/jest": "^29.5.8",
    "@types/jsonwebtoken": "^8.5.9",
    "@types/node": "18.11.0",
    "@types/nodemailer": "^6.4.6",
@@ -64,9 +65,11 @@
    "eslint": "8.25.0",
    "eslint-config-airbnb-base": "^15.0.0",
    "eslint-config-next": "12.3.1",
-    "jest": "^29.3.1",
-    "jest-environment-jsdom": "^29.3.1",
-    "postcss": "^8.4.18",
+    "jest": "^29.7.0",
+    "jest-environment-jsdom": "^29.7.0",
+    "jest-fetch-mock": "^3.0.3",
+    "next-router-mock": "^0.9.10",
+    "postcss": "^8.4.31",
    "prettier": "^2.7.1",
    "resize-observer-polyfill": "^1.5.1",
    "sass": "^1.55.0",
--- a/pages/api/clearfailed.ts
+++ b/pages/api/clearfailed.ts
@@ -0,0 +1,29 @@
+import { writeFile } from 'fs/promises';
+import type { NextApiRequest, NextApiResponse } from 'next';
+import verifyUser from '../../utils/verifyUser';
+
+type SettingsGetResponse = {
+   cleared?: boolean,
+   error?: string,
+}
+
+export default async function handler(req: NextApiRequest, res: NextApiResponse) {
+   const authorized = verifyUser(req, res);
+   if (authorized !== 'authorized') {
+      return res.status(401).json({ error: authorized });
+   }
+   if (req.method === 'PUT') {
+      return clearFailedQueue(req, res);
+   }
+   return res.status(502).json({ error: 'Unrecognized Route.' });
+}
+
+const clearFailedQueue = async (req: NextApiRequest, res: NextApiResponse<SettingsGetResponse>) => {
+   try {
+      await writeFile(`${process.cwd()}/data/failed_queue.json`, JSON.stringify([]), { encoding: 'utf-8' });
+      return res.status(200).json({ cleared: true });
+   } catch (error) {
+      console.log('[ERROR] Cleraring Failed Queue File.', error);
+      return res.status(200).json({ error: 'Error Cleraring Failed Queue!' });
+   }
+};
--- a/pages/api/cron.ts
+++ b/pages/api/cron.ts
@@ -3,7 +3,7 @@ import db from '../../database/database';
 import Keyword from '../../database/models/keyword';
 import { getAppSettings } from './settings';
 import verifyUser from '../../utils/verifyUser';
-import { refreshAndUpdateKeywords } from './refresh';
+import refreshAndUpdateKeywords from '../../utils/refresh';

 type CRONRefreshRes = {
   started: boolean
--- a/pages/api/insight.ts
+++ b/pages/api/insight.ts
@@ -37,7 +37,10 @@ const getDomainSearchConsoleInsight = async (req: NextApiRequest, res: NextApiRe

   // First try and read the  Local SC Domain Data file.
   const localSCData = await readLocalSCData(domainname);
-   if (localSCData && localSCData.stats && localSCData.stats.length) {
+   const oldFetchedDate = localSCData.lastFetched;
+   const fetchTimeDiff = new Date().getTime() - (oldFetchedDate ? new Date(oldFetchedDate as string).getTime() : 0);
+
+   if (localSCData && localSCData.stats && localSCData.stats.length && fetchTimeDiff <= 86400000) {
      const response = getInsightFromSCData(localSCData);
      return res.status(200).json({ data: response });
   }
--- a/pages/api/keywords.ts
+++ b/pages/api/keywords.ts
@@ -2,11 +2,11 @@ import type { NextApiRequest, NextApiResponse } from 'next';
 import { Op } from 'sequelize';
 import db from '../../database/database';
 import Keyword from '../../database/models/keyword';
-import { refreshAndUpdateKeywords } from './refresh';
 import { getAppSettings } from './settings';
 import verifyUser from '../../utils/verifyUser';
 import parseKeywords from '../../utils/parseKeywords';
 import { integrateKeywordSCData, readLocalSCData } from '../../utils/searchConsole';
+import refreshAndUpdateKeywords from '../../utils/refresh';

 type KeywordsGetResponse = {
   keywords?: KeywordType[],
@@ -153,10 +153,13 @@ const updateKeywords = async (req: NextApiRequest, res: NextApiResponse<Keywords
      }
      if (tags) {
         const tagsKeywordIDs = Object.keys(tags);
+         const multipleKeywords = tagsKeywordIDs.length > 1;
         for (const keywordID of tagsKeywordIDs) {
-            const response = await Keyword.findOne({ where: { ID: keywordID } });
-            if (response) {
-               await response.update({ tags: JSON.stringify(tags[keywordID]) });
+            const selectedKeyword = await Keyword.findOne({ where: { ID: keywordID } });
+            const currentTags = selectedKeyword && selectedKeyword.tags ? JSON.parse(selectedKeyword.tags) : [];
+            const mergedTags = Array.from(new Set([...currentTags, ...tags[keywordID]]));
+            if (selectedKeyword) {
+               await selectedKeyword.update({ tags: JSON.stringify(multipleKeywords ? mergedTags : tags[keywordID]) });
            }
         }
         return res.status(200).json({ keywords });
--- a/pages/api/login.ts
+++ b/pages/api/login.ts
@@ -18,10 +18,11 @@ const loginUser = async (req: NextApiRequest, res: NextApiResponse<loginResponse
   if (!req.body.username || !req.body.password) {
      return res.status(401).json({ error: 'Username Password Missing' });
   }
+   const userName = process.env.USER_NAME ? process.env.USER_NAME : process.env.USER;

-   if (req.body.username === process.env.USER
+   if (req.body.username === userName
      && req.body.password === process.env.PASSWORD && process.env.SECRET) {
-      const token = jwt.sign({ user: process.env.USER }, process.env.SECRET);
+      const token = jwt.sign({ user: userName }, process.env.SECRET);
      const cookies = new Cookies(req, res);
      const expireDate = new Date();
      const sessDuration = process.env.SESSION_DURATION;
@@ -30,7 +31,7 @@ const loginUser = async (req: NextApiRequest, res: NextApiResponse<loginResponse
      return res.status(200).json({ success: true, error: null });
   }

-   const error = req.body.username !== process.env.USER ? 'Incorrect Username' : 'Incorrect Password';
+   const error = req.body.username !== userName ? 'Incorrect Username' : 'Incorrect Password';

   return res.status(401).json({ success: false, error });
 };
--- a/pages/api/refresh.ts
+++ b/pages/api/refresh.ts
@@ -2,11 +2,10 @@ import type { NextApiRequest, NextApiResponse } from 'next';
 import { Op } from 'sequelize';
 import db from '../../database/database';
 import Keyword from '../../database/models/keyword';
-import refreshKeywords from '../../utils/refresh';
+import refreshAndUpdateKeywords from '../../utils/refresh';
 import { getAppSettings } from './settings';
 import verifyUser from '../../utils/verifyUser';
 import parseKeywords from '../../utils/parseKeywords';
-import { removeFromRetryQueue, retryScrape } from '../../utils/scraper';

 type KeywordsRefreshRes = {
   keywords?: KeywordType[]
@@ -63,57 +62,3 @@ const refresTheKeywords = async (req: NextApiRequest, res: NextApiResponse<Keywo
      return res.status(400).json({ error: 'Error refreshing keywords!' });
   }
 };
-
-export const refreshAndUpdateKeywords = async (initKeywords:Keyword[], settings:SettingsType) => {
-   const formattedKeywords = initKeywords.map((el) => el.get({ plain: true }));
-   const refreshed: any = await refreshKeywords(formattedKeywords, settings);
-   // const fetchKeywords = await refreshKeywords(initialKeywords.map( k=> k.keyword ));
-   const updatedKeywords: KeywordType[] = [];
-
-   for (const keywordRaw of initKeywords) {
-      const keywordPrased = parseKeywords([keywordRaw.get({ plain: true })]);
-      const keyword = keywordPrased[0];
-      const udpatedkeyword = refreshed.find((item:any) => item.ID && item.ID === keyword.ID);
-
-      if (udpatedkeyword && keyword) {
-         const newPos = udpatedkeyword.position;
-         const newPosition = newPos !== false ? newPos : keyword.position;
-         const { history } = keyword;
-         const theDate = new Date();
-         history[`${theDate.getFullYear()}-${theDate.getMonth() + 1}-${theDate.getDate()}`] = newPosition;
-
-         const updatedVal = {
-            position: newPosition,
-            updating: false,
-            url: udpatedkeyword.url,
-            lastResult: udpatedkeyword.result,
-            history,
-            lastUpdated: udpatedkeyword.error ? keyword.lastUpdated : theDate.toJSON(),
-            lastUpdateError: udpatedkeyword.error
-               ? JSON.stringify({ date: theDate.toJSON(), error: `${udpatedkeyword.error}`, scraper: settings.scraper_type })
-               : 'false',
-         };
-         updatedKeywords.push({ ...keyword, ...{ ...updatedVal, lastUpdateError: JSON.parse(updatedVal.lastUpdateError) } });
-
-         // If failed, Add to Retry Queue Cron
-         if (udpatedkeyword.error) {
-            await retryScrape(keyword.ID);
-         } else {
-            await removeFromRetryQueue(keyword.ID);
-         }
-
-         // Update the Keyword Position in Database
-         try {
-            await keywordRaw.update({
-               ...updatedVal,
-               lastResult: Array.isArray(udpatedkeyword.result) ? JSON.stringify(udpatedkeyword.result) : udpatedkeyword.result,
-               history: JSON.stringify(history),
-            });
-            console.log('[SUCCESS] Updating the Keyword: ', keyword.keyword);
-         } catch (error) {
-            console.log('[ERROR] Updating SERP for Keyword', keyword.keyword, error);
-         }
-      }
-   }
-   return updatedKeywords;
-};
--- a/pages/api/settings.ts
+++ b/pages/api/settings.ts
@@ -1,6 +1,7 @@
+import { writeFile, readFile } from 'fs/promises';
 import type { NextApiRequest, NextApiResponse } from 'next';
 import Cryptr from 'cryptr';
-import { writeFile, readFile } from 'fs/promises';
+import getConfig from 'next/config';
 import verifyUser from '../../utils/verifyUser';
 import allScrapers from '../../scrapers/index';

@@ -26,7 +27,9 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
 const getSettings = async (req: NextApiRequest, res: NextApiResponse<SettingsGetResponse>) => {
   const settings = await getAppSettings();
   if (settings) {
-      return res.status(200).json({ settings });
+      const { publicRuntimeConfig } = getConfig();
+      const version = publicRuntimeConfig?.version;
+      return res.status(200).json({ settings: { ...settings, version } });
   }
   return res.status(400).json({ error: 'Error Loading Settings!' });
 };
@@ -52,8 +55,11 @@ const updateSettings = async (req: NextApiRequest, res: NextApiResponse<Settings
 };

 export const getAppSettings = async () : Promise<SettingsType> => {
+   const screenshotAPIKey = process.env.SCREENSHOT_API || '69408-serpbear';
   try {
      const settingsRaw = await readFile(`${process.cwd()}/data/settings.json`, { encoding: 'utf-8' });
+      const failedQueueRaw = await readFile(`${process.cwd()}/data/failed_queue.json`, { encoding: 'utf-8' });
+      const failedQueue: string[] = failedQueueRaw ? JSON.parse(failedQueueRaw) : [];
      const settings: SettingsType = settingsRaw ? JSON.parse(settingsRaw) : {};
      let decryptedSettings = settings;

@@ -67,6 +73,8 @@ export const getAppSettings = async () : Promise<SettingsType> => {
            smtp_password,
            search_console_integrated: !!(process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL),
            available_scapers: allScrapers.map((scraper) => ({ label: scraper.name, value: scraper.id })),
+            failed_queue: failedQueue,
+            screenshot_key: screenshotAPIKey,
         };
      } catch (error) {
         console.log('Error Decrypting Settings API Keys!');
@@ -75,7 +83,7 @@ export const getAppSettings = async () : Promise<SettingsType> => {
      return decryptedSettings;
   } catch (error) {
      console.log('[ERROR] Getting App Settings. ', error);
-      const settings = {
+      const settings: SettingsType = {
         scraper_type: 'none',
         notification_interval: 'never',
         notification_email: '',
@@ -84,8 +92,15 @@ export const getAppSettings = async () : Promise<SettingsType> => {
         smtp_port: '',
         smtp_username: '',
         smtp_password: '',
+         scrape_retry: false,
+         screenshot_key: screenshotAPIKey,
+      };
+      const otherSettings = {
+         available_scapers: allScrapers.map((scraper) => ({ label: scraper.name, value: scraper.id })),
+         failed_queue: [],
      };
      await writeFile(`${process.cwd()}/data/settings.json`, JSON.stringify(settings), { encoding: 'utf-8' });
-      return settings;
+      await writeFile(`${process.cwd()}/data/failed_queue.json`, JSON.stringify([]), { encoding: 'utf-8' });
+      return { ...settings, ...otherSettings };
   }
 };
--- a/pages/domains/index.tsx
+++ b/pages/domains/index.tsx
@@ -1,28 +1,51 @@
-import React, { useEffect, useState } from 'react';
+import React, { useEffect, useMemo, useState } from 'react';
 import type { NextPage } from 'next';
 import Head from 'next/head';
 import { useRouter } from 'next/router';
 import { CSSTransition } from 'react-transition-group';
+import toast, { Toaster } from 'react-hot-toast';
 import TopBar from '../../components/common/TopBar';
 import AddDomain from '../../components/domains/AddDomain';
 import Settings from '../../components/settings/Settings';
 import { useFetchSettings } from '../../services/settings';
-import { useFetchDomains } from '../../services/domains';
+import { fetchDomainScreenshot, useFetchDomains } from '../../services/domains';
 import DomainItem from '../../components/domains/DomainItem';
 import Icon from '../../components/common/Icon';

-const SingleDomain: NextPage = () => {
+type thumbImages = { [domain:string] : string }
+
+const Domains: NextPage = () => {
   const router = useRouter();
   const [noScrapprtError, setNoScrapprtError] = useState(false);
   const [showSettings, setShowSettings] = useState(false);
   const [showAddDomain, setShowAddDomain] = useState(false);
+   const [domainThumbs, setDomainThumbs] = useState<thumbImages>({});
   const { data: appSettings } = useFetchSettings();
   const { data: domainsData, isLoading } = useFetchDomains(router, true);

-   useEffect(() => {
-      console.log('Domains Data: ', domainsData);
+   const totalKeywords = useMemo(() => {
+      let keywords = 0;
+      if (domainsData?.domains) {
+         domainsData.domains.forEach(async (domain:DomainType) => {
+            keywords += domain?.keywordCount || 0;
+         });
+      }
+      return keywords;
   }, [domainsData]);

+   useEffect(() => {
+      if (domainsData?.domains && domainsData.domains.length > 0 && appSettings?.settings?.screenshot_key) {
+         domainsData.domains.forEach(async (domain:DomainType) => {
+            if (domain.domain) {
+               const domainThumb = await fetchDomainScreenshot(domain.domain, appSettings.settings.screenshot_key);
+               if (domainThumb) {
+                  setDomainThumbs((currentThumbs) => ({ ...currentThumbs, [domain.domain]: domainThumb }));
+               }
+            }
+         });
+      }
+   }, [domainsData, appSettings]);
+
   useEffect(() => {
      // console.log('appSettings.settings: ', appSettings && appSettings.settings);
      if (appSettings && appSettings.settings && (!appSettings.settings.scraper_type || (appSettings.settings.scraper_type === 'none'))) {
@@ -30,8 +53,20 @@ const SingleDomain: NextPage = () => {
      }
   }, [appSettings]);

+   const manuallyUpdateThumb = async (domain: string) => {
+      if (domain && appSettings?.settings?.screenshot_key) {
+         const domainThumb = await fetchDomainScreenshot(domain, appSettings.settings.screenshot_key, true);
+         if (domainThumb) {
+            toast(`${domain} Screenshot Updated Successfully!`, { icon: '✔️' });
+            setDomainThumbs((currentThumbs) => ({ ...currentThumbs, [domain]: domainThumb }));
+         } else {
+            toast(`Failed to Fetch ${domain} Screenshot!`, { icon: '⚠️' });
+         }
+      }
+   };
+
   return (
-      <div className="Domain ">
+      <div data-testid="domains" className="Domain flex flex-col min-h-screen">
         {noScrapprtError && (
               <div className=' p-3 bg-red-600 text-white text-sm text-center'>
                  A Scrapper/Proxy has not been set up Yet. Open Settings to set it up and start using the app.
@@ -44,9 +79,12 @@ const SingleDomain: NextPage = () => {

         <div className="flex flex-col w-full max-w-5xl mx-auto p-6 lg:mt-24 lg:p-0">
            <div className='flex justify-between mb-2 items-center'>
-               <div className=' text-sm'>{domainsData?.domains?.length || 0} Domains</div>
+               <div className=' text-sm text-gray-600'>
+                  {domainsData?.domains?.length || 0} Domains <span className=' text-gray-300 ml-1 mr-1'>|</span> {totalKeywords} keywords
+               </div>
               <div>
                  <button
+                  data-testid="addDomainButton"
                  className={'ml-2 inline-block py-2 text-blue-700 font-bold text-sm'}
                  onClick={() => setShowAddDomain(true)}>
                     <span
@@ -62,6 +100,8 @@ const SingleDomain: NextPage = () => {
                           domain={domain}
                           selected={false}
                           isConsoleIntegrated={!!(appSettings && appSettings?.settings?.search_console_integrated) }
+                           thumb={domainThumbs[domain.domain]}
+                           updateThumb={manuallyUpdateThumb}
                           // isConsoleIntegrated={false}
                           />;
               })}
@@ -84,8 +124,12 @@ const SingleDomain: NextPage = () => {
         <CSSTransition in={showSettings} timeout={300} classNames="settings_anim" unmountOnExit mountOnEnter>
             <Settings closeSettings={() => setShowSettings(false)} />
         </CSSTransition>
+         <footer className='text-center flex flex-1 justify-center pb-5 items-end'>
+            <span className='text-gray-500 text-xs'><a href='https://github.com/towfiqi/serpbear' target="_blank" rel='noreferrer'>SerpBear v{appSettings?.settings?.version || '0.0.0'}</a></span>
+         </footer>
+         <Toaster position='bottom-center' containerClassName="react_toaster" />
      </div>
   );
 };

-export default SingleDomain;
+export default Domains;
--- a/pages/index.tsx
+++ b/pages/index.tsx
@@ -8,7 +8,7 @@ import Icon from '../components/common/Icon';
 const Home: NextPage = () => {
   const router = useRouter();
   useEffect(() => {
-      router.push('/domains');
+      if (router) router.push('/domains');
   }, [router]);

  return (
--- a/scrapers/index.ts
+++ b/scrapers/index.ts
@@ -2,12 +2,16 @@ import scrapingAnt from './services/scrapingant';
 import scrapingRobot from './services/scrapingrobot';
 import serpapi from './services/serpapi';
 import serply from './services/serply';
+import spaceserp from './services/spaceserp';
 import proxy from './services/proxy';
+import searchapi from './services/searchapi';

 export default [
   scrapingRobot,
   scrapingAnt,
   serpapi,
   serply,
+   spaceserp,
   proxy,
+   searchapi,
 ];
--- a/scrapers/services/scrapingant.ts
+++ b/scrapers/services/scrapingant.ts
@@ -2,11 +2,17 @@ const scrapingAnt:ScraperSettings = {
   id: 'scrapingant',
   name: 'ScrapingAnt',
   website: 'scrapingant.com',
+   headers: (keyword) => {
+      // eslint-disable-next-line max-len
+      const mobileAgent = 'Mozilla/5.0 (Linux; Android 10; SM-G996U Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36';
+      return keyword && keyword.device === 'mobile' ? { 'Ant-User-Agent': mobileAgent } : {};
+   },
   scrapeURL: (keyword, settings, countryData) => {
      const scraperCountries = ['AE', 'BR', 'CN', 'DE', 'ES', 'FR', 'GB', 'HK', 'PL', 'IN', 'IT', 'IL', 'JP', 'NL', 'RU', 'SA', 'US', 'CZ'];
      const country = scraperCountries.includes(keyword.country.toUpperCase()) ? keyword.country : 'US';
      const lang = countryData[country][2];
-      return `https://api.scrapingant.com/v2/extended?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fnum%3D100%26hl%3D${lang}%26q%3D${encodeURI(keyword.keyword)}&x-api-key=${settings.scaping_api}&proxy_country=${country}&browser=false`;
+      const url = encodeURI(`https://www.google.com/search?num=100&hl=${lang}&q=${keyword.keyword}`);
+      return `https://api.scrapingant.com/v2/extended?url=${url}&x-api-key=${settings.scaping_api}&proxy_country=${country}&browser=false`;
   },
   resultObjectKey: 'result',
 };
--- a/scrapers/services/scrapingrobot.ts
+++ b/scrapers/services/scrapingrobot.ts
@@ -4,8 +4,10 @@ const scrapingRobot:ScraperSettings = {
   website: 'scrapingrobot.com',
   scrapeURL: (keyword, settings, countryData) => {
      const country = keyword.country || 'US';
+      const device = keyword.device === 'mobile' ? '&mobile=true' : '';
      const lang = countryData[country][2];
-      return `https://api.scrapingrobot.com/?token=${settings.scaping_api}&proxyCountry=${country}&render=false${keyword.device === 'mobile' ? '&mobile=true' : ''}&url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fnum%3D100%26hl%3D${lang}%26q%3D${encodeURI(keyword.keyword)}`;
+      const url = encodeURI(`https://www.google.com/search?num=100&hl=${lang}&q=${keyword.keyword}`);
+      return `https://api.scrapingrobot.com/?token=${settings.scaping_api}&proxyCountry=${country}&render=false${device}&url=${url}`;
   },
   resultObjectKey: 'result',
 };
--- a/scrapers/services/searchapi.ts
+++ b/scrapers/services/searchapi.ts
@@ -0,0 +1,38 @@
+const searchapi:ScraperSettings = {
+  id: 'searchapi',
+  name: 'SearchApi.io',
+  website: 'searchapi.io',
+  headers: (keyword, settings) => {
+     return {
+        'Content-Type': 'application/json',
+        Authorization: `Bearer ${settings.scaping_api}`,
+     };
+  },
+  scrapeURL: (keyword) => {
+     return `https://www.searchapi.io/api/v1/search?engine=google&q=${encodeURI(keyword.keyword)}&num=100&gl=${keyword.country}&device=${keyword.device}`;
+  },
+  resultObjectKey: 'organic_results',
+  serpExtractor: (content) => {
+     const extractedResult = [];
+     const results: SearchApiResult[] = (typeof content === 'string') ? JSON.parse(content) : content as SearchApiResult[];
+
+     for (const { link, title, position } of results) {
+        if (title && link) {
+           extractedResult.push({
+              title,
+              url: link,
+              position,
+           });
+        }
+     }
+     return extractedResult;
+  },
+};
+
+interface SearchApiResult {
+  title: string,
+  link: string,
+  position: number,
+}
+
+export default searchapi;
--- a/scrapers/services/spaceserp.ts
+++ b/scrapers/services/spaceserp.ts
@@ -0,0 +1,34 @@
+interface SpaceSerpResult {
+   title: string,
+   link: string,
+   domain: string,
+   position: number
+}
+
+const spaceSerp:ScraperSettings = {
+   id: 'spaceSerp',
+   name: 'Space Serp',
+   website: 'spaceserp.com',
+   scrapeURL: (keyword, settings, countryData) => {
+      const country = keyword.country || 'US';
+      const lang = countryData[country][2];
+      return `https://api.spaceserp.com/google/search?apiKey=${settings.scaping_api}&q=${encodeURI(keyword.keyword)}&pageSize=100&gl=${country}&hl=${lang}${keyword.device === 'mobile' ? '&device=mobile' : ''}&resultBlocks=`;
+   },
+   resultObjectKey: 'organic_results',
+   serpExtractor: (content) => {
+      const extractedResult = [];
+      const results: SpaceSerpResult[] = (typeof content === 'string') ? JSON.parse(content) : content as SpaceSerpResult[];
+      for (const result of results) {
+         if (result.title && result.link) {
+            extractedResult.push({
+               title: result.title,
+               url: result.link,
+               position: result.position,
+            });
+         }
+      }
+      return extractedResult;
+   },
+};
+
+export default spaceSerp;
--- a/services/domains.tsx
+++ b/services/domains.tsx
@@ -19,6 +19,36 @@ export async function fetchDomains(router: NextRouter, withStats:boolean) {
   return res.json();
 }

+export async function fetchDomainScreenshot(domain: string, screenshot_key:string, forceFetch = false): Promise<string | false> {
+   const domainThumbsRaw = localStorage.getItem('domainThumbs');
+   const domThumbs = domainThumbsRaw ? JSON.parse(domainThumbsRaw) : {};
+   if (!domThumbs[domain] || forceFetch) {
+      try {
+         const screenshotURL = `https://image.thum.io/get/auth/${screenshot_key}/maxAge/96/width/200/https://${domain}`;
+         const domainImageRes = await fetch(screenshotURL);
+         const domainImageBlob = domainImageRes.status === 200 ? await domainImageRes.blob() : false;
+         if (domainImageBlob) {
+            const reader = new FileReader();
+            await new Promise((resolve, reject) => {
+               reader.onload = resolve;
+               reader.onerror = reject;
+               reader.readAsDataURL(domainImageBlob);
+            });
+            const imageBase: string = reader.result && typeof reader.result === 'string' ? reader.result : '';
+            localStorage.setItem('domainThumbs', JSON.stringify({ ...domThumbs, [domain]: imageBase }));
+            return imageBase;
+         }
+         return false;
+      } catch (error) {
+         return false;
+      }
+   } else if (domThumbs[domain]) {
+         return domThumbs[domain];
+   }
+
+   return false;
+}
+
 export function useFetchDomains(router: NextRouter, withStats:boolean = false) {
   return useQuery('domains', () => fetchDomains(router, withStats));
 }
--- a/services/settings.ts
+++ b/services/settings.ts
@@ -10,7 +10,7 @@ export function useFetchSettings() {
   return useQuery('settings', () => fetchSettings());
 }

-const useUpdateSettings = (onSuccess:Function|undefined) => {
+export const useUpdateSettings = (onSuccess:Function|undefined) => {
   const queryClient = useQueryClient();

   return useMutation(async (settings: SettingsType) => {
@@ -38,4 +38,25 @@ const useUpdateSettings = (onSuccess:Function|undefined) => {
   });
 };

-export default useUpdateSettings;
+export function useClearFailedQueue(onSuccess:Function) {
+   const queryClient = useQueryClient();
+   return useMutation(async () => {
+      const headers = new Headers({ 'Content-Type': 'application/json', Accept: 'application/json' });
+      const fetchOpts = { method: 'PUT', headers };
+      const res = await fetch(`${window.location.origin}/api/clearfailed`, fetchOpts);
+      if (res.status >= 400 && res.status < 600) {
+         throw new Error('Bad response from server');
+      }
+      return res.json();
+   }, {
+      onSuccess: async () => {
+         onSuccess();
+         toast('Failed Queue Cleared', { icon: '✔️' });
+         queryClient.invalidateQueries(['settings']);
+      },
+      onError: () => {
+         console.log('Error Clearing Failed Queue!!!');
+         toast('Error Clearing Failed Queue.', { icon: '⚠️' });
+      },
+   });
+}
--- a/styles/globals.css
+++ b/styles/globals.css
@@ -84,8 +84,8 @@ body {

 .domKeywords_head--alpha_desc .domKeywords_head_keyword::after,
 .domKeywords_head--pos_desc .domKeywords_head_position::after,
-.domKeywords_head--imp_desc .domKeywords_head_imp::after,
-.domKeywords_head--visits_desc .domKeywords_head_visits::after,
+.domKeywords_head--imp_asc .domKeywords_head_imp::after,
+.domKeywords_head--visits_asc .domKeywords_head_visits::after,
 .domKeywords_head--ctr_desc .domKeywords_head_ctr::after {
   content: "↓";
   display: inline-block;
@@ -98,8 +98,8 @@ body {

 .domKeywords_head--alpha_asc .domKeywords_head_keyword::after,
 .domKeywords_head--pos_asc .domKeywords_head_position::after,
-.domKeywords_head--imp_asc .domKeywords_head_imp::after,
-.domKeywords_head--visits_asc .domKeywords_head_visits::after,
+.domKeywords_head--imp_desc .domKeywords_head_imp::after,
+.domKeywords_head--visits_desc .domKeywords_head_visits::after,
 .domKeywords_head--ctr_asc .domKeywords_head_ctr::after {
   content: "↑";
   display: inline-block;
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -24,7 +24,8 @@
      "next-env.d.ts",
      "**/*.ts",
      "**/*.tsx",
-      "types.d.ts"
+      "types.d.ts",
+      "./jest.setup.js"
   ],
   "exclude": [
      "node_modules"
--- a/types.d.ts
+++ b/types.d.ts
@@ -78,7 +78,13 @@ type SettingsType = {
   smtp_username?: string,
   smtp_password?: string,
   search_console_integrated?: boolean,
-   available_scapers?: Array
+   available_scapers?: Array,
+   scrape_interval?: string,
+   scrape_delay?: string,
+   scrape_retry?: boolean,
+   failed_queue?: string[]
+   version?: string,
+   screenshot_key?: string,
 }

 type KeywordSCDataChild = {
--- a/utils/SCsortFilter.ts
+++ b/utils/SCsortFilter.ts
@@ -9,16 +9,16 @@ export const SCsortKeywords = (theKeywords:SCKeywordType[], sortBy:string) : SCK
   const keywords = theKeywords.map((k) => ({ ...k, position: k.position === 0 ? 111 : k.position }));
   switch (sortBy) {
      case 'imp_asc':
-            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => b.impressions - a.impressions);
+            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => (a.impressions > b.impressions ? 1 : -1));
            break;
      case 'imp_desc':
-            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => a.impressions - b.impressions);
+            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => (b.impressions > a.impressions ? 1 : -1));
            break;
      case 'visits_asc':
-            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => b.clicks - a.clicks);
+            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => (a.clicks > b.clicks ? 1 : -1));
            break;
      case 'visits_desc':
-            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => a.clicks - b.clicks);
+            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => (b.clicks > a.clicks ? 1 : -1));
            break;
       case 'ctr_asc':
            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => b.ctr - a.ctr);
@@ -27,17 +27,17 @@ export const SCsortKeywords = (theKeywords:SCKeywordType[], sortBy:string) : SCK
            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => a.ctr - b.ctr);
            break;
      case 'pos_asc':
-            sortedItems = keywords.sort((a: SCKeywordType, b: SCKeywordType) => (b.position > a.position ? 1 : -1));
+            sortedItems = keywords.sort((a: SCKeywordType, b: SCKeywordType) => (b.position < a.position ? 1 : -1));
            sortedItems = sortedItems.map((k) => ({ ...k, position: k.position === 111 ? 0 : k.position }));
            break;
      case 'pos_desc':
-            sortedItems = keywords.sort((a: SCKeywordType, b: SCKeywordType) => (a.position > b.position ? 1 : -1));
+            sortedItems = keywords.sort((a: SCKeywordType, b: SCKeywordType) => (a.position < b.position ? 1 : -1));
            sortedItems = sortedItems.map((k) => ({ ...k, position: k.position === 111 ? 0 : k.position }));
            break;
-      case 'alpha_asc':
+      case 'alpha_desc':
            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => (b.keyword > a.keyword ? 1 : -1));
            break;
-      case 'alpha_desc':
+      case 'alpha_asc':
            sortedItems = theKeywords.sort((a: SCKeywordType, b: SCKeywordType) => (a.keyword > b.keyword ? 1 : -1));
         break;
      default:
--- a/utils/refresh.ts
+++ b/utils/refresh.ts
@@ -1,32 +1,117 @@
 import { performance } from 'perf_hooks';
-import { RefreshResult, scrapeKeywordFromGoogle } from './scraper';
+import { setTimeout as sleep } from 'timers/promises';
+import { RefreshResult, removeFromRetryQueue, retryScrape, scrapeKeywordFromGoogle } from './scraper';
+import parseKeywords from './parseKeywords';
+import Keyword from '../database/models/keyword';

 /**
 * Refreshes the Keywords position by Scraping Google Search Result by
 * Determining whether the keywords should be scraped in Parallel or not
- * @param {KeywordType[]} keywords - Keywords to scrape
+ * @param {Keyword[]} rawkeyword - Keywords to scrape
 * @param {SettingsType} settings - The App Settings that contain the Scraper settings
 * @returns {Promise}
 */
-const refreshKeywords = async (keywords:KeywordType[], settings:SettingsType): Promise<RefreshResult[]> => {
-   if (!keywords || keywords.length === 0) { return []; }
+const refreshAndUpdateKeywords = async (rawkeyword:Keyword[], settings:SettingsType): Promise<KeywordType[]> => {
+   const keywords:KeywordType[] = rawkeyword.map((el) => el.get({ plain: true }));
+   if (!rawkeyword || rawkeyword.length === 0) { return []; }
   const start = performance.now();
+   const updatedKeywords: KeywordType[] = [];

-   let refreshedResults: RefreshResult[] = [];
-
-   if (['scrapingant', 'serpapi'].includes(settings.scraper_type)) {
-      refreshedResults = await refreshParallel(keywords, settings);
+   if (['scrapingant', 'serpapi', 'searchapi'].includes(settings.scraper_type)) {
+      const refreshedResults = await refreshParallel(keywords, settings);
+      if (refreshedResults.length > 0) {
+         for (const keyword of rawkeyword) {
+            const refreshedkeywordData = refreshedResults.find((k) => k && k.ID === keyword.ID);
+            if (refreshedkeywordData) {
+               const updatedkeyword = await updateKeywordPosition(keyword, refreshedkeywordData, settings);
+               updatedKeywords.push(updatedkeyword);
+            }
+         }
+      }
   } else {
-      for (const keyword of keywords) {
+      for (const keyword of rawkeyword) {
         console.log('START SCRAPE: ', keyword.keyword);
-         const refreshedkeywordData = await scrapeKeywordFromGoogle(keyword, settings);
-         refreshedResults.push(refreshedkeywordData);
+         const updatedkeyword = await refreshAndUpdateKeyword(keyword, settings);
+         updatedKeywords.push(updatedkeyword);
+         if (keywords.length > 0 && settings.scrape_delay && settings.scrape_delay !== '0') {
+            await sleep(parseInt(settings.scrape_delay, 10));
+         }
      }
   }

   const end = performance.now();
   console.log(`time taken: ${end - start}ms`);
-   return refreshedResults;
+   return updatedKeywords;
+};
+
+/**
+ * Scrape Serp for given keyword and update the position in DB.
+ * @param {Keyword} keyword - Keywords to scrape
+ * @param {SettingsType} settings - The App Settings that contain the Scraper settings
+ * @returns {Promise<KeywordType>}
+ */
+const refreshAndUpdateKeyword = async (keyword: Keyword, settings: SettingsType): Promise<KeywordType> => {
+   const currentkeyword = keyword.get({ plain: true });
+   const refreshedkeywordData = await scrapeKeywordFromGoogle(currentkeyword, settings);
+   const updatedkeyword = refreshedkeywordData ? await updateKeywordPosition(keyword, refreshedkeywordData, settings) : currentkeyword;
+   return updatedkeyword;
+};
+
+/**
+ * Processes the scraped data for the given keyword and updates the keyword serp position in DB.
+ * @param {Keyword} keywordRaw - Keywords to Update
+ * @param {RefreshResult} udpatedkeyword - scraped Data for that Keyword
+ * @param {SettingsType} settings - The App Settings that contain the Scraper settings
+ * @returns {Promise<KeywordType>}
+ */
+export const updateKeywordPosition = async (keywordRaw:Keyword, udpatedkeyword: RefreshResult, settings: SettingsType): Promise<KeywordType> => {
+   const keywordPrased = parseKeywords([keywordRaw.get({ plain: true })]);
+      const keyword = keywordPrased[0];
+      // const udpatedkeyword = refreshed;
+      let updated = keyword;
+
+      if (udpatedkeyword && keyword) {
+         const newPos = udpatedkeyword.position;
+         const newPosition = newPos !== 0 ? newPos : keyword.position;
+         const { history } = keyword;
+         const theDate = new Date();
+         const dateKey = `${theDate.getFullYear()}-${theDate.getMonth() + 1}-${theDate.getDate()}`;
+         history[dateKey] = newPosition;
+
+         const updatedVal = {
+            position: newPosition,
+            updating: false,
+            url: udpatedkeyword.url,
+            lastResult: udpatedkeyword.result,
+            history,
+            lastUpdated: udpatedkeyword.error ? keyword.lastUpdated : theDate.toJSON(),
+            lastUpdateError: udpatedkeyword.error
+               ? JSON.stringify({ date: theDate.toJSON(), error: `${udpatedkeyword.error}`, scraper: settings.scraper_type })
+               : 'false',
+         };
+
+         // If failed, Add to Retry Queue Cron
+         if (udpatedkeyword.error && settings?.scrape_retry) {
+            await retryScrape(keyword.ID);
+         } else {
+            await removeFromRetryQueue(keyword.ID);
+         }
+
+         // Update the Keyword Position in Database
+         try {
+            await keywordRaw.update({
+               ...updatedVal,
+               lastResult: Array.isArray(udpatedkeyword.result) ? JSON.stringify(udpatedkeyword.result) : udpatedkeyword.result,
+               history: JSON.stringify(history),
+            });
+            console.log('[SUCCESS] Updating the Keyword: ', keyword.keyword);
+            updated = { ...keyword, ...updatedVal, lastUpdateError: JSON.parse(updatedVal.lastUpdateError) };
+         } catch (error) {
+            console.log('[ERROR] Updating SERP for Keyword', keyword.keyword, error);
+         }
+      }
+
+      return updated;
 };

 /**
@@ -49,4 +134,4 @@ const refreshParallel = async (keywords:KeywordType[], settings:SettingsType) :
   });
 };

-export default refreshKeywords;
+export default refreshAndUpdateKeywords;
--- a/utils/scraper.ts
+++ b/utils/scraper.ts
@@ -12,14 +12,14 @@ type SearchResult = {
 }

 type SERPObject = {
-   postion:number|boolean,
+   postion:number,
   url:string
 }

 export type RefreshResult = false | {
   ID: number,
   keyword: string,
-   position:number | boolean,
+   position:number,
   url: string,
   result: SearchResult[],
   error?: boolean | string
@@ -49,7 +49,7 @@ export const getScraperClient = (keyword:KeywordType, settings:SettingsType, scr
      // Set Scraper Header
      const scrapeHeaders = scraper.headers ? scraper.headers(keyword, settings) : null;
      const scraperAPIURL = scraper.scrapeURL ? scraper.scrapeURL(keyword, settings, countries) : null;
-      if (scrapeHeaders) {
+      if (scrapeHeaders && Object.keys(scrapeHeaders).length > 0) {
         Object.keys(scrapeHeaders).forEach((headerItemKey:string) => {
            headers[headerItemKey] = scrapeHeaders[headerItemKey as keyof object];
         });
@@ -114,7 +114,7 @@ export const scrapeKeywordFromGoogle = async (keyword:KeywordType, settings:Sett
      const scraperResult = scraperObj?.resultObjectKey && res[scraperObj.resultObjectKey] ? res[scraperObj.resultObjectKey] : '';
      const scrapeResult:string = (res.data || res.html || res.results || scraperResult || '');
      if (res && scrapeResult) {
-         const extracted = scraperObj?.serpExtractor ? scraperObj.serpExtractor(scrapeResult) : extractScrapedResult(scrapeResult);
+         const extracted = scraperObj?.serpExtractor ? scraperObj.serpExtractor(scrapeResult) : extractScrapedResult(scrapeResult, keyword.device);
         // await writeFile('result.txt', JSON.stringify(scrapeResult), { encoding: 'utf-8' }).catch((err) => { console.log(err); });
         const serp = getSerp(keyword.domain, extracted);
         refreshedResults = { ID: keyword.ID, keyword: keyword.keyword, position: serp.postion, url: serp.url, result: extracted, error: false };
@@ -141,9 +141,10 @@ export const scrapeKeywordFromGoogle = async (keyword:KeywordType, settings:Sett
 /**
 * Extracts the Google Search result as object array from the Google Search's HTML content
 * @param {string} content - scraped google search page html data.
+ * @param {string} device - The device of the keyword.
 * @returns {SearchResult[]}
 */
-export const extractScrapedResult = (content: string): SearchResult[] => {
+export const extractScrapedResult = (content: string, device: string): SearchResult[] => {
   const extractedResult = [];

   const $ = cheerio.load(content);
@@ -163,6 +164,24 @@ export const extractScrapedResult = (content: string): SearchResult[] => {
      }
  }

+  // Mobile Scraper
+  if (extractedResult.length === 0 && device === 'mobile') {
+      const items = $('body').find('#rso > div');
+      for (let i = 0; i < items.length; i += 1) {
+         const item = $(items[i]);
+         const linkDom = item.find('a[role="presentation"]');
+         if (linkDom) {
+            const url = linkDom.attr('href');
+            const titleDom = linkDom.find('[role="link"]');
+            const title = titleDom ? titleDom.text() : '';
+            if (title && url) {
+               lastPosition += 1;
+               extractedResult.push({ title, url, position: lastPosition });
+            }
+         }
+      }
+   }
+
  return extractedResult;
 };

@@ -173,7 +192,7 @@ export const extractScrapedResult = (content: string): SearchResult[] => {
 * @returns {SERPObject}
 */
 export const getSerp = (domain:string, result:SearchResult[]) : SERPObject => {
-   if (result.length === 0 || !domain) { return { postion: false, url: '' }; }
+   if (result.length === 0 || !domain) { return { postion: 0, url: '' }; }
   const foundItem = result.find((item) => {
      const itemDomain = item.url.replace('www.', '').match(/^(?:https?:)?(?:\/\/)?([^/?]+)/i);
      return itemDomain && itemDomain.includes(domain.replace('www.', ''));
@@ -188,15 +207,15 @@ export const getSerp = (domain:string, result:SearchResult[]) : SERPObject => {
 * @returns {void}
 */
 export const retryScrape = async (keywordID: number) : Promise<void> => {
-   if (!keywordID) { return; }
+   if (!keywordID && !Number.isInteger(keywordID)) { return; }
   let currentQueue: number[] = [];

   const filePath = `${process.cwd()}/data/failed_queue.json`;
   const currentQueueRaw = await readFile(filePath, { encoding: 'utf-8' }).catch((err) => { console.log(err); return '[]'; });
-   currentQueue = JSON.parse(currentQueueRaw);
+   currentQueue = currentQueueRaw ? JSON.parse(currentQueueRaw) : [];

   if (!currentQueue.includes(keywordID)) {
-      currentQueue.push(keywordID);
+      currentQueue.push(Math.abs(keywordID));
   }

   await writeFile(filePath, JSON.stringify(currentQueue), { encoding: 'utf-8' }).catch((err) => { console.log(err); return '[]'; });
@@ -208,13 +227,13 @@ export const retryScrape = async (keywordID: number) : Promise<void> => {
 * @returns {void}
 */
 export const removeFromRetryQueue = async (keywordID: number) : Promise<void> => {
-   if (!keywordID) { return; }
+   if (!keywordID && !Number.isInteger(keywordID)) { return; }
   let currentQueue: number[] = [];

   const filePath = `${process.cwd()}/data/failed_queue.json`;
   const currentQueueRaw = await readFile(filePath, { encoding: 'utf-8' }).catch((err) => { console.log(err); return '[]'; });
-   currentQueue = JSON.parse(currentQueueRaw);
-   currentQueue = currentQueue.filter((item) => item !== keywordID);
+   currentQueue = currentQueueRaw ? JSON.parse(currentQueueRaw) : [];
+   currentQueue = currentQueue.filter((item) => item !== Math.abs(keywordID));

   await writeFile(filePath, JSON.stringify(currentQueue), { encoding: 'utf-8' }).catch((err) => { console.log(err); return '[]'; });
 };
--- a/utils/searchConsole.ts
+++ b/utils/searchConsole.ts
@@ -54,7 +54,7 @@ const fetchSearchConsoleData = async (domainName:string, days:number, type?:stri
               date: row.keys[0],
               clicks: row.clicks,
               impressions: row.impressions,
-               ctr: row.ctr,
+               ctr: row.ctr * 100,
               position: row.position,
            });
         });
@@ -101,7 +101,7 @@ export const parseSearchConsoleItem = (SCItem: SearchAnalyticsRawItem, domainNam
   const page = SCItem.keys[3] ? SCItem.keys[3].replace('https://', '').replace('http://', '').replace('www', '').replace(domainName, '') : '';
   const uid = `${country.toLowerCase()}:${device}:${keyword.replaceAll(' ', '_')}`;

-   return { keyword, uid, device, country, clicks, impressions, ctr, position, page };
+   return { keyword, uid, device, country, clicks, impressions, ctr: ctr * 100, position, page };
 };

 export const integrateKeywordSCData = (keyword: KeywordType, SCData:SCDomainDataType) : KeywordType => {
--- a/utils/sortFilter.ts
+++ b/utils/sortFilter.ts
@@ -28,7 +28,7 @@ export const sortKeywords = (theKeywords:KeywordType[], sortBy:string, scDataTyp
      case 'alpha_desc':
            sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => (a.keyword > b.keyword ? 1 : -1));
         break;
-      case 'imp_asc':
+      case 'imp_desc':
            if (scDataType) {
                  sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => {
                  const bImpressionData = b.scData?.impressions[scDataType as keyof KeywordSCDataChild] || 0;
@@ -37,7 +37,7 @@ export const sortKeywords = (theKeywords:KeywordType[], sortBy:string, scDataTyp
               });
            }
            break;
-      case 'imp_desc':
+      case 'imp_asc':
            if (scDataType) {
                  sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => {
                  const bImpressionData = b.scData?.impressions[scDataType as keyof KeywordSCDataChild] || 0;
@@ -46,21 +46,21 @@ export const sortKeywords = (theKeywords:KeywordType[], sortBy:string, scDataTyp
               });
            }
         break;
-      case 'visits_asc':
-            if (scDataType) {
-                  sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => {
-                  const bImpressionData = b.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
-                  const aImpressionData = a.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
-                  return aImpressionData > bImpressionData ? 1 : -1;
-               });
-            }
-            break;
      case 'visits_desc':
            if (scDataType) {
                  sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => {
-                  const bImpressionData = b.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
-                  const aImpressionData = a.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
-                  return bImpressionData > aImpressionData ? 1 : -1;
+                  const bVisitsData = b.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
+                  const aVisitsData = a.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
+                  return aVisitsData > bVisitsData ? 1 : -1;
+               });
+            }
+            break;
+      case 'visits_asc':
+            if (scDataType) {
+                  sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => {
+                  const bVisitsData = b.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
+                  const aVisitsData = a.scData?.visits[scDataType as keyof KeywordSCDataChild] || 0;
+                  return bVisitsData > aVisitsData ? 1 : -1;
               });
            }
            break;
@@ -98,7 +98,8 @@ export const filterKeywords = (keywords: KeywordType[], filterParams: KeywordFil
   const filteredItems:KeywordType[] = [];
   keywords.forEach((keywrd) => {
       const countryMatch = filterParams.countries.length === 0 ? true : filterParams.countries && filterParams.countries.includes(keywrd.country);
-       const searchMatch = !filterParams.search ? true : filterParams.search && keywrd.keyword.includes(filterParams.search);
+       const searchMatch = !filterParams.search ? true : filterParams.search
+       && keywrd.keyword.toLowerCase().includes(filterParams.search.toLowerCase());
       const tagsMatch = filterParams.tags.length === 0 ? true : filterParams.tags && keywrd.tags.find((x) => filterParams.tags.includes(x));

       if (countryMatch && searchMatch && tagsMatch) {
--- a/utils/validators.ts
+++ b/utils/validators.ts
@@ -0,0 +1,35 @@
+/* eslint-disable import/prefer-default-export */
+export const isValidDomain = (domain:string): boolean => {
+   if (typeof domain !== 'string') return false;
+   if (!domain.includes('.')) return false;
+   let value = domain;
+   const validHostnameChars = /^[a-zA-Z0-9-.]{1,253}\.?$/g;
+   if (!validHostnameChars.test(value)) {
+     return false;
+   }
+
+   if (value.endsWith('.')) {
+     value = value.slice(0, value.length - 1);
+   }
+
+   if (value.length > 253) {
+     return false;
+   }
+
+   const labels = value.split('.');
+
+   const isValid = labels.every((label) => {
+     const validLabelChars = /^([a-zA-Z0-9-]+)$/g;
+
+     const validLabel = (
+       validLabelChars.test(label)
+       && label.length < 64
+       && !label.startsWith('-')
+       && !label.endsWith('-')
+     );
+
+     return validLabel;
+   });
+
+   return isValid;
+ };
Author	SHA1	Message	Date
towfiqi	08f44911d7	chore(release): 0.3.3	2023-11-12 20:59:08 +06:00
towfiqi	9dce1d5b48	chore: resolves test issues	2023-11-12 20:58:03 +06:00
towfiqi	fbd23ede25	feat: Shows total keywords count in domains page	2023-11-11 12:48:31 +06:00
towfiqi	60c68bd339	feat: Adds ability to visit pages from Insight tab	2023-11-11 11:42:14 +06:00
towfiqi	2339e31af9	feat: Domains now show their favicon. closes #130	2023-11-11 11:29:59 +06:00
towfiqi	4a60271cac	fix: Resolves Website Thumbnail missing issue You can now set your own thum.io key to fetch screenshot. More in docs. closes #131	2023-11-11 11:17:44 +06:00
towfiqi	c870250fbd	chore(release): 0.3.2	2023-11-09 20:35:49 +06:00
towfiqi	da92f11afa	chore: resolves linter nag.	2023-11-09 20:35:22 +06:00
towfiqi	9b9b74af4c	fix: Resolves issue with adding long tld emails closes #127	2023-11-09 20:35:01 +06:00
Towfiq I	291aa60bbb	Merge pull request #129 from SearchApi/feature/integrate-searchapi feat: Integrates SearchAPI fix: Resolves build issue due to missing jest types. fix: Resolves keyword SERP fetch issue.	2023-11-09 20:01:02 +06:00
SebastjanPrachovskij	8a35e358e6	Integrate SearchApi to SerpBear Remove unnecessary spacing Fix keyword.id & authorization for searchapi	2023-11-09 14:50:11 +02:00
towfiqi	f164b287be	chore(release): 0.3.1	2023-11-04 22:05:59 +06:00
towfiqi	97dd0b131b	fix: Updates vulnerable dependencies	2023-11-04 22:05:10 +06:00
towfiqi	454454a422	fix: Removes dev files from docker volumes	2023-11-04 11:25:56 +06:00
towfiqi	4620f11c4b	Fixed: Resolves missing Scrapers list on new install	2023-11-04 10:47:22 +06:00
towfiqi	7ab435ed8b	chore(release): 0.3.0	2023-11-03 22:24:46 +06:00
towfiqi	9feff13f18	fix: Fxies special character keyword scrape issue. closes #113 #122	2023-11-03 21:44:23 +06:00
towfiqi	f57bca23da	fix: Resolves missing keyword scrape spinner issue	2023-11-03 21:30:00 +06:00
towfiqi	392122a710	fix: Fixes the weekly cron day issue. closes #118	2023-11-03 21:05:28 +06:00
towfiqi	fc183d246d	feat: Displays the Best position of the keywords closes #89	2023-11-03 20:45:36 +06:00
towfiqi	994afbcedb	chore: changes toggle button ui	2023-11-03 13:51:50 +06:00
towfiqi	d3d336fa71	feat: Remembers last selected coutry closes #101	2023-11-03 12:47:29 +06:00
towfiqi	6f34d64fd5	chore: opens keyword detail view on chart click	2023-11-03 12:43:56 +06:00
towfiqi	a0014c7650	chore: breakup Settings component	2023-11-03 12:20:02 +06:00
towfiqi	dc3c7a722b	feat: Adds ability to disable/clear retry queue for failed keywords	2023-11-03 11:59:51 +06:00
towfiqi	8a949ce4c0	fix: Cron stopped on failing to parse failed queue closes #116	2023-11-03 11:58:17 +06:00
towfiqi	312d12f589	chore: fixes tag menu overlap issue.	2023-11-02 22:46:55 +06:00
towfiqi	be80ed7ef3	fix: Fixes import order error in some instances. closes #114	2023-11-02 22:35:25 +06:00
towfiqi	4748ffc382	feat: Adds ability to search w/o case sensitivity closes #115	2023-11-02 22:21:07 +06:00
towfiqi	c0470cfa9d	fix: Fixes issue with adding hyphenated subdomains.	2023-11-02 22:15:26 +06:00
towfiqi	1d6b2be95a	feat: Refresh All feature now shows update real-time	2023-11-02 21:51:06 +06:00
towfiqi	0d846b29f1	chore(release): 0.2.6	2023-03-29 21:01:13 +06:00
towfiqi	3b96dab9cc	chore: Adds Space Serp Details.	2023-03-29 20:54:55 +06:00
towfiqi	0a83924ffe	feat: Add option to Delay Between scrapes. fixes #87	2023-03-29 20:54:30 +06:00
towfiqi	d9505158c4	fix: Fixes first Keryword Error cut off issue.	2023-03-29 20:11:56 +06:00
towfiqi	9757fde02e	fix: Fixes lags when tracking thousands of keywords fixes #88	2023-03-29 12:59:22 +06:00
towfiqi	0538a8c016	feat: Integrates Space Serp.	2023-03-29 12:14:28 +06:00
Towfiq I	cace34f39a	Merge pull request #85 from Teeth-Talk/hotfix-typo fix(components): fix typo "Goolge" -> "Google"	2023-03-29 10:49:04 +06:00
Martin Silha	dce7c412e8	fix(components): fix typo "Goolge" -> "Google"	2023-03-16 21:44:59 +00:00
towfiqi	e61dfb5b90	chore(release): 0.2.5	2023-03-07 11:08:42 +06:00
towfiqi	b9d58a721d	fix: Settings Update Toast was not showing up.	2023-03-05 19:39:14 +06:00
towfiqi	b83df5f3db	feat: Adds current App version Number in Footer.	2023-03-05 19:23:07 +06:00
towfiqi	3b6d034d6f	feat: Adds Keyword Scraping Interval Settings. fixes #81, #76	2023-03-05 12:28:21 +06:00
towfiqi	5dd366b91e	fix: Fixes Broken Image thumbnail loading issue.	2023-03-05 12:14:08 +06:00
towfiqi	5fc1779783	chore(release): 0.2.4	2023-02-15 23:33:37 +06:00
towfiqi	c5af94a146	feat: Keyword ranking pages can now be clicked.	2023-02-15 23:33:23 +06:00
towfiqi	c406588953	fix: Fixes broken Login on windows fixes: 77, 80	2023-02-15 23:32:38 +06:00
towfiqi	3c48d130b6	fix: Fixes Node Cron memory leak issue.	2023-02-15 23:30:16 +06:00
towfiqi	99dbbd1dd9	chore(release): 0.2.3	2023-01-12 22:01:40 +06:00
towfiqi	acc0b39d80	fix: ScrapingAnt Mobile Keyword Scrape not working	2023-01-12 12:26:54 +06:00
towfiqi	a1108d240e	fix: Mobile Keyword Scraping not working. fixes #58	2023-01-12 12:25:42 +06:00
towfiqi	d9e0d0107a	chore: keyword details title style issue fix.	2023-01-11 21:30:34 +06:00
towfiqi	9e9dad7631	feat: Ability to tag multiple keywords at once fixes #54	2023-01-11 21:29:52 +06:00
towfiqi	8139e399c1	fix: Fixes Position and View Sort. fixes #46	2023-01-11 13:15:09 +06:00
towfiqi	cb24696a1f	fix: Fixes wrong CTR value for Search Console Data fixes #48	2023-01-11 12:42:27 +06:00
towfiqi	b50733defc	feat: Set USERNAME as well as USER variable Mac Users weren't able to set USER variable for the project.	2023-01-11 12:29:29 +06:00