chore(release): 0.1.7

fix: Email notifcations now sent everyday at 3am
fix: Throws better error logs in cron for debugging
2025-06-26 18:15:54 +00:00 · 2022-12-08 09:51:06 +06:00 · 2022-12-08 09:50:43 +06:00 · 2022-12-08 09:47:53 +06:00 · 2022-12-08 09:47:02 +06:00 · 2022-12-08 09:46:22 +06:00
29 changed files with 1782 additions and 605 deletions
--- a/.stylelintrc.json
+++ b/.stylelintrc.json
@@ -0,0 +1,22 @@
+{
+   "extends": "stylelint-config-standard",
+   "rules": {
+      "indentation": 3,
+      "declaration-block-single-line-max-declarations": 2,
+      "selector-class-pattern":null,
+      "at-rule-no-unknown": [
+        true,
+        {
+          "ignoreAtRules": [
+            "tailwind",
+            "apply",
+            "variants",
+            "responsive",
+            "screen"
+          ]
+        }
+      ],
+      "declaration-block-trailing-semicolon": null,
+      "no-descending-specificity": null
+    }
+ }
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,70 @@

 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.

+### [0.1.7](https://github.com/towfiqi/serpbear/compare/v0.1.6...v0.1.7) (2022-12-08)
+
+
+### Bug Fixes
+
+* Email notifcations now sent everyday at 3am ([f000063](https://github.com/towfiqi/serpbear/commit/f00006371d56c509eae00a72e164658c84fecd00))
+* shortens hours and minutes in notif emails ([480767d](https://github.com/towfiqi/serpbear/commit/480767deb24072f9e250e4dd7bd3d710c4b6046c))
+* Throws better error logs in cron for debugging ([a6af9d3](https://github.com/towfiqi/serpbear/commit/a6af9d347544f847e512c4ae55b14c640a897240))
+
+### [0.1.6](https://github.com/towfiqi/serpbear/compare/v0.1.5...v0.1.6) (2022-12-05)
+
+
+### Bug Fixes
+
+* CSS Linter issues. ([a599035](https://github.com/towfiqi/serpbear/commit/a59903551eccb3f03f2bc026673bbf9fd0d4bc1e))
+* invalid json markup ([e9d7730](https://github.com/towfiqi/serpbear/commit/e9d7730ae7ec647d333713248b271bae8693e77b))
+* Sort was buggy for keyword with >100 position ([d22992b](https://github.com/towfiqi/serpbear/commit/d22992bf6489b11002faba60fa06b5c467867c8b)), closes [#23](https://github.com/towfiqi/serpbear/issues/23)
+* **UI:** Adds tooltip for Domain action icons. ([b450540](https://github.com/towfiqi/serpbear/commit/b450540d9593d022c94708c9679b5bf7c0279c50))
+
+### [0.1.5](https://github.com/towfiqi/serpbear/compare/v0.1.4...v0.1.5) (2022-12-03)
+
+
+### Features
+
+* keyword not in first 100 now shows >100 ([e1799fb](https://github.com/towfiqi/serpbear/commit/e1799fb2f35ab8c0f65eb90e66dcda10b8cb6f16))
+
+
+### Bug Fixes
+
+* domains with - were not loading the keywords. ([efb565b](https://github.com/towfiqi/serpbear/commit/efb565ba0086d1b3e69ea71456a892ca254856f7)), closes [#11](https://github.com/towfiqi/serpbear/issues/11)
+* failed scrape messes up lastResult data in db ([dd6a801](https://github.com/towfiqi/serpbear/commit/dd6a801ffda3eacda957dd20d2c97fb6197fbdc2))
+* First search result items were being skipped. ([d6da18f](https://github.com/towfiqi/serpbear/commit/d6da18fb0135e23dd869d1fb500e12ee2e782bfa)), closes [#13](https://github.com/towfiqi/serpbear/issues/13)
+* removes empty spaces when adding domain. ([a11b0f2](https://github.com/towfiqi/serpbear/commit/a11b0f223c0647537ab23564df1d2f0b29eef4ae))
+
+### [0.1.4](https://github.com/towfiqi/serpbear/compare/v0.1.3...v0.1.4) (2022-12-01)
+
+
+### Features
+
+* Failed scrape now shows error details in UI. ([8c8064f](https://github.com/towfiqi/serpbear/commit/8c8064f222ea8177b26b6dd28866d1f421faca39))
+
+
+### Bug Fixes
+
+* Domains with www weren't loading keywords. ([3d1c690](https://github.com/towfiqi/serpbear/commit/3d1c690076a03598f0ac3f3663d905479d945897)), closes [#8](https://github.com/towfiqi/serpbear/issues/8)
+* Emails were sending serps of previous day. ([6910558](https://github.com/towfiqi/serpbear/commit/691055811c2ae70ce1b878346300048c1e23f2eb))
+* Fixes Broken ScrapingRobot Integration. ([1ed298f](https://github.com/towfiqi/serpbear/commit/1ed298f633a9ae5b402b431f1e50b35ffd44a6dc))
+* scraper fails if matched domain  has www ([38dc164](https://github.com/towfiqi/serpbear/commit/38dc164514b066b2007f2f3b2ae68005621963cc)), closes [#6](https://github.com/towfiqi/serpbear/issues/6) [#7](https://github.com/towfiqi/serpbear/issues/7)
+* scraper fails when result has domain w/o www ([6d7cfec](https://github.com/towfiqi/serpbear/commit/6d7cfec95304fa7a61beaab07f7cd6af215255c3))
+
+### [0.1.3](https://github.com/towfiqi/serpbear/compare/v0.1.2...v0.1.3) (2022-12-01)
+
+
+### Features
+
+* Adds a search field in Country select field. ([be4db26](https://github.com/towfiqi/serpbear/commit/be4db26316e7522f567a4ce6fc27e0a0f73f89f2)), closes [#2](https://github.com/towfiqi/serpbear/issues/2)
+
+
+### Bug Fixes
+
+* could not add 2 character domains. ([5acbe18](https://github.com/towfiqi/serpbear/commit/5acbe181ec978b50b588af378d17fb3070c241d1)), closes [#1](https://github.com/towfiqi/serpbear/issues/1)
+* license location. ([a45237b](https://github.com/towfiqi/serpbear/commit/a45237b230a9830461cf7fccd4c717235112713b))
+* No hint on how to add multiple keywords. ([9fa80cf](https://github.com/towfiqi/serpbear/commit/9fa80cf6098854d2a5bd5a8202aa0fd6886d1ba0)), closes [#3](https://github.com/towfiqi/serpbear/issues/3)
+
 ### 0.1.2 (2022-11-30)


--- a/21
+++ b/21
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Amruth Pillai
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
--- a/README.md
+++ b/README.md
@@ -1,30 +1,34 @@
 ![SerpBear](https://i.imgur.com/0S2zIH3.png) 
 # SerpBear

+![Codacy Badge](https://app.codacy.com/project/badge/Grade/7e7a0030c3f84c6fb56a3ce6273fbc1d) ![GitHub](https://img.shields.io/github/license/towfiqi/serpbear) ![GitHub package.json version](https://img.shields.io/github/package-json/v/towfiqi/serpbear) ![Docker Pulls](https://img.shields.io/docker/pulls/towfiqi/serpbear)
+
+#### [Documentation](https://docs.serpbear.com/) | [Changelog](https://github.com/towfiqi/serpbear/blob/main/CHANGELOG.md) | [Docker Image](https://hub.docker.com/r/towfiqi/serpbear)
+
 SerpBear is an Open Source Search Engine Position Tracking App. It allows you to track your website's keyword positions in Google and get notified of their positions.

 ![Easy to Use Search Engine Rank Tracker](https://i.imgur.com/bRzpmCK.gif)

-**Features**
- - **Unlimited Keywords:** Add unlimited domains and unlimited keywords to track their SERP.
- - **Email Notification:** Get notified of your keyword position changes daily/weekly/monthly through email.
- - **SERP API:** SerpBear comes with built-in API that you can use for your marketing & data reporting tools.
- - **Mobile App:** Add the PWA app to your mobile for a better mobile experience. 
- - **Zero Cost to RUN:** Run the App on mogenius.com or Fly.io for free.
+#### Features
+-   **Unlimited Keywords:** Add unlimited domains and unlimited keywords to track their SERP.
+-   **Email Notification:** Get notified of your keyword position changes daily/weekly/monthly through email.
+-   **SERP API:** SerpBear comes with built-in API that you can use for your marketing & data reporting tools.
+-   **Mobile App:** Add the PWA app to your mobile for a better mobile experience. 
+-   **Zero Cost to RUN:** Run the App on mogenius.com or Fly.io for free.

-**How it Works**
+#### How it Works
 The App uses third party website scrapers like ScrapingAnt, ScrapingRobot or Your given Proxy ips to scrape google search results to see if your domain appears in the search result for the given keyword.

-**Getting Started**
- - **Step 1:** Deploy & Run the App.
- - **Step 2:** Access your App and Login.
- - **Step 3:** Add your First domain.
- - **Step 4:** Get an free API key from either ScrapingAnt or ScrapingRobot. Skip if you want to use Proxy ips.
- - **Step 5:** Setup the Scraping API/Proxy from the App's Settings interface.
- - **Step 6:** Add your keywords and start tracking.
- - **Step 7:** Optional. From the Settings panel, setup SMTP details to get notified of your keywords positions through email. You can use ElasticEmail and Sendpulse SMTP services that are free.  
+#### Getting Started
+-   **Step 1:** Deploy & Run the App.
+-   **Step 2:** Access your App and Login.
+-   **Step 3:** Add your First domain.
+-   **Step 4:** Get an free API key from either ScrapingAnt or ScrapingRobot. Skip if you want to use Proxy ips.
+-   **Step 5:** Setup the Scraping API/Proxy from the App's Settings interface.
+-   **Step 6:** Add your keywords and start tracking.
+-   **Step 7:** Optional. From the Settings panel, setup SMTP details to get notified of your keywords positions through email. You can use ElasticEmail and Sendpulse SMTP services that are free.  

-**Compare SerpBear with other SERP tracking services:**
+#### Compare SerpBear with other SERP tracking services

 |Service  | Cost | SERP Lookup | API |
 |--|--|--|--|
@@ -33,9 +37,10 @@ The App uses third party website scrapers like ScrapingAnt, ScrapingRobot or You
 | SerpWatch.io | $29/mo | 7500/mo | Yes |
 | Serpwatcher.com | $49/mo| 3000/mo | No |
 | whatsmyserp.com | $49/mo| 30,000/mo| No |
+| serply.io | $49/mo | 5000/mo | Yes |

 (*) Free upto a limit. If you are using ScrapingAnt you can lookup 10,000 times per month for free.

 **Stack**
- - Next.js for Frontend & Backend.  
- - Sqlite for Database.
+-   Next.js for Frontend & Backend.  
+-   Sqlite for Database.
--- a/components/common/SelectField.tsx
+++ b/components/common/SelectField.tsx
@@ -30,7 +30,9 @@ const SelectField = (props: SelectFieldProps) => {
      flags = false,
      emptyMsg = '' } = props;

-   const [showOptions, setShowOptions] = useState(false);
+   const [showOptions, setShowOptions] = useState<boolean>(false);
+   const [filterInput, setFilterInput] = useState<string>('');
+   const [filterdOptions, setFilterdOptions] = useState<SelectionOption[]>([]);

   const selectedLabels = useMemo(() => {
      return options.reduce((acc:string[], item:SelectionOption) :string[] => {
@@ -51,6 +53,18 @@ const SelectField = (props: SelectFieldProps) => {
      if (!multiple) { setShowOptions(false); }
   };

+   const filterOptions = (event:React.FormEvent<HTMLInputElement>) => {
+      setFilterInput(event.currentTarget.value);
+      const filteredItems:SelectionOption[] = [];
+      const userVal = event.currentTarget.value.toLowerCase();
+      options.forEach((option:SelectionOption) => {
+         if (flags ? option.label.toLowerCase().startsWith(userVal) : option.label.toLowerCase().includes(userVal)) {
+            filteredItems.push(option);
+         }
+      });
+      setFilterdOptions(filteredItems);
+   };
+
   return (
       <div className="select font-semibold text-gray-500">
         <div
@@ -68,8 +82,19 @@ const SelectField = (props: SelectFieldProps) => {
            <div
            className={`select_list mt-1 border absolute min-w-[${minWidth}px] 
            ${rounded === 'rounded-3xl' ? 'rounded-lg' : rounded} max-h-${maxHeight} bg-white z-50 overflow-y-auto styled-scrollbar`}>
+               {options.length > 20 && (
+                  <div className=''>
+                     <input
+                     className=' border-b-[1px] p-3 w-full focus:outline-0 focus:border-blue-100'
+                     type="text"
+                     placeholder='Search..'
+                     onChange={filterOptions}
+                     value={filterInput}
+                     />
+                  </div>
+               )}
               <ul>
-                  {options.map((opt) => {
+                  {(options.length > 20 && filterdOptions.length > 0 && filterInput ? filterdOptions : options).map((opt) => {
                     const itemActive = selected.includes(opt.value);
                     return (
                        <li
--- a/components/domains/AddDomain.tsx
+++ b/components/domains/AddDomain.tsx
@@ -13,10 +13,10 @@ const AddDomain = ({ closeModal }: AddDomainProps) => {

   const addDomain = () => {
      // console.log('ADD NEW DOMAIN', newDomain);
-      if (/^[a-zA-Z0-9][a-zA-Z0-9-]{1,61}[a-zA-Z0-9](?:\.[a-zA-Z]{2,})+$/.test(newDomain)) {
+      if (/^[a-zA-Z0-9][a-zA-Z0-9-]{0,61}[a-zA-Z0-9](?:\.[a-zA-Z]{2,})+$/.test(newDomain.trim())) {
         setNewDomainError(false);
         // TODO: Domain Action
-         addMutate(newDomain);
+         addMutate(newDomain.trim());
      } else {
         setNewDomainError(true);
      }
--- a/components/domains/DomainHeader.tsx
+++ b/components/domains/DomainHeader.tsx
@@ -19,6 +19,7 @@ const DomainHeader = ({ domain, showAddModal, showSettingsModal, exportCsv, doma
   const { mutate: refreshMutate } = useRefreshKeywords(() => {});

   const buttonStyle = 'leading-6 inline-block px-2 py-2 text-gray-500 hover:text-gray-700';
+   const buttonLabelStyle = 'ml-2 text-sm not-italic lg:invisible lg:opacity-0';
   return (
      <div className='domain_kewywords_head flex w-full justify-between'>
         <div>
@@ -45,26 +46,23 @@ const DomainHeader = ({ domain, showAddModal, showSettingsModal, exportCsv, doma
            lg:z-auto lg:relative lg:mt-0 lg:border-0 lg:w-auto lg:bg-transparent`}
            style={{ display: showOptions ? 'block' : undefined }}>
               <button
-               className={`${buttonStyle}`}
+               className={`domheader_action_button relative ${buttonStyle}`}
               aria-pressed="false"
-               title='Export as CSV'
               onClick={() => exportCsv()}>
-                  <Icon type='download' size={20} /><i className='ml-2 text-sm not-italic lg:hidden'>Export as csv</i>
+                  <Icon type='download' size={20} /><i className={`${buttonLabelStyle}`}>Export as csv</i>
               </button>
               <button
-               className={`${buttonStyle} lg:ml-3`}
+               className={`domheader_action_button relative ${buttonStyle} lg:ml-3`}
               aria-pressed="false"
-               title='Refresh All Keyword Positions'
               onClick={() => refreshMutate({ ids: [], domain: domain.domain })}>
-                  <Icon type='reload' size={14} /><i className='ml-2 text-sm not-italic lg:hidden'>Reload All Serps</i>
+                  <Icon type='reload' size={14} /><i className={`${buttonLabelStyle}`}>Reload All Serps</i>
               </button>
               <button
               data-testid="show_domain_settings"
-               className={`${buttonStyle} lg:ml-3`}
+               className={`domheader_action_button relative ${buttonStyle} lg:ml-3`}
               aria-pressed="false"
-               title='Domain Settings'
               onClick={() => showSettingsModal(true)}><Icon type='settings' size={20} />
-               <i className='ml-2 text-sm not-italic lg:hidden'>Domain Settings</i></button>
+               <i className={`${buttonLabelStyle}`}>Domain Settings</i></button>
            </div>
            <button
            data-testid="add_keyword"
--- a/components/keywords/AddKeywords.tsx
+++ b/components/keywords/AddKeywords.tsx
@@ -27,14 +27,14 @@ const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {

   const addKeywords = () => {
      if (newKeywordsData.keywords) {
-         const keywordsArray = newKeywordsData.keywords.replaceAll('\n', ',').split(',').map((item:string) => item.trim());
+         const keywordsArray = [...new Set(newKeywordsData.keywords.split('\n').map((item) => item.trim()).filter((item) => !!item))];
         const currentKeywords = keywords.map((k) => `${k.keyword}-${k.device}-${k.country}`);
         const keywordExist = keywordsArray.filter((k) => currentKeywords.includes(`${k}-${newKeywordsData.device}-${newKeywordsData.country}`));
         if (keywordExist.length > 0) {
            setError(`Keywords ${keywordExist.join(',')} already Exist`);
            setTimeout(() => { setError(''); }, 3000);
         } else {
-            addMutate(newKeywordsData);
+            addMutate({ ...newKeywordsData, keywords: keywordsArray.join('\n') });
         }
      } else {
         setError('Please Insert a Keyword');
@@ -49,7 +49,7 @@ const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {
               <div>
                  <textarea
                     className='w-full h-40 border rounded border-gray-200 p-4 outline-none focus:border-indigo-300'
-                     placeholder='Type or Paste Keywords here...'
+                     placeholder="Type or Paste Keywords here. Insert Each keyword in a New line."
                     value={newKeywordsData.keywords}
                     onChange={(e) => setNewKeywordsData({ ...newKeywordsData, keywords: e.target.value })}>
                  </textarea>
--- a/components/keywords/Keyword.tsx
+++ b/components/keywords/Keyword.tsx
@@ -21,7 +21,7 @@ type KeywordProps = {
 const Keyword = (props: KeywordProps) => {
   const { keywordData, refreshkeyword, favoriteKeyword, removeKeyword, selectKeyword, selected, showKeywordDetails, manageTags, lastItem } = props;
   const {
-      keyword, domain, ID, position, url = '', lastUpdated, country, sticky, history = {}, updating = false, lastUpdateError = 'false',
+      keyword, domain, ID, position, url = '', lastUpdated, country, sticky, history = {}, updating = false, lastUpdateError = false,
   } = keywordData;
   const [showOptions, setShowOptions] = useState(false);
   const [showPositionError, setPositionError] = useState(false);
@@ -50,7 +50,7 @@ const Keyword = (props: KeywordProps) => {

   const renderPosition = () => {
      if (position === 0) {
-         return <span title='Not in Top 100'>{'-'}</span>;
+         return <span className='text-gray-400' title='Not in Top 100'>{'>100'}</span>;
      }
      if (updating) {
         return <span title='Updating Keyword Position'><Icon type="loading" /></span>;
@@ -77,7 +77,7 @@ const Keyword = (props: KeywordProps) => {
               <span className={`fflag fflag-${country} w-[18px] h-[12px] mr-2`} title={countries[country][0]} />{keyword}
            </a>
            {sticky && <button className='ml-2 relative top-[2px]' title='Favorite'><Icon type="star-filled" size={16} color="#fbd346" /></button>}
-            {lastUpdateError !== 'false'
+            {lastUpdateError && lastUpdateError.date
               && <button className='ml-2 relative top-[2px]' onClick={() => setPositionError(true)}>
                  <Icon type="error" size={18} color="#FF3672" />
               </button>
@@ -133,16 +133,19 @@ const Keyword = (props: KeywordProps) => {
               </ul>
            )}
         </div>
-         {lastUpdateError !== 'false' && showPositionError
-            && <div className=' absolute mt-[-70px] p-2 bg-white z-30 border border-red-200 rounded w-[220px] left-4 shadow-sm text-xs'>
+         {lastUpdateError && lastUpdateError.date && showPositionError && (
+            <div className=' absolute mt-[-70px] p-2 bg-white z-30 border border-red-200 rounded w-[220px] left-4 shadow-sm text-xs lg:bottom-12'>
               Error Updating Keyword position (Tried <TimeAgo
-                                                         title={dayjs(lastUpdateError).format('DD-MMM-YYYY, hh:mm:ss A')}
-                                                         date={lastUpdateError} />)
+                                                         title={dayjs(lastUpdateError.date).format('DD-MMM-YYYY, hh:mm:ss A')}
+                                                         date={lastUpdateError.date} />)
               <i className='absolute top-0 right-0 ml-2 p-2 font-semibold not-italic cursor-pointer' onClick={() => setPositionError(false)}>
                  <Icon type="close" size={16} color="#999" />
               </i>
+               <div className=' border-t-[1px] border-red-100 mt-2 pt-1'>
+                  {lastUpdateError.scraper && <strong className='capitalize'>{lastUpdateError.scraper}: </strong>}{lastUpdateError.error}
+               </div>
            </div>
-         }
+            )}
      </div>
   );
 };
--- a/components/settings/Settings.tsx
+++ b/components/settings/Settings.tsx
@@ -105,6 +105,7 @@ const Settings = ({ closeSettings }:SettingsProps) => {
      { label: 'Proxy', value: 'proxy' },
      { label: 'ScrapingAnt.com', value: 'scrapingant' },
      { label: 'ScrapingRobot.com', value: 'scrapingrobot' },
+      { label: 'serply.io', value: 'serply' },
   ];

   const tabStyle = 'inline-block px-4 py-1 rounded-full mr-3 cursor-pointer text-sm';
@@ -150,7 +151,7 @@ const Settings = ({ closeSettings }:SettingsProps) => {
                           minWidth={270}
                           />
                        </div>
-                        {['scrapingant', 'scrapingrobot'].includes(settings.scraper_type) && (
+                        {['scrapingant', 'scrapingrobot', 'serply'].includes(settings.scraper_type) && (
                           <div className="settings__section__input mr-3">
                              <label className={labelStyle}>Scraper API Key or Token</label>
                              <input
--- a/cron.js
+++ b/cron.js
@@ -35,8 +35,7 @@ const getAppSettings = async () => {
      }
      return decryptedSettings;
   } catch (error) {
-      console.log(error);
-
+      // console.log('CRON ERROR: Reading Settings File. ', error);
      await promises.writeFile(`${process.cwd()}/data/settings.json`, JSON.stringify(defaultSettings), { encoding: 'utf-8' });
      return defaultSettings;
   }
@@ -50,6 +49,9 @@ const generateCronTime = (interval) => {
   if (interval === 'daily') {
      cronTime = '0 0 0 * * *';
   }
+   if (interval === 'daily_morning') {
+      cronTime = '0 0 0 3 * *';
+   }
   if (interval === 'weekly') {
      cronTime = '0 0 0 */7 * *';
   }
@@ -70,7 +72,7 @@ const runAppCronJobs = () => {
      .then((res) => res.json())
      .then((data) => console.log(data))
      .catch((err) => {
-         console.log('ERROR Making Cron Request..');
+         console.log('ERROR Making Daily Scraper Cron Request..');
         console.log(err);
      });
   }, { scheduled: true });
@@ -89,7 +91,7 @@ const runAppCronJobs = () => {
               .then((res) => res.json())
               .then((refreshedData) => console.log(refreshedData))
               .catch((fetchErr) => {
-                  console.log('ERROR Making Cron Request..');
+                  console.log('ERROR Making failed_queue Cron Request..');
                  console.log(fetchErr);
               });
            }
@@ -103,7 +105,7 @@ const runAppCronJobs = () => {
   getAppSettings().then((settings) => {
      const notif_interval = (!settings.notification_interval || settings.notification_interval === 'never') ? false : settings.notification_interval;
      if (notif_interval) {
-         const cronTime = generateCronTime(notif_interval);
+         const cronTime = generateCronTime(notif_interval === 'daily' ? 'daily_morning' : notif_interval);
         if (cronTime) {
            cron.schedule(cronTime, () => {
               // console.log('### Sending Notification Email...');
@@ -112,7 +114,7 @@ const runAppCronJobs = () => {
               .then((res) => res.json())
               .then((data) => console.log(data))
               .catch((err) => {
-                  console.log('ERROR Making Cron Request..');
+                  console.log('ERROR Making Cron Email Notification Request..');
                  console.log(err);
               });
            }, { scheduled: true });
--- a/package-lock.json
+++ b/package-lock.json
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
 {
  "name": "serpbear",
-  "version": "0.1.2",
+  "version": "0.1.7",
  "private": true,
  "scripts": {
    "dev": "next dev",
@@ -67,6 +67,7 @@
    "resize-observer-polyfill": "^1.5.1",
    "sass": "^1.55.0",
    "standard-version": "^9.5.0",
+    "stylelint-config-standard": "^29.0.0",
    "tailwindcss": "^3.1.8",
    "typescript": "4.8.4"
  }
--- a/pages/api/domains.ts
+++ b/pages/api/domains.ts
@@ -61,8 +61,8 @@ export const addDomain = async (req: NextApiRequest, res: NextApiResponse<Domain
   }
   const { domain } = req.body || {};
   const domainData = {
-      domain,
-      slug: domain.replaceAll('.', '-'),
+      domain: domain.trim(),
+      slug: domain.trim().replaceAll('-', '_').replaceAll('.', '-'),
      lastUpdated: new Date().toJSON(),
      added: new Date().toJSON(),
   };
--- a/pages/api/keywords.ts
+++ b/pages/api/keywords.ts
@@ -44,7 +44,8 @@ const getKeywords = async (req: NextApiRequest, res: NextApiResponse<KeywordsGet
   if (!req.query.domain && typeof req.query.domain !== 'string') {
      return res.status(400).json({ error: 'Domain is Required!' });
   }
-   const domain = (req.query.domain as string).replace('-', '.');
+   const domain = (req.query.domain as string).replaceAll('-', '.').replaceAll('_', '-');
+
   try {
      const allKeywords:Keyword[] = await Keyword.findAll({ where: { domain } });
      const keywords: KeywordType[] = parseKeywords(allKeywords.map((e) => e.get({ plain: true })));
--- a/pages/api/refresh.ts
+++ b/pages/api/refresh.ts
@@ -79,18 +79,21 @@ export const refreshAndUpdateKeywords = async (initKeywords:Keyword[], settings:
         const newPos = udpatedkeyword.position;
         const newPosition = newPos !== false ? newPos : keyword.position;
         const { history } = keyword;
-         const currentDate = new Date();
-         history[`${currentDate.getFullYear()}-${currentDate.getMonth() + 1}-${currentDate.getDate()}`] = newPosition;
+         const theDate = new Date();
+         history[`${theDate.getFullYear()}-${theDate.getMonth() + 1}-${theDate.getDate()}`] = newPosition;
+
         const updatedVal = {
            position: newPosition,
            updating: false,
            url: udpatedkeyword.url,
            lastResult: udpatedkeyword.result,
            history,
-            lastUpdated: udpatedkeyword.error ? keyword.lastUpdated : new Date().toJSON(),
-            lastUpdateError: udpatedkeyword.error ? new Date().toJSON() : 'false',
+            lastUpdated: udpatedkeyword.error ? keyword.lastUpdated : theDate.toJSON(),
+            lastUpdateError: udpatedkeyword.error
+               ? JSON.stringify({ date: theDate.toJSON(), error: `${udpatedkeyword.error}`, scraper: settings.scraper_type })
+               : 'false',
         };
-         updatedKeywords.push({ ...keyword, ...updatedVal });
+         updatedKeywords.push({ ...keyword, ...{ ...updatedVal, lastUpdateError: JSON.parse(updatedVal.lastUpdateError) } });

         // If failed, Add to Retry Queue Cron
         if (udpatedkeyword.error) {
@@ -103,7 +106,7 @@ export const refreshAndUpdateKeywords = async (initKeywords:Keyword[], settings:
         try {
            await keywordRaw.update({
               ...updatedVal,
-               lastResult: JSON.stringify(udpatedkeyword.result),
+               lastResult: Array.isArray(udpatedkeyword.result) ? JSON.stringify(udpatedkeyword.result) : udpatedkeyword.result,
               history: JSON.stringify(history),
            });
            console.log('[SUCCESS] Updating the Keyword: ', keyword.keyword);
--- a/pages/index.tsx
+++ b/pages/index.tsx
@@ -2,13 +2,10 @@ import type { NextPage } from 'next';
 import { useEffect, useState } from 'react';
 import Head from 'next/head';
 import { useRouter } from 'next/router';
-// import { useEffect, useState } from 'react';
 import { Toaster } from 'react-hot-toast';
 import Icon from '../components/common/Icon';
 import AddDomain from '../components/domains/AddDomain';

-// import verifyUser from '../utils/verifyUser';
-
 const Home: NextPage = () => {
   const [loading, setLoading] = useState<boolean>(false);
   const [domains, setDomains] = useState<Domain[]>([]);
@@ -54,32 +51,4 @@ const Home: NextPage = () => {
  );
 };

-// export const getServerSideProps = async (context:NextPageContext) => {
-//    const { req, res } = context;
-//    const authorized = verifyUser(req as NextApiRequest, res as NextApiResponse);
-//    // console.log('####### authorized: ', authorized);
-
-//    if (authorized !== 'authorized') {
-//      return { redirect: { destination: '/login', permanent: false } };
-//    }
-
-//    let domains: Domain[] = [];
-//    try {
-//       const fetchOpts = { method: 'GET', headers: { Authorization: `Bearer ${process.env.APIKEY}` } };
-//       const domainsRes = await fetch(`${process.env.NEXT_PUBLIC_APP_URL}/api/domains`, fetchOpts).then((result) => result.json());
-//       // console.log(domainsRes);
-
-//       domains = domainsRes.domains;
-//       if (domains.length > 0) {
-//          const firstDomainItem = domains[0].slug;
-//          return { redirect: { destination: `/domain/${firstDomainItem}`, permanent: false } };
-//       }
-//    } catch (error) {
-//       console.log(error);
-//    }
-
-//    // console.log('domains: ', domains);
-//    return { props: { authorized, domains } };
-// };
-
 export default Home;
--- a/postcss.config.js
+++ b/postcss.config.js
@@ -3,4 +3,4 @@ module.exports = {
    tailwindcss: {},
    autoprefixer: {},
  },
-}
+};
--- a/styles/Home.module.css
+++ b/styles/Home.module.css
@@ -1,129 +0,0 @@
-.container {
-  padding: 0 2rem;
-}
-
-.main {
-  min-height: 100vh;
-  padding: 4rem 0;
-  flex: 1;
-  display: flex;
-  flex-direction: column;
-  justify-content: center;
-  align-items: center;
-}
-
-.footer {
-  display: flex;
-  flex: 1;
-  padding: 2rem 0;
-  border-top: 1px solid #eaeaea;
-  justify-content: center;
-  align-items: center;
-}
-
-.footer a {
-  display: flex;
-  justify-content: center;
-  align-items: center;
-  flex-grow: 1;
-}
-
-.title a {
-  color: #0070f3;
-  text-decoration: none;
-}
-
-.title a:hover,
-.title a:focus,
-.title a:active {
-  text-decoration: underline;
-}
-
-.title {
-  margin: 0;
-  line-height: 1.15;
-  font-size: 4rem;
-}
-
-.title,
-.description {
-  text-align: center;
-}
-
-.description {
-  margin: 4rem 0;
-  line-height: 1.5;
-  font-size: 1.5rem;
-}
-
-.code {
-  background: #fafafa;
-  border-radius: 5px;
-  padding: 0.75rem;
-  font-size: 1.1rem;
-  font-family: Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono,
-    Bitstream Vera Sans Mono, Courier New, monospace;
-}
-
-.grid {
-  display: flex;
-  align-items: center;
-  justify-content: center;
-  flex-wrap: wrap;
-  max-width: 800px;
-}
-
-.card {
-  margin: 1rem;
-  padding: 1.5rem;
-  text-align: left;
-  color: inherit;
-  text-decoration: none;
-  border: 1px solid #eaeaea;
-  border-radius: 10px;
-  transition: color 0.15s ease, border-color 0.15s ease;
-  max-width: 300px;
-}
-
-.card:hover,
-.card:focus,
-.card:active {
-  color: #0070f3;
-  border-color: #0070f3;
-}
-
-.card h2 {
-  margin: 0 0 1rem 0;
-  font-size: 1.5rem;
-}
-
-.card p {
-  margin: 0;
-  font-size: 1.25rem;
-  line-height: 1.5;
-}
-
-.logo {
-  height: 1em;
-  margin-left: 0.5rem;
-}
-
-@media (max-width: 600px) {
-  .grid {
-    width: 100%;
-    flex-direction: column;
-  }
-}
-
-@media (prefers-color-scheme: dark) {
-  .card,
-  .footer {
-    border-color: #222;
-  }
-  .code {
-    background: #111;
-  }
-  .logo img {
-    filter: invert(1);
-  }
-}
--- a/styles/fflag.css
+++ b/styles/fflag.css
@@ -1,7 +1,6 @@
-
 .fflag {
-   background-image:url('../public/flagSprite42.png');
-   background-repeat:no-repeat;
+   background-image: url("../public/flagSprite42.png");
+   background-repeat: no-repeat;
   background-size: 100% 49494%;
   display: inline-block;
   overflow: hidden;
@@ -9,248 +8,249 @@
   vertical-align: middle;
   box-sizing: content-box;
 }
+
 .fflag-CH,
-.fflag-NP {box-shadow: none!important}
-.fflag-DZ {background-position:center 0.2287%}
-.fflag-AO {background-position:center 0.4524%}
-.fflag-BJ {background-position:center 0.6721%}
-.fflag-BW {background-position:center 0.8958%}
-.fflag-BF {background-position:center 1.1162%}
-.fflag-BI {background-position:center 1.3379%}
-.fflag-CM {background-position:center 1.5589%}
-.fflag-CV {background-position:center 1.7805%}
-.fflag-CF {background-position:center 2.0047%}
-.fflag-TD {background-position:center 2.2247%}
-.fflag-CD {background-position:left 2.4467%}
-.fflag-DJ {background-position:left 2.6674%}
-.fflag-EG {background-position:center 2.8931%}
-.fflag-GQ {background-position:center 3.1125%}
-.fflag-ER {background-position:left 3.3325%}
-.fflag-ET {background-position:center 3.5542%}
-.fflag-GA {background-position:center 3.7759%}
-.fflag-GM {background-position:center 4.0015%}
-.fflag-GH {background-position:center 4.2229%}
-.fflag-GN {background-position:center 4.441%}
-.fflag-GW {background-position:left 4.66663%}
-.fflag-CI {background-position:center 4.8844%}
-.fflag-KE {background-position:center 5.1061%}
-.fflag-LS {background-position:center 5.3298%}
-.fflag-LR {background-position:left 5.5495%}
-.fflag-LY {background-position:center 5.7712%}
-.fflag-MG {background-position:center 5.994%}
-.fflag-MW {background-position:center 6.2156%}
-.fflag-ML {background-position:center 6.4363%}
-.fflag-MR {background-position:center 6.658%}
-.fflag-MU {background-position:center 6.8805%}
-.fflag-YT {background-position:center 7.1038%}
-.fflag-MA {background-position:center 7.3231%}
-.fflag-MZ {background-position:left 7.5448%}
-.fflag-NA {background-position:left 7.7661%}
-.fflag-NE {background-position:center 7.98937%}
-.fflag-NG {background-position:center 8.2099%}
-.fflag-CG {background-position:center 8.4316%}
-.fflag-RE {background-position:center 8.6533%}
-.fflag-RW {background-position:right 8.875%}
-.fflag-SH {background-position:center 9.0967%}
-.fflag-ST {background-position:center 9.32237%}
-.fflag-SN {background-position:center 9.5426%}
-.fflag-SC {background-position:left 9.7628%}
-.fflag-SL {background-position:center 9.9845%}
-.fflag-SO {background-position:center 10.2052%}
-.fflag-ZA {background-position:left 10.4269%}
-.fflag-SS {background-position:left 10.6486%}
-.fflag-SD {background-position:center 10.8703%}
-.fflag-SR {background-position:center 11.0945%}
-.fflag-SZ {background-position:center 11.3135%}
-.fflag-TG {background-position:left 11.5354%}
-.fflag-TN {background-position:center 11.7593%}
-.fflag-UG {background-position:center 11.9799%}
-.fflag-TZ {background-position:center 12.2005%}
-.fflag-EH {background-position:center 12.4222%}
-.fflag-YE {background-position:center 12.644%}
-.fflag-ZM {background-position:center 12.8664%}
-.fflag-ZW {background-position:left 13.0873%}
-.fflag-AI {background-position:center 13.309%}
-.fflag-AG {background-position:center 13.5307%}
-.fflag-AR {background-position:center 13.7524%}
-.fflag-AW {background-position:left 13.9741%}
-.fflag-BS {background-position:left 14.1958%}
-.fflag-BB {background-position:center 14.4175%}
-.fflag-BQ {background-position:center 14.6415%}
-.fflag-BZ {background-position:center 14.8609%}
-.fflag-BM {background-position:center 15.0826%}
-.fflag-BO {background-position:center 15.306%}
-.fflag-VG {background-position:center 15.528%}
-.fflag-BR {background-position:center 15.7496%}
-.fflag-CA {background-position:center 15.9694%}
-.fflag-KY {background-position:center 16.1911%}
-.fflag-CL {background-position:left 16.4128%}
-.fflag-CO {background-position:left 16.6345%}
-.fflag-KM {background-position:center 16.8562%}
-.fflag-CR {background-position:center 17.0779%}
-.fflag-CU {background-position:left 17.2996%}
-.fflag-CW {background-position:center 17.5213%}
-.fflag-DM {background-position:center 17.743%}
-.fflag-DO {background-position:center 17.968%}
-.fflag-EC {background-position:center 18.1864%}
-.fflag-SV {background-position:center 18.4081%}
-.fflag-FK {background-position:center 18.6298%}
-.fflag-GF {background-position:center 18.8515%}
-.fflag-GL {background-position:left 19.0732%}
-.fflag-GD {background-position:center 19.2987%}
-.fflag-GP {background-position:center 19.518%}
-.fflag-GT {background-position:center 19.7383%}
-.fflag-GY {background-position:center 19.96%}
-.fflag-HT {background-position:center 20.1817%}
-.fflag-HN {background-position:center 20.4034%}
-.fflag-JM {background-position:center 20.6241%}
-.fflag-MQ {background-position:center 20.8468%}
-.fflag-MX {background-position:center 21.0685%}
-.fflag-MS {background-position:center 21.2902%}
-.fflag-NI {background-position:center 21.5119%}
-.fflag-PA {background-position:center 21.7336%}
-.fflag-PY {background-position:center 21.9553%}
-.fflag-PE {background-position:center 22.177%}
-.fflag-PR {background-position:left 22.4002%}
-.fflag-BL {background-position:center 22.6204%}
-.fflag-KN {background-position:center 22.8421%}
-.fflag-LC {background-position:center 23.0638%}
-.fflag-PM {background-position:center 23.2855%}
-.fflag-VC {background-position:center 23.5072%}
-.fflag-SX {background-position:left 23.732%}
-.fflag-TT {background-position:center 23.9506%}
-.fflag-TC {background-position:center 24.1723%}
-.fflag-US {background-position:center 24.394%}
-.fflag-VI {background-position:center 24.6157%}
-.fflag-UY {background-position:left 24.8374%}
-.fflag-VE {background-position:center 25.0591%}
-.fflag-AB {background-position:center 25.279%}
-.fflag-AF {background-position:center 25.5025%}
-.fflag-AZ {background-position:center 25.7242%}
-.fflag-BD {background-position:center 25.9459%}
-.fflag-BT {background-position:center 26.1676%}
-.fflag-BN {background-position:center 26.3885%}
-.fflag-KH {background-position:center 26.611%}
-.fflag-CN {background-position:left 26.8327%}
-.fflag-GE {background-position:center 27.0544%}
-.fflag-HK {background-position:center 27.2761%}
-.fflag-IN {background-position:center 27.4978%}
-.fflag-ID {background-position:center 27.7195%}
-.fflag-JP {background-position:center 27.9412%}
-.fflag-KZ {background-position:center 28.1615%}
-.fflag-LA {background-position:center 28.3846%}
-.fflag-MO {background-position:center 28.6063%}
-.fflag-MY {background-position:center 28.829%}
-.fflag-MV {background-position:center 29.0497%}
-.fflag-MN {background-position:left 29.2714%}
-.fflag-MM {background-position:center 29.4931%}
-.fflag-NP {background-position:left 29.7148%}
-.fflag-KP {background-position:left 29.9365%}
-.fflag-MP {background-position:center 30.1582%}
-.fflag-PW {background-position:center 30.3799%}
-.fflag-PG {background-position:center 30.6016%}
-.fflag-PH {background-position:left 30.8233%}
-.fflag-SG {background-position:left 31.045%}
-.fflag-KR {background-position:center 31.2667%}
-.fflag-LK {background-position:right 31.4884%}
-.fflag-TW {background-position:left 31.7101%}
-.fflag-TJ {background-position:center 31.9318%}
-.fflag-TH {background-position:center 32.1535%}
-.fflag-TL {background-position:left 32.3752%}
-.fflag-TM {background-position:center 32.5969%}
-.fflag-VN {background-position:center 32.8186%}
-.fflag-AL {background-position:center 33.0403%}
-.fflag-AD {background-position:center 33.25975%}
-.fflag-AM {background-position:center 33.4837%}
-.fflag-AT {background-position:center 33.7054%}
-.fflag-BY {background-position:left 33.9271%}
-.fflag-BE {background-position:center 34.1488%}
-.fflag-BA {background-position:center 34.3705%}
-.fflag-BG {background-position:center 34.5922%}
-.fflag-HR {background-position:center 34.8139%}
-.fflag-CY {background-position:center 35.0356%}
-.fflag-CZ {background-position:left 35.2555%}
-.fflag-DK {background-position:center 35.479%}
-.fflag-EE {background-position:center 35.7007%}
-.fflag-FO {background-position:center 35.9224%}
-.fflag-FI {background-position:center 36.1441%}
-.fflag-FR {background-position:center 36.3658%}
-.fflag-DE {background-position:center 36.5875%}
-.fflag-GI {background-position:center 36.8092%}
-.fflag-GR {background-position:left 37.0309%}
-.fflag-GG {background-position:center 37.2526%}
-.fflag-HU {background-position:center 37.4743%}
-.fflag-IS {background-position:center 37.696%}
-.fflag-IE {background-position:center 37.9177%}
-.fflag-IM {background-position:center 38.1394%}
-.fflag-IT {background-position:center 38.3611%}
-.fflag-JE {background-position:center 38.5828%}
-.fflag-XK {background-position:center 38.8045%}
-.fflag-LV {background-position:center 39.0262%}
-.fflag-LI {background-position:left 39.2479%}
-.fflag-LT {background-position:center 39.4696%}
-.fflag-LU {background-position:center 39.6913%}
-.fflag-MT {background-position:left 39.913%}
-.fflag-MD {background-position:center 40.1347%}
-.fflag-MC {background-position:center 40.3564%}
-.fflag-ME {background-position:center 40.5781%}
-.fflag-NL {background-position:center 40.7998%}
-.fflag-MK {background-position:center 41.0215%}
-.fflag-NO {background-position:center 41.2432%}
-.fflag-PL {background-position:center 41.4649%}
-.fflag-PT {background-position:center 41.6866%}
-.fflag-RO {background-position:center 41.9083%}
-.fflag-RU {background-position:center 42.13%}
-.fflag-SM {background-position:center 42.3517%}
-.fflag-RS {background-position:center 42.5734%}
-.fflag-SK {background-position:center 42.7951%}
-.fflag-SI {background-position:center 43.0168%}
-.fflag-ES {background-position:left 43.2385%}
-.fflag-SE {background-position:center 43.4602%}
-.fflag-CH {background-position:center 43.6819%}
-.fflag-TR {background-position:center 43.9036%}
-.fflag-UA {background-position:center 44.1253%}
-.fflag-GB {background-position:center 44.347%}
-.fflag-VA {background-position:right 44.5687%}
-.fflag-BH {background-position:center 44.7904%}
-.fflag-IR {background-position:center 45.0121%}
-.fflag-IQ {background-position:center 45.2338%}
-.fflag-IL {background-position:center 45.4555%}
-.fflag-KW {background-position:left 45.6772%}
-.fflag-JO {background-position:left 45.897%}
-.fflag-KG {background-position:center 46.1206%}
-.fflag-LB {background-position:center 46.3423%}
-.fflag-OM {background-position:left 46.561%}
-.fflag-PK {background-position:center 46.7857%}
-.fflag-PS {background-position:center 47.0074%}
-.fflag-QA {background-position:center 47.2291%}
-.fflag-SA {background-position:center 47.4508%}
-.fflag-SY {background-position:center 47.6725%}
-.fflag-AE {background-position:center 47.8942%}
-.fflag-UZ {background-position:left 48.1159%}
-.fflag-AS {background-position:right 48.3376%}
-.fflag-AU {background-position:center 48.5593%}
-.fflag-CX {background-position:center 48.781%}
-.fflag-CC {background-position:center 49.002%}
-.fflag-CK {background-position:center 49.2244%}
-.fflag-FJ {background-position:center 49.4445%}
-.fflag-PF {background-position:center 49.6678%}
-.fflag-GU {background-position:center 49.8895%}
-.fflag-KI {background-position:center 50.1112%}
-.fflag-MH {background-position:left 50.3329%}
-.fflag-FM {background-position:center 50.5546%}
-.fflag-NC {background-position:center 50.7763%}
-.fflag-NZ {background-position:center 50.998%}
-.fflag-NR {background-position:left 51.2197%}
-.fflag-NU {background-position:center 51.4414%}
-.fflag-NF {background-position:center 51.6631%}
-.fflag-WS {background-position:left 51.8848%}
-.fflag-SB {background-position:left 52.1065%}
-.fflag-TK {background-position:center 52.3282%}
-.fflag-TO {background-position:left 52.5499%}
-.fflag-TV {background-position:center 52.7716%}
-.fflag-VU {background-position:left 52.9933%}
-.fflag-WF {background-position:center 53.215%}
+.fflag-DZ { background-position: center 0.2287%; }
+.fflag-AO { background-position: center 0.4524%; }
+.fflag-BJ { background-position: center 0.6721%; }
+.fflag-BW { background-position: center 0.8958%; }
+.fflag-BF { background-position: center 1.1162%; }
+.fflag-BI { background-position: center 1.3379%; }
+.fflag-CM { background-position: center 1.5589%; }
+.fflag-CV { background-position: center 1.7805%; }
+.fflag-CF { background-position: center 2.0047%; }
+.fflag-TD { background-position: center 2.2247%; }
+.fflag-CD { background-position: left 2.4467%; }
+.fflag-DJ { background-position: left 2.6674%; }
+.fflag-EG { background-position: center 2.8931%; }
+.fflag-GQ { background-position: center 3.1125%; }
+.fflag-ER { background-position: left 3.3325%; }
+.fflag-ET { background-position: center 3.5542%; }
+.fflag-GA { background-position: center 3.7759%; }
+.fflag-GM { background-position: center 4.0015%; }
+.fflag-GH { background-position: center 4.2229%; }
+.fflag-GN { background-position: center 4.441%; }
+.fflag-GW { background-position: left 4.6666%; }
+.fflag-CI { background-position: center 4.8844%; }
+.fflag-KE { background-position: center 5.1061%; }
+.fflag-LS { background-position: center 5.3298%; }
+.fflag-LR { background-position: left 5.5495%; }
+.fflag-LY { background-position: center 5.7712%; }
+.fflag-MG { background-position: center 5.994%; }
+.fflag-MW { background-position: center 6.2156%; }
+.fflag-ML { background-position: center 6.4363%; }
+.fflag-MR { background-position: center 6.658%; }
+.fflag-MU { background-position: center 6.8805%; }
+.fflag-YT { background-position: center 7.1038%; }
+.fflag-MA { background-position: center 7.3231%; }
+.fflag-MZ { background-position: left 7.5448%; }
+.fflag-NA { background-position: left 7.7661%; }
+.fflag-NE { background-position: center 7.9894%; }
+.fflag-NG { background-position: center 8.2099%; }
+.fflag-CG { background-position: center 8.4316%; }
+.fflag-RE { background-position: center 8.6533%; }
+.fflag-RW { background-position: right 8.875%; }
+.fflag-SH { background-position: center 9.0967%; }
+.fflag-ST { background-position: center 9.3224%; }
+.fflag-SN { background-position: center 9.5426%; }
+.fflag-SC { background-position: left 9.7628%; }
+.fflag-SL { background-position: center 9.9845%; }
+.fflag-SO { background-position: center 10.2052%; }
+.fflag-ZA { background-position: left 10.4269%; }
+.fflag-SS { background-position: left 10.6486%; }
+.fflag-SD { background-position: center 10.8703%; }
+.fflag-SR { background-position: center 11.0945%; }
+.fflag-SZ { background-position: center 11.3135%; }
+.fflag-TG { background-position: left 11.5354%; }
+.fflag-TN { background-position: center 11.7593%; }
+.fflag-UG { background-position: center 11.9799%; }
+.fflag-TZ { background-position: center 12.2005%; }
+.fflag-EH { background-position: center 12.4222%; }
+.fflag-YE { background-position: center 12.644%; }
+.fflag-ZM { background-position: center 12.8664%; }
+.fflag-ZW { background-position: left 13.0873%; }
+.fflag-AI { background-position: center 13.309%; }
+.fflag-AG { background-position: center 13.5307%; }
+.fflag-AR { background-position: center 13.7524%; }
+.fflag-AW { background-position: left 13.9741%; }
+.fflag-BS { background-position: left 14.1958%; }
+.fflag-BB { background-position: center 14.4175%; }
+.fflag-BQ { background-position: center 14.6415%; }
+.fflag-BZ { background-position: center 14.8609%; }
+.fflag-BM { background-position: center 15.0826%; }
+.fflag-BO { background-position: center 15.306%; }
+.fflag-VG { background-position: center 15.528%; }
+.fflag-BR { background-position: center 15.7496%; }
+.fflag-CA { background-position: center 15.9694%; }
+.fflag-KY { background-position: center 16.1911%; }
+.fflag-CL { background-position: left 16.4128%; }
+.fflag-CO { background-position: left 16.6345%; }
+.fflag-KM { background-position: center 16.8562%; }
+.fflag-CR { background-position: center 17.0779%; }
+.fflag-CU { background-position: left 17.2996%; }
+.fflag-CW { background-position: center 17.5213%; }
+.fflag-DM { background-position: center 17.743%; }
+.fflag-DO { background-position: center 17.968%; }
+.fflag-EC { background-position: center 18.1864%; }
+.fflag-SV { background-position: center 18.4081%; }
+.fflag-FK { background-position: center 18.6298%; }
+.fflag-GF { background-position: center 18.8515%; }
+.fflag-GL { background-position: left 19.0732%; }
+.fflag-GD { background-position: center 19.2987%; }
+.fflag-GP { background-position: center 19.518%; }
+.fflag-GT { background-position: center 19.7383%; }
+.fflag-GY { background-position: center 19.96%; }
+.fflag-HT { background-position: center 20.1817%; }
+.fflag-HN { background-position: center 20.4034%; }
+.fflag-JM { background-position: center 20.6241%; }
+.fflag-MQ { background-position: center 20.8468%; }
+.fflag-MX { background-position: center 21.0685%; }
+.fflag-MS { background-position: center 21.2902%; }
+.fflag-NI { background-position: center 21.5119%; }
+.fflag-PA { background-position: center 21.7336%; }
+.fflag-PY { background-position: center 21.9553%; }
+.fflag-PE { background-position: center 22.177%; }
+.fflag-PR { background-position: left 22.4002%; }
+.fflag-BL { background-position: center 22.6204%; }
+.fflag-KN { background-position: center 22.8421%; }
+.fflag-LC { background-position: center 23.0638%; }
+.fflag-PM { background-position: center 23.2855%; }
+.fflag-VC { background-position: center 23.5072%; }
+.fflag-SX { background-position: left 23.732%; }
+.fflag-TT { background-position: center 23.9506%; }
+.fflag-TC { background-position: center 24.1723%; }
+.fflag-US { background-position: center 24.394%; }
+.fflag-VI { background-position: center 24.6157%; }
+.fflag-UY { background-position: left 24.8374%; }
+.fflag-VE { background-position: center 25.0591%; }
+.fflag-AB { background-position: center 25.279%; }
+.fflag-AF { background-position: center 25.5025%; }
+.fflag-AZ { background-position: center 25.7242%; }
+.fflag-BD { background-position: center 25.9459%; }
+.fflag-BT { background-position: center 26.1676%; }
+.fflag-BN { background-position: center 26.3885%; }
+.fflag-KH { background-position: center 26.611%; }
+.fflag-CN { background-position: left 26.8327%; }
+.fflag-GE { background-position: center 27.0544%; }
+.fflag-HK { background-position: center 27.2761%; }
+.fflag-IN { background-position: center 27.4978%; }
+.fflag-ID { background-position: center 27.7195%; }
+.fflag-JP { background-position: center 27.9412%; }
+.fflag-KZ { background-position: center 28.1615%; }
+.fflag-LA { background-position: center 28.3846%; }
+.fflag-MO { background-position: center 28.6063%; }
+.fflag-MY { background-position: center 28.829%; }
+.fflag-MV { background-position: center 29.0497%; }
+.fflag-MN { background-position: left 29.2714%; }
+.fflag-MM { background-position: center 29.4931%; }
+.fflag-NP { background-position: left 29.7148%; }
+.fflag-KP { background-position: left 29.9365%; }
+.fflag-MP { background-position: center 30.1582%; }
+.fflag-PW { background-position: center 30.3799%; }
+.fflag-PG { background-position: center 30.6016%; }
+.fflag-PH { background-position: left 30.8233%; }
+.fflag-SG { background-position: left 31.045%; }
+.fflag-KR { background-position: center 31.2667%; }
+.fflag-LK { background-position: right 31.4884%; }
+.fflag-TW { background-position: left 31.7101%; }
+.fflag-TJ { background-position: center 31.9318%; }
+.fflag-TH { background-position: center 32.1535%; }
+.fflag-TL { background-position: left 32.3752%; }
+.fflag-TM { background-position: center 32.5969%; }
+.fflag-VN { background-position: center 32.8186%; }
+.fflag-AL { background-position: center 33.0403%; }
+.fflag-AD { background-position: center 33.2597%; }
+.fflag-AM { background-position: center 33.4837%; }
+.fflag-AT { background-position: center 33.7054%; }
+.fflag-BY { background-position: left 33.9271%; }
+.fflag-BE { background-position: center 34.1488%; }
+.fflag-BA { background-position: center 34.3705%; }
+.fflag-BG { background-position: center 34.5922%; }
+.fflag-HR { background-position: center 34.8139%; }
+.fflag-CY { background-position: center 35.0356%; }
+.fflag-CZ { background-position: left 35.2555%; }
+.fflag-DK { background-position: center 35.479%; }
+.fflag-EE { background-position: center 35.7007%; }
+.fflag-FO { background-position: center 35.9224%; }
+.fflag-FI { background-position: center 36.1441%; }
+.fflag-FR { background-position: center 36.3658%; }
+.fflag-DE { background-position: center 36.5875%; }
+.fflag-GI { background-position: center 36.8092%; }
+.fflag-GR { background-position: left 37.0309%; }
+.fflag-GG { background-position: center 37.2526%; }
+.fflag-HU { background-position: center 37.4743%; }
+.fflag-IS { background-position: center 37.696%; }
+.fflag-IE { background-position: center 37.9177%; }
+.fflag-IM { background-position: center 38.1394%; }
+.fflag-IT { background-position: center 38.3611%; }
+.fflag-JE { background-position: center 38.5828%; }
+.fflag-XK { background-position: center 38.8045%; }
+.fflag-LV { background-position: center 39.0262%; }
+.fflag-LI { background-position: left 39.2479%; }
+.fflag-LT { background-position: center 39.4696%; }
+.fflag-LU { background-position: center 39.6913%; }
+.fflag-MT { background-position: left 39.913%; }
+.fflag-MD { background-position: center 40.1347%; }
+.fflag-MC { background-position: center 40.3564%; }
+.fflag-ME { background-position: center 40.5781%; }
+.fflag-NL { background-position: center 40.7998%; }
+.fflag-MK { background-position: center 41.0215%; }
+.fflag-NO { background-position: center 41.2432%; }
+.fflag-PL { background-position: center 41.4649%; }
+.fflag-PT { background-position: center 41.6866%; }
+.fflag-RO { background-position: center 41.9083%; }
+.fflag-RU { background-position: center 42.13%; }
+.fflag-SM { background-position: center 42.3517%; }
+.fflag-RS { background-position: center 42.5734%; }
+.fflag-SK { background-position: center 42.7951%; }
+.fflag-SI { background-position: center 43.0168%; }
+.fflag-ES { background-position: left 43.2385%; }
+.fflag-SE { background-position: center 43.4602%; }
+.fflag-CH { background-position: center 43.6819%; }
+.fflag-TR { background-position: center 43.9036%; }
+.fflag-UA { background-position: center 44.1253%; }
+.fflag-GB { background-position: center 44.347%; }
+.fflag-VA { background-position: right 44.5687%; }
+.fflag-BH { background-position: center 44.7904%; }
+.fflag-IR { background-position: center 45.0121%; }
+.fflag-IQ { background-position: center 45.2338%; }
+.fflag-IL { background-position: center 45.4555%; }
+.fflag-KW { background-position: left 45.6772%; }
+.fflag-JO { background-position: left 45.897%; }
+.fflag-KG { background-position: center 46.1206%; }
+.fflag-LB { background-position: center 46.3423%; }
+.fflag-OM { background-position: left 46.561%; }
+.fflag-PK { background-position: center 46.7857%; }
+.fflag-PS { background-position: center 47.0074%; }
+.fflag-QA { background-position: center 47.2291%; }
+.fflag-SA { background-position: center 47.4508%; }
+.fflag-SY { background-position: center 47.6725%; }
+.fflag-AE { background-position: center 47.8942%; }
+.fflag-UZ { background-position: left 48.1159%; }
+.fflag-AS { background-position: right 48.3376%; }
+.fflag-AU { background-position: center 48.5593%; }
+.fflag-CX { background-position: center 48.781%; }
+.fflag-CC { background-position: center 49.002%; }
+.fflag-CK { background-position: center 49.2244%; }
+.fflag-FJ { background-position: center 49.4445%; }
+.fflag-PF { background-position: center 49.6678%; }
+.fflag-GU { background-position: center 49.8895%; }
+.fflag-KI { background-position: center 50.1112%; }
+.fflag-MH { background-position: left 50.3329%; }
+.fflag-FM { background-position: center 50.5546%; }
+.fflag-NC { background-position: center 50.7763%; }
+.fflag-NZ { background-position: center 50.998%; }
+.fflag-NR { background-position: left 51.2197%; }
+.fflag-NU { background-position: center 51.4414%; }
+.fflag-NF { background-position: center 51.6631%; }
+.fflag-WS { background-position: left 51.8848%; }
+.fflag-SB { background-position: left 52.1065%; }
+.fflag-TK { background-position: center 52.3282%; }
+.fflag-TO { background-position: left 52.5499%; }
+.fflag-TV { background-position: center 52.7716%; }
+.fflag-VU { background-position: left 52.9933%; }
+.fflag-WF { background-position: center 53.215%; }
+
 .fflag-TD.ff-round,
 .fflag-GN.ff-round,
 .fflag-CI.ff-round,
@@ -260,20 +260,21 @@
 .fflag-FR.ff-round,
 .fflag-IE.ff-round,
 .fflag-IT.ff-round,
-.fflag-RO.ff-round {background-size:100%  50000%}
+.fflag-RO.ff-round { background-size: 100%  50000%; }

-.fflag.ff-sm {width: 18px;height: 11px}
-.fflag.ff-md {width: 27px;height: 17px}
-.fflag.ff-lg {width: 42px;height: 27px}
-.fflag.ff-xl {width: 60px;height: 37px}
+.fflag.ff-sm { width: 18px; height: 11px }
+.fflag.ff-md { width: 27px; height: 17px }
+.fflag.ff-lg { width: 42px; height: 27px }
+.fflag.ff-xl { width: 60px; height: 37px }

 /* ff-round = circular icons */
 .ff-round {
-  background-size: 160%;
-  background-clip: content-box;
-  border-radius: 50%;
+   background-size: 160%;
+   background-clip: content-box;
+   border-radius: 50%;
 }
-.ff-round.ff-sm {width: 12px; height: 12px}
-.ff-round.ff-md {width: 18px; height: 18px}
-.ff-round.ff-lg {width: 24px; height: 24px}
-.ff-round.ff-xl {width: 32px; height: 32px}
+
+.ff-round.ff-sm { width: 12px; height: 12px }
+.ff-round.ff-md { width: 18px; height: 18px }
+.ff-round.ff-lg { width: 24px; height: 24px }
+.ff-round.ff-xl { width: 32px; height: 32px }
--- a/styles/globals.css
+++ b/styles/globals.css
@@ -1,114 +1,129 @@
+@import url("./fflag.css");
+
@tailwind base;
@tailwind components;
@tailwind utilities;
-@import url('./fflag.css');

-body{
+body {
   background-color: #f8f9ff;
 }

-.domKeywords{
+.domKeywords {
   min-height: 70vh;
-   border-color: #E9EBFF;
-   box-shadow: 0 0 20px rgba(20, 34, 71, 0.05);
+   border-color: #e9ebff;
+   box-shadow: 0 0 20px rgb(20 34 71 / 5%);
 }

-
-.customShadow{
-   border-color: #E9EBFF;
-   box-shadow: 0 0 20px rgba(20, 34, 71, 0.05);
+.customShadow {
+   border-color: #e9ebff;
+   box-shadow: 0 0 20px rgb(20 34 71 / 5%);
 }

 .styled-scrollbar {
   scrollbar-color: #d6dbec transparent;
   scrollbar-width: thin;
 }
+
 .styled-scrollbar::-webkit-scrollbar {
   width: 6px;
   height: 6px;
-   border-radius: 0px;
+   border-radius: 0;
   background: #f5f7ff;
   margin-right: 4px;
-   border: 0px solid transparent;
+   border: 0 solid transparent;
 }
-   
+
 .styled-scrollbar::-webkit-scrollbar-thumb {
   width: 6px;
   height: 6px;
-   border-radius: 0px;
+   border-radius: 0;
   color: #d6dbec;
-   background:  #d6d8e1;
-   border: 0px solid transparent;
+   background: #d6d8e1;
+   border: 0 solid transparent;
   box-shadow: none;
 }

 .ct-area {
   fill: #10b98d73;
 }
+
 .ct-label.ct-horizontal {
   font-size: 11px;
 }
+
 .ct-label.ct-vertical {
   font-size: 12px;
 }
-.chart_tooltip{
-   width: 95px; 
-   height: 75px; 
+
+.chart_tooltip {
+   width: 95px;
+   height: 75px;
   background-color: white;
-   position: absolute; 
-   display: none; 
-   padding: 0 8px; 
-   box-sizing: border-box; 
-   font-size: 12px; 
-   text-align: left; 
-   z-index: 1000; 
-   top: 12px; 
-   left: 12px; 
-   pointer-events: none; 
-   border: 1px solid; 
+   position: absolute;
+   display: none;
+   padding: 0 8px;
+   box-sizing: border-box;
+   font-size: 12px;
+   text-align: left;
+   z-index: 1000;
+   top: 12px;
+   left: 12px;
+   pointer-events: none;
+   border: 1px solid;
   border-radius: 4px;
-    border-color: #a7aed3;
-    box-shadow: 0 0 10px rgb(0 0 0 / 12%);
-   font-family: 'Trebuchet MS', Roboto, Ubuntu, sans-serif;
+   border-color: #a7aed3;
+   box-shadow: 0 0 10px rgb(0 0 0 / 12%);
+   font-family: "Trebuchet MS", Roboto, Ubuntu, sans-serif;
   -webkit-font-smoothing: antialiased;
   -moz-osx-font-smoothing: grayscale;
 }

-
-.react_toaster{
+.react_toaster {
   font-size: 13px;
 }

-.domKeywords_head--alpha_desc .domKeywords_head_keyword:after,
-.domKeywords_head--pos_desc .domKeywords_head_position:after
-{content: '↓' ; display: inline-block; margin-left: 2px; font-size: 14px; opacity: 0.8;}
+.domKeywords_head--alpha_desc .domKeywords_head_keyword::after,
+.domKeywords_head--pos_desc .domKeywords_head_position::after {
+   content: "↓";
+   display: inline-block;
+   margin-left: 2px;
+   font-size: 14px;
+   opacity: 0.8;
+}

-.domKeywords_head--alpha_asc .domKeywords_head_keyword:after,
-.domKeywords_head--pos_asc .domKeywords_head_position:after
-{content: '↑' ; display: inline-block; margin-left: 2px; font-size: 14px; opacity: 0.8;}
+.domKeywords_head--alpha_asc .domKeywords_head_keyword::after,
+.domKeywords_head--pos_asc .domKeywords_head_position::after {
+   content: "↑";
+   display: inline-block;
+   margin-left: 2px;
+   font-size: 14px;
+   opacity: 0.8;
+}

-.keywordDetails__section__results{
+.keywordDetails__section__results {
   height: calc(100vh - 550px);
 }

-.settings__content{
+.settings__content {
   height: calc(100vh - 185px);
   overflow: auto;
 }

-
 /* Animation */
 .modal_anim-enter {
   opacity: 0;
 }
+
 .modal_anim-enter-active {
   opacity: 1;
   transition: opacity 300ms;
 }
-.modal_anim-enter .modal__content{
+
+.modal_anim-enter .modal__content {
   transform: translateY(50px);
 }
-.modal_anim-enter-active .modal__content{
+
+.modal_anim-enter-active .modal__content {
   transform: translateY(0);
   transition: all 300ms;
 }
@@ -116,34 +131,75 @@ body{
 .modal_anim-exit {
   opacity: 1;
 }
+
 .modal_anim-exit-active {
   opacity: 0;
   transition: all 300ms;
 }
-.modal_anim-exit .modal__content{
-   transform: translateY(0px);
+
+.modal_anim-exit .modal__content {
+   transform: translateY(0);
 }
-.modal_anim-exit-active .modal__content{
+
+.modal_anim-exit-active .modal__content {
   transform: translateY(50px);
   transition: all 300ms;
 }

-
 .settings_anim-enter {
   opacity: 0;
   transform: translateX(400px);
 }
+
 .settings_anim-enter-active {
   opacity: 1;
   transform: translateX(0);
   transition: all 300ms;
 }
+
 .settings_anim-exit {
   opacity: 1;
   transform: translateX(0);
 }
+
 .settings_anim-exit-active {
   opacity: 0;
   transform: translateX(400px);
   transition: all 300ms;
-}
+}
+
+@media (min-width: 1024px) {
+   /* Domain Header Button Tooltips */
+   .domheader_action_button:hover i {
+      visibility: visible;
+      opacity: 1;
+   }
+
+   .domheader_action_button i {
+      display: block;
+      position: absolute;
+      width: 100px;
+      left: -40px;
+      top: -22px;
+      background: #222;
+      border-radius: 3px;
+      color: #fff;
+      font-size: 12px;
+      padding-bottom: 3px;
+      transition: all 0.2s linear;
+   }
+
+   .domheader_action_button i::after {
+      content: "";
+      width: 0;
+      height: 0;
+      border-style: solid;
+      border-width: 5px 5px 0;
+      border-color: #222 transparent transparent;
+      bottom: -5px;
+      position: absolute;
+      left: 0;
+      right: 0;
+      margin: 0 auto;
+   }
+}
--- a/tailwind.config.js
+++ b/tailwind.config.js
@@ -2,23 +2,23 @@
 module.exports = {
   purge: {
      content: [
-         "./pages/**/*.{js,ts,jsx,tsx}",
-         "./components/**/*.{js,ts,jsx,tsx}",
+         './pages/**/*.{js,ts,jsx,tsx}',
+         './components/**/*.{js,ts,jsx,tsx}',
      ],
   },
   content: [
-      "./pages/**/*.{js,ts,jsx,tsx}",
-      "./components/**/*.{js,ts,jsx,tsx}",
+      './pages/**/*.{js,ts,jsx,tsx}',
+      './components/**/*.{js,ts,jsx,tsx}',
   ],
   safelist: [
      'max-h-48',
      'w-[150px]',
      'w-[240px]',
      'min-w-[270px]',
-      'min-w-[180px]'
+      'min-w-[180px]',
    ],
   theme: {
     extend: {},
   },
   plugins: [],
- }
+ };
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -1,21 +1,32 @@
 {
-  "compilerOptions": {
-    "target": "esnext",
-    "lib": ["dom", "dom.iterable", "esnext"],
-    "allowJs": true,
-    "skipLibCheck": true,
-    "strict": true,
-    "forceConsistentCasingInFileNames": true,
-    "noEmit": true,
-    "esModuleInterop": true,
-    "module": "esnext",
-    "moduleResolution": "node",
-    "resolveJsonModule": true,
-    "isolatedModules": true,
-    "jsx": "preserve",
-    "incremental": true,
-    "experimentalDecorators": true,
-  },
-  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", "types.d.ts", "utils/generateEmail__.js"],
-  "exclude": ["node_modules"],
-}
+   "compilerOptions": {
+      "target": "esnext",
+      "lib": [
+         "dom",
+         "dom.iterable",
+         "esnext"
+      ],
+      "allowJs": true,
+      "skipLibCheck": true,
+      "strict": true,
+      "forceConsistentCasingInFileNames": true,
+      "noEmit": true,
+      "esModuleInterop": true,
+      "module": "esnext",
+      "moduleResolution": "node",
+      "resolveJsonModule": true,
+      "isolatedModules": true,
+      "jsx": "preserve",
+      "incremental": true,
+      "experimentalDecorators": true
+   },
+   "include": [
+      "next-env.d.ts",
+      "**/*.ts",
+      "**/*.tsx",
+      "types.d.ts"
+   ],
+   "exclude": [
+      "node_modules"
+   ]
+}
--- a/types.d.ts
+++ b/types.d.ts
@@ -31,7 +31,7 @@ type KeywordType = {
   url: string,
   tags: string[],
   updating: boolean,
-   lastUpdateError: string
+   lastUpdateError: {date: string, error: string, scraper: string} | false
 }

 type KeywordLastResult = {
--- a/utils/generateEmail.ts
+++ b/utils/generateEmail.ts
@@ -24,12 +24,12 @@ const timeSince = (date:number) : string => {
   if (interval >= 1) return `${interval} days ago`;

   interval = Math.floor(seconds / 3600);
-   if (interval >= 1) return `${interval} hours ago`;
+   if (interval >= 1) return `${interval} hrs ago`;

   interval = Math.floor(seconds / 60);
-   if (interval > 1) return `${interval} minutes ago`;
+   if (interval > 1) return `${interval} mins ago`;

-   return `${Math.floor(seconds)} seconds ago`;
+   return `${Math.floor(seconds)} secs ago`;
 };

 /**
@@ -98,9 +98,6 @@ const generateEmail = async (domainName:string, keywords:KeywordType[]) : Promis
         .replace('{{stat}}', stat)
         .replace('{{preheader}}', stat);

-   // const writePath = path.join(__dirname, '..', 'email',  'email_update.html');
-   // await writeFile(writePath, updatedEmail, {encoding:'utf-8'});
-
   return updatedEmail;
 };

--- a/utils/parseKeywords.ts
+++ b/utils/parseKeywords.ts
@@ -11,6 +11,7 @@ const parseKeywords = (allKeywords: Keyword[]) : KeywordType[] => {
         history: JSON.parse(keywrd.history),
         tags: JSON.parse(keywrd.tags),
         lastResult: JSON.parse(keywrd.lastResult),
+         lastUpdateError: keywrd.lastUpdateError !== 'false' && keywrd.lastUpdateError.includes('{') ? JSON.parse(keywrd.lastUpdateError) : false,
      }));
   return parsedItems;
 };
--- a/utils/refresh.ts
+++ b/utils/refresh.ts
@@ -26,7 +26,6 @@ const refreshKeywords = async (keywords:KeywordType[], settings:SettingsType): P

   const end = performance.now();
   console.log(`time taken: ${end - start}ms`);
-   // console.log('refreshedResults: ', refreshedResults);
   return refreshedResults;
 };

--- a/utils/scraper.ts
+++ b/utils/scraper.ts
@@ -20,10 +20,16 @@ type SERPObject = {
 export type RefreshResult = false | {
   ID: number,
   keyword: string,
-   position:number|boolean,
+   position:number | boolean,
   url: string,
   result: SearchResult[],
-   error?: boolean
+   error?: boolean | string
+}
+
+interface SerplyResult {
+   title: string,
+   link: string,
+   realPosition: number,
 }

 /**
@@ -34,7 +40,7 @@ export type RefreshResult = false | {
 */
 export const getScraperClient = (keyword:KeywordType, settings:SettingsType): Promise<AxiosResponse|Response> | false => {
   let apiURL = ''; let client: Promise<AxiosResponse|Response> | false = false;
-   const headers = {
+   const headers: any = {
      'Content-Type': 'application/json',
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246',
      Accept: 'application/json; charset=utf8;',
@@ -56,7 +62,21 @@ export const getScraperClient = (keyword:KeywordType, settings:SettingsType): Pr
   if (settings && settings.scraper_type === 'scrapingrobot' && settings.scaping_api) {
      const country = keyword.country || 'US';
      const lang = countries[country][2];
-      apiURL = `https://api.scrapingrobot.com/?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fnum%3D100%26hl%3D${lang}%26q%3D${encodeURI(keyword.keyword)}&token=${settings.scaping_api}&proxyCountry=${country}&render=false${keyword.device === 'mobile' ? '&mobile=true' : ''}`;
+      apiURL = `https://api.scrapingrobot.com/?token=${settings.scaping_api}&proxyCountry=${country}&render=false${keyword.device === 'mobile' ? '&mobile=true' : ''}&url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fnum%3D100%26hl%3D${lang}%26q%3D${encodeURI(keyword.keyword)}`;
+   }
+
+    // Serply.io docs https://docs.serply.io/api
+   if (settings && settings.scraper_type === 'serply' && settings.scaping_api) {
+      const scraperCountries = ['US', 'CA', 'IE', 'GB', 'FR', 'DE', 'SE', 'IN', 'JP', 'KR', 'SG', 'AU', 'BR'];
+      const country = scraperCountries.includes(keyword.country.toUpperCase()) ? keyword.country : 'US';
+      if (keyword.device === 'mobile') {
+          headers['X-User-Agent'] = 'mobile';
+      } else {
+         headers['X-User-Agent'] = 'desktop';
+      }
+      headers['X-Proxy-Location'] = country;
+      headers['X-Api-Key'] = settings.scaping_api;
+      apiURL = `https://api.serply.io/v1/search/q=${encodeURI(keyword.keyword)}&num=100&hl=${country}`;
   }

   if (settings && settings.scraper_type === 'proxy' && settings.proxy) {
@@ -76,7 +96,7 @@ export const getScraperClient = (keyword:KeywordType, settings:SettingsType): Pr
      const axiosClient = axios.create(axiosConfig);
      client = axiosClient.get(`https://www.google.com/search?num=100&q=${encodeURI(keyword.keyword)}`);
   } else {
-      client = fetch(apiURL, { method: 'GET', headers }).then((res) => res.json());
+      client = fetch(apiURL, { method: 'GET', headers });
   }

   return client;
@@ -100,19 +120,27 @@ export const scrapeKeywordFromGoogle = async (keyword:KeywordType, settings:Sett
   const scraperClient = getScraperClient(keyword, settings);

   if (!scraperClient) { return false; }
-
+   let res:any = null; let scraperError:any = null;
   try {
-      const res:any = await scraperClient;
-      if (res && (res.data || res.html)) {
-         // writeFile('result.txt', res.data, { encoding: 'utf-8' });
-         const extracted = extractScrapedResult(res.data || res.html, settings.scraper_type);
+      if (settings && settings.scraper_type === 'proxy' && settings.proxy) {
+         res = await scraperClient;
+      } else {
+         res = await scraperClient.then((result:any) => result.json());
+      }
+
+      if (res && (res.data || res.html || res.result || res.results)) {
+         const extracted = extractScrapedResult(res.data || res.html || res.result || res.results, settings.scraper_type);
+         // await writeFile('result.txt', JSON.stringify(extracted), { encoding: 'utf-8' }).catch((err) => { console.log(err); });
         const serp = getSerp(keyword.domain, extracted);
         refreshedResults = { ID: keyword.ID, keyword: keyword.keyword, position: serp.postion, url: serp.url, result: extracted, error: false };
         console.log('SERP: ', keyword.keyword, serp.postion, serp.url);
+      } else {
+         scraperError = res.detail || res.error || 'Unknown Error';
+         throw new Error(res);
      }
   } catch (error:any) {
-      console.log('#### SCRAPE ERROR: ', keyword.keyword, error?.code, error?.response?.status, error?.response?.data, error);
-      // If Failed, Send back the original Keyword
+      console.log('#### SCRAPE ERROR: ', keyword.keyword, '. Error: ', scraperError);
+      refreshedResults.error = scraperError;
   }

   return refreshedResults;
@@ -124,31 +152,48 @@ export const scrapeKeywordFromGoogle = async (keyword:KeywordType, settings:Sett
 * @param {string} scraper_type - the type of scraper (Proxy or Scraper)
 * @returns {SearchResult[]}
 */
-export const extractScrapedResult = (content:string, scraper_type:string): SearchResult[] => {
+export const extractScrapedResult = (content: string, scraper_type:string): SearchResult[] => {
   const extractedResult = [];
-   const $ = cheerio.load(content);

+   const $ = cheerio.load(content);
   const hasNumberofResult = $('body').find('#search  > div > div');
   const searchResult = hasNumberofResult.children();
+   let lastPosition = 0;

   if (scraper_type === 'proxy') {
      const mainContent = $('body').find('#main');
      const children = $(mainContent).find('h3');

-      for (let index = 1; index < children.length; index += 1) {
+      for (let index = 0; index < children.length; index += 1) {
         const title = $(children[index]).text();
         const url = $(children[index]).closest('a').attr('href');
         const cleanedURL = url ? url.replace('/url?q=', '').replace(/&sa=.*/, '') : '';
-         extractedResult.push({ title, url: cleanedURL, position: index });
+         if (title && url) {
+            lastPosition += 1;
+            extractedResult.push({ title, url: cleanedURL, position: lastPosition });
+         }
+      }
+   } else if (scraper_type === 'serply') {
+      // results already in json
+      const results: SerplyResult[] = (typeof content === 'string') ? JSON.parse(content) : content as SerplyResult[];
+      for (const result of results) {
+         if (result.title && result.link) {
+            extractedResult.push({
+               title: result.title,
+               url: result.link,
+               position: result.realPosition,
+            });
+         }
      }
   } else {
-      for (let i = 1; i < searchResult.length; i += 1) {
+      for (let i = 0; i < searchResult.length; i += 1) {
         if (searchResult[i]) {
            const title = $(searchResult[i]).find('h3').html();
            const url = $(searchResult[i]).find('a').attr('href');
+            // console.log(i, url?.slice(0, 40), title?.slice(0, 40));
            if (title && url) {
-               extractedResult.push({ title, url, position: i });
-               // console.log(i, ' ',title, ' ', url);
+               lastPosition += 1;
+               extractedResult.push({ title, url, position: lastPosition });
            }
         }
     }
@@ -166,8 +211,8 @@ export const extractScrapedResult = (content:string, scraper_type:string): Searc
 export const getSerp = (domain:string, result:SearchResult[]) : SERPObject => {
   if (result.length === 0 || !domain) { return { postion: false, url: '' }; }
   const foundItem = result.find((item) => {
-      const itemDomain = item.url.match(/^(?:https?:)?(?:\/\/)?([^/?]+)/i);
-      return itemDomain && itemDomain.includes(domain);
+      const itemDomain = item.url.replace('www.', '').match(/^(?:https?:)?(?:\/\/)?([^/?]+)/i);
+      return itemDomain && itemDomain.includes(domain.replace('www.', ''));
   });
   return { postion: foundItem ? foundItem.position : 0, url: foundItem && foundItem.url ? foundItem.url : '' };
 };
@@ -182,7 +227,6 @@ export const retryScrape = async (keywordID: number) : Promise<void> => {
   if (!keywordID) { return; }
   let currentQueue: number[] = [];

-   // const filePath = path.join(__dirname, '..', '..', '..', '..', 'data', 'failed_queue.json');
   const filePath = `${process.cwd()}/data/failed_queue.json`;
   const currentQueueRaw = await readFile(filePath, { encoding: 'utf-8' }).catch((err) => { console.log(err); return '[]'; });
   currentQueue = JSON.parse(currentQueueRaw);
@@ -203,7 +247,6 @@ export const removeFromRetryQueue = async (keywordID: number) : Promise<void> =>
   if (!keywordID) { return; }
   let currentQueue: number[] = [];

-   // const filePath = path.join(__dirname, '..', '..', '..', '..', 'data', 'failed_queue.json');
   const filePath = `${process.cwd()}/data/failed_queue.json`;
   const currentQueueRaw = await readFile(filePath, { encoding: 'utf-8' }).catch((err) => { console.log(err); return '[]'; });
   currentQueue = JSON.parse(currentQueueRaw);
--- a/utils/sortFilter.ts
+++ b/utils/sortFilter.ts
@@ -6,6 +6,7 @@
 */
 export const sortKeywords = (theKeywords:KeywordType[], sortBy:string) : KeywordType[] => {
   let sortedItems = [];
+   const keywords = theKeywords.map((k) => ({ ...k, position: k.position === 0 ? 111 : k.position }));
   switch (sortBy) {
      case 'date_asc':
            sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => new Date(b.added).getTime() - new Date(a.added).getTime());
@@ -14,10 +15,12 @@ export const sortKeywords = (theKeywords:KeywordType[], sortBy:string) : Keyword
            sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => new Date(a.added).getTime() - new Date(b.added).getTime());
            break;
      case 'pos_asc':
-            sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => (b.position > a.position ? 1 : -1));
+            sortedItems = keywords.sort((a: KeywordType, b: KeywordType) => (b.position > a.position ? 1 : -1));
+            sortedItems = sortedItems.map((k) => ({ ...k, position: k.position === 111 ? 0 : k.position }));
            break;
      case 'pos_desc':
-            sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => (a.position > b.position ? 1 : -1));
+            sortedItems = keywords.sort((a: KeywordType, b: KeywordType) => (a.position > b.position ? 1 : -1));
+            sortedItems = sortedItems.map((k) => ({ ...k, position: k.position === 111 ? 0 : k.position }));
            break;
      case 'alpha_asc':
            sortedItems = theKeywords.sort((a: KeywordType, b: KeywordType) => (b.keyword > a.keyword ? 1 : -1));
Author	SHA1	Message	Date
Towfiq	6ac6c23168	chore(release): 0.1.7	2022-12-08 09:51:06 +06:00
Towfiq	f00006371d	fix: Email notifcations now sent everyday at 3am	2022-12-08 09:50:43 +06:00
Towfiq	a6af9d3475	fix: Throws better error logs in cron for debugging	2022-12-08 09:47:53 +06:00
Towfiq	480767deb2	fix: shortens hours and minutes in notif emails	2022-12-08 09:47:02 +06:00
Towfiq	62e0fba311	chore: Removes Unecessay code	2022-12-08 09:46:22 +06:00
Towfiq	6fb862e2a3	chore: Removes duplicate license badge.	2022-12-06 18:04:16 +06:00
Towfiq	612cd8d6b3	chore: Adds codacy badge.	2022-12-06 18:02:20 +06:00
Towfiq	1dfc01673e	style: Fixes more codacy nags	2022-12-06 18:01:25 +06:00
Towfiq	329ed24db8	style: Fixes Codacy issues.	2022-12-06 13:31:32 +06:00
Towfiq	dc8676a027	style: linter fixes.	2022-12-06 13:22:56 +06:00
Towfiq	a3362bc6ad	style: fixes stylelint issues.	2022-12-06 13:08:58 +06:00
Towfiq	1e8b4b6bf6	chore: adds stylelint	2022-12-06 13:08:27 +06:00
Towfiq	02a2d5fec2	chore(release): 0.1.6	2022-12-05 22:58:54 +06:00
Towfiq	1711a10b1f	Merge branch 'main' of https://github.com/towfiqi/serpbear	2022-12-05 22:52:13 +06:00
Towfiq I	a42b69bd2b	Merge pull request #20 from DennisCiba/main fix: Filter duplicates and empty lines on keyword creation	2022-12-05 22:50:26 +06:00
Dennis Ciba	7dfb4f99f8	Fix state mutation	2022-12-05 17:17:52 +01:00
Towfiq	d22992bf64	fix: Sort was buggy for keyword with >100 position resolves: #23	2022-12-05 19:15:24 +06:00
Towfiq	b450540d95	fix(UI): Adds tooltip for Domain action icons.	2022-12-04 20:34:26 +06:00
Towfiq	8688f323a5	chore: removed unnecessary file.	2022-12-04 20:33:09 +06:00
Towfiq	e9d7730ae7	fix: invalid json markup	2022-12-04 20:32:44 +06:00
Towfiq	a59903551e	fix: CSS Linter issues.	2022-12-04 20:32:30 +06:00
Dennis Ciba	2e85529183	Filter duplicates and empty lines on keyword creation	2022-12-04 12:24:31 +01:00
Towfiq	57182f17f6	chore(release): 0.1.5	2022-12-03 12:29:23 +06:00
Towfiq	d6da18fb01	fix: First search result items were being skipped. resolvres: #13	2022-12-03 12:29:04 +06:00
Towfiq	dd6a801ffd	fix: failed scrape messes up lastResult data in db	2022-12-03 12:28:21 +06:00
Towfiq	e1799fb2f3	feat: keyword not in first 100 now shows >100	2022-12-03 12:27:09 +06:00
Towfiq	c8ee418822	Merge branch 'main' of https://github.com/towfiqi/serpbear	2022-12-02 09:20:02 +06:00
Towfiq I	e7ab7d2db2	Merge pull request #9 from serply-inc/add-serply-io added support for serplyio	2022-12-02 09:19:00 +06:00
Towfiq	efb565ba00	fix: domains with - were not loading the keywords. resolves: #11	2022-12-02 07:36:09 +06:00
Towfiq	a11b0f223c	fix: removes empty spaces when adding domain.	2022-12-02 07:34:52 +06:00
googio	e6136db742	merge in main branch into add-serply	2022-12-01 13:28:43 -05:00
Towfiq	d01b65db04	chore(release): 0.1.4	2022-12-01 23:41:11 +06:00
googio	f51380442b	added support for serplyio	2022-12-01 12:24:28 -05:00
Towfiq	691055811c	fix: Emails were sending serps of previous day. Emails are now sent after 7 hours of daily SERP scrape CRON.	2022-12-01 22:18:42 +06:00
Towfiq	6d7cfec953	fix: scraper fails when result has domain w/o www When user adds a domain with www and the google search result has the domain without www, the scraper fails.	2022-12-01 22:15:47 +06:00
Towfiq	8c8064f222	feat: Failed scrape now shows error details in UI.	2022-12-01 22:08:54 +06:00
Towfiq	3d1c690076	fix: Domains with www weren't loading keywords. resolves: #8	2022-12-01 22:04:56 +06:00
Towfiq	1ed298f633	fix: Fixes Broken ScrapingRobot Integration.	2022-12-01 20:25:44 +06:00
Towfiq	38dc164514	fix: scraper fails if matched domain has www resolves: #6 , #7	2022-12-01 19:05:56 +06:00
Towfiq	7446b7868a	chore(release): 0.1.3	2022-12-01 12:01:30 +06:00
Towfiq	be4db26316	feat: Adds a search field in Country select field. resolves: #2	2022-12-01 12:01:13 +06:00
Towfiq	9fa80cf609	fix: No hint on how to add multiple keywords. resolves: #3	2022-12-01 12:00:27 +06:00
Towfiq	5acbe181ec	fix: could not add 2 character domains. resolves #1	2022-12-01 11:59:10 +06:00
Towfiq	a45237b230	fix: license location.	2022-11-30 23:37:13 +06:00
Towfiq	79b99a3896	chore: adds license.	2022-11-30 23:35:12 +06:00
Towfiq	4da725167e	chore: Added Doc Link to readme.	2022-11-30 20:54:29 +06:00
Towfiq	ddd10909ad	chore: Remove Unnecessary comments	2022-11-30 20:50:32 +06:00